One change
optimizing-for-cost-and-intelligence
about-claude/models/optimizing-for-cost-and-intelligence
Nearest release: v2.1.239, published 2 hours before this site recorded the change. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
about-claude/models/optimizing-for-cost-and-intelligence Changed · +104 / -29 lines
from line 70
Those breakpoint placements follow the standard pattern in [Explicit cache breakpoints](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#explicit-cache-breakpoints). -Three settings can break your cache during a task. Changing [`effort`](https://platform.claude.com/docs/en/build-with-claude/effort) between requests invalidates the cached prefix, so change it only where you would re-cache anyway, such as at a [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) boundary. Changing a [task budget](https://platform.claude.com/docs/en/build-with-claude/task-budgets) partway through does the same, so set it once, on the first request. Every [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing#context-editing-and-prompt-caching) pass invalidates the prefix from the point it clears and the next request pays to re-cache everything after it, so clear in a few large batches rather than many small ones. Make all three changes at natural breaks, then confirm cache reads have not dropped; if they have, [cache diagnostics](https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics) shows where the prefix diverged. +Four things can break your cache during a task. Anything that changes per request, such as a timestamp or a queue position, placed ahead of the stable prefix turns every request into a full cache write: on the triage run in [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens), a 25-token status line at the front of the system prompt cost $4.24 per run instead of $0.59, more than running with caching off. Keep per-request text in the newest user turn. Changing [`effort`](https://platform.claude.com/docs/en/build-with-claude/effort) between requests invalidates the cached prefix, so change it only where you would re-cache anyway, such as at a [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) boundary. Changing a [task budget](https://platform.claude.com/docs/en/build-with-claude/task-budgets) partway through does the same, so set it once, on the first request. Every [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing#context-editing-and-prompt-caching) pass invalidates the prefix from the point it clears and the next request pays to re-cache everything after it, so clear in a few large batches rather than many small ones. Make the last three changes at natural breaks, then confirm cache reads have not dropped; if they have, [cache diagnostics](https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics) shows where the prefix diverged. ### Trim input and context tokens Most agent requests carry tokens that never influence the answer. Trimming them costs nothing in output quality, although not every lever here saved money when measured. Two places to look: -* **Input trimming.** [Dynamic filtering](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool#dynamic-filtering) in the web fetch tool keeps boilerplate out of fetched pages, [image resizing](https://platform.claude.com/docs/en/build-with-claude/vision#evaluate-image-size) right-sizes vision inputs, and [tool search with deferred loading](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) loads tool definitions only when needed. [Programmatic tool calling](https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling) lets Claude run several tool calls from code so only the filtered result enters the context; its documentation reports 24% fewer input tokens on agentic search benchmarks, with a higher score. [Manage tool context](https://platform.claude.com/docs/en/agents-and-tools/tool-use/manage-tool-context) compares tool search, programmatic tool calling, prompt caching, and context editing. +* **Input trimming.** [Dynamic filtering](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool#dynamic-filtering) in the web fetch tool keeps boilerplate out of fetched pages, [image resizing](https://platform.claude.com/docs/en/build-with-claude/vision#evaluate-image-size) right-sizes vision inputs, and [tool search with deferred loading](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) loads tool definitions only when needed (measured later in this section). [Programmatic tool calling](https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling) lets Claude run several tool calls from code so only the filtered result enters the context; its documentation reports 24% fewer input tokens on agentic search benchmarks, with a higher score. [Manage tool context](https://platform.claude.com/docs/en/agents-and-tools/tool-use/manage-tool-context) compares tool search, programmatic tool calling, prompt caching, and context editing. * **Context lifecycle.** [Context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing) clears stale tool results, and [automatic compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) with its threshold stops long loops from carrying their whole history forward. -The levers interact with the cache and each other, so judge them by net effect, and use [cache diagnostics](https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics) to confirm your cached prefix survives each change. Anthropic turned the levers on one at a time for an issue-triage agent working through 20 real bug reports with screenshots from a public repository (and, for the second panel, a longer variant of the same job): +Defer tool definitions you are not using. Every tool definition attached to a request is input on every turn, and a few MCP servers add up to hundreds of them. Anthropic ran the triage agent with its own two tools plus a catalog of real tool definitions from public MCP servers, for a total of up to 502 tools, loading all of them or marking the extras `defer_loading` behind [tool search](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool): - + -Caching did almost all the work, and trimming took the total to 88%. Each bar is one run, so differences of $0.10 are noise; those shown here are not. Compaction needs a session long enough to trigger it: the 20-issue run never reached the 50,000-token floor once its inputs were trimmed, but on the second panel's longer variant it fired once and cut the bill a further 38%. +With every definition loaded, the run cost rose from $0.55 to $1.02, tracking the schema tokens on each request. With tool search, it stayed at $0.56 at every catalog size, 45% less at 502 tools. Accuracy was 15 to 18 of 20 in every cell either way, and the model never called a wrong tool, so at this scale the catalog costs money, not correctness. The same holds for tools that come through the [MCP connector](https://platform.claude.com/docs/en/agents-and-tools/mcp-connector): with a public GitHub MCP server attached, deferring its toolset (`default_config: {defer_loading: true}`) cut the run 20% at the same accuracy. -Context editing is the one lever here that is not free. Every clearing pass rewrites the cached conversation, which works against prompt caching; in this run, context editing cost more than it saved. Use it to make room in the context window, and [clear in a few large batches](https://platform.claude.com/docs/en/build-with-claude/context-editing#context-editing-and-prompt-caching). +Keep data files out of the prompt. When the model has to compute over a table, upload it with the [Files API](https://platform.claude.com/docs/en/build-with-claude/files) and let the model query it with [code execution](https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool) instead of pasting it in. Anthropic asked 25 aggregate questions[15](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) (sums, filtered counts, group-bys, and a date filter) over a 1,862-row public CSV, with the answers computed by pandas: + + +Pasted into the prompt, the table is about 91,000 input tokens on every request, and Claude Sonnet 5 answered 6 of 25 questions correctly at $5.01 per run. Uploaded, with code execution, it answered 25 of 25 at $0.40. Claude Opus 5 showed the same pattern (6 of 25 at $13.45 against 25 of 25 at $1.91). + +The levers interact with the cache and each other, so judge them by net effect, and use [cache diagnostics](https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics) to confirm your cached prefix survives each change. Anthropic measured them on an issue-triage agent working through 20 real bug reports with screenshots from a public repository, and on a longer variant of the same job with 2.6 times the tokens. With caching on, input trimming (image resizing and tool search) took a further 26% off the short run and 21% off the long one. The context levers are where the two runs diverge: + + + +The context levers only pay on a session long enough to need them. On the 20-issue run, context editing cost 74% more, and compaction and the prune changed nothing. On the long run, context editing changed nothing, compaction saved 32%, and the prune saved 39%. The prune is a few lines you write yourself: at each task boundary, replace large stale tool results with a one-line extract. It caches well because the edits sit at the tail of the conversation, where the next task adds new content anyway: 89% cache reads on the first request after a boundary and 81% on the requests between boundaries. Run-wide, the prune and context editing cache equally well. The prune is cheaper because context editing rewrites content mid-task that the prune deletes (about two thirds of the gap) and because it keeps the context about half the size (the other third). If you use context editing, [clear in a few large batches](https://platform.claude.com/docs/en/build-with-claude/context-editing#context-editing-and-prompt-caching). The prune, adapted from the harness: + +```python +import re + +PRUNED = "[pruned at issue boundary]" + + +def prune_task_boundary(messages, tool_name_by_id, threshold=2000): + """Call once per task boundary. Replaces large, stale search results with a one-line extract.""" + for message in messages: + if message["role"] != "user" or not isinstance(message["content"], list): + continue + for block in message["content"]: + if not (isinstance(block, dict) and block.get("type") == "tool_result"): + continue + if tool_name_by_id.get(block.get("tool_use_id")) != "search_issues": + continue + result_text = block.get("content") + if not isinstance(result_text, str) or len(result_text) <= threshold: + continue + if result_text.startswith(PRUNED): + continue # already pruned on an earlier boundary + # cap single-line results so the extract stays short + first_line = result_text.split("\n", 1)[0].strip()[:200] + refs = re.findall(r"#(\d+)", result_text)[:5] + extract = f"{PRUNED} {first_line}" + if refs: + extract += " kept refs: " + " ".join("#" + r for r in refs) + block["content"] = extract +``` + ### Batch work that can wait The [Batch API](https://platform.claude.com/docs/en/build-with-claude/batch-processing) takes 50% off every token of a request, including cached ones, in exchange for results arriving any time within 24 hours. Route every request no one is waiting on through a batch, and keep the interactive path for the rest. Batching is the second-largest free lever after caching for unattended agent work: evaluation runs, backfills, and scheduled jobs such as a recurring run of the issue-triage agent from the [token-trimming measurement](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens). It combines with everything on this page except interactivity, but is not available for Claude Managed Agents sessions, which are interactive by design (see [Claude Managed Agents pricing](https://platform.claude.com/docs/en/about-claude/pricing#claude-managed-agents-pricing)).
from line 244
* **`max_tokens`** caps a single response, invisibly to the model, so lowering it does not make the model economize. The turns that needed the room are discarded and still billed. On an internal repository-task benchmark[12](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs), a 16,384-token cap ended 15% of Claude Opus 5's attempts and a third of Claude Fable 5's, none of them solved. Capped runs spent less per attempt but bought proportionally fewer solves, so cost per solved task was the same as at 64,000. At that setting nothing was cut off, and Fable solved 54.6% of tasks instead of 36.6% on the problems both runs scored (on a separate cut of the SWE-bench Pro[3](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) subset, described in reference 12, 92% instead of 90%). Retrying capped attempts only adds cost: at the same cap they never succeeded, and at a higher one you also pay for the wasted attempt. Set `max_tokens` to 64,000 for agentic work (128,000, the maximum, at `xhigh` or `max` effort), [stream responses](https://platform.claude.com/docs/en/build-with-claude/streaming) that large, treat [`stop_reason: max_tokens`](https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons#max-tokens) as a failure, and save money with effort and task budgets, which the model can see. * **Session budgets on Claude Managed Agents** are the hard stop. A [session budget](https://platform.claude.com/docs/en/managed-agents/budgets) is a dollar cap on one session at list rates for tokens, searches, and session time. At the cap, the session pauses with `stop_reason: budget_reached`; raising the budget resumes it. It is platform-enforced, works on any model with a list price (including Claude Sonnet 5), and combines with the advisory task budget. Deployments apply the same field to every run. +Ask for shorter answers. Output tokens cost five times input tokens on Claude Sonnet 5, and in an agent loop every token the model writes comes back as input on every later turn, so you pay for a long answer again and again. Anthropic ran the triage job under three final-answer instructions, three runs each, with the same model and tools. The original asked for two lines: + +```text wrap +4. Finish with exactly two lines: +LABEL: <one of: bug-confirmed, needs-more-info, duplicate-candidate, feature-request, upstream-issue, perf, ui-polish> +SUMMARY: <one or two sentences for the engineering team> +``` + +The shorter variant asked for one: + +```text wrap +4. Finish with exactly one line in this form: +DECISION | LABEL | REASON +where DECISION is one of: triage-now, needs-info, close-duplicate; LABEL is one of: bug-confirmed, needs-more-info, duplicate-candidate, feature-request, upstream-issue, perf, ui-polish; REASON is one clause under 15 words. Output nothing after that line. +``` + +The longer variant asked for a memo with five headed sections: problem summary, evidence, duplicate check, recommended label, and next steps. For one issue, a queued prompt that never sends after a skipped question, the first two answers were: + +```text wrap +LABEL: bug-confirmed +SUMMARY: When a user submits a new prompt instead of answering an agent's pending question, the question is cancelled/skipped but the new prompt remains stuck in "QUEUED" state indefinitely since it's waiting on a response to the now-cancelled question; the queued prompt should be processed immediately after cancellation. +``` + +```text wrap +triage-now | bug-confirmed | Clear repro steps show prompt queues indefinitely after cancelled question. +``` + + + +The one-line answer used 39% fewer output tokens and cost $0.49 for the run against $0.57, with the same accuracy against the gold labels (78% compared with 80%, inside run-to-run noise). The memo used six times the output tokens of the original and cost $1.40, 2.8 times the one-line answer, for 85%, also inside the noise. The three formats are equally accurate; they differ in what you pay. Ask for the answer you will read, not the one that looks thorough. + The first of two `max_tokens` charts plots cost per attempt and per solved task at each cap: 
from line 306
The first is the gap between the models. The advisor can only hand over capability the executor lacks: on GPQA Diamond[9](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) a Claude Haiku 4.5 executor gained a great deal from a Claude Opus 5 advisor, a Claude Sonnet 5 executor gained a few points, and a frontier executor almost nothing. -The second, and the fragile one, is whether the executor actually asks (the consult rate). An executor at low effort can stop noticing it is stuck: a pairing that consults on most tasks at the default effort can fall to consulting on almost none when effort is lowered, and then scores below the executor alone. The rate also varies by task: on DeepSWE[10](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) a low-effort Sonnet 5 executor kept asking and gained 23 points; on SWE-bench Pro[3](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) the same executor stopped. When the executor does ask, it gets most of the way there. Across the pairings in the following chart, the advisor closed 60% to 90% of the gap to the stronger model while that model was paid for only on the consultations, which makes the cost cases possible: +The second, and the fragile one, is whether the executor actually asks (the consult rate). An executor at low effort can stop detecting that it is stuck: a pairing that consults on most tasks at the default effort can fall to consulting on almost none when effort is lowered, and then scores below the executor alone. The rate also varies by task: on DeepSWE[10](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) a low-effort Sonnet 5 executor kept asking and gained 23 points; on SWE-bench Pro[3](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#refs) the same executor stopped. When the executor does ask, it recovers most of the gap. Across the pairings in the following chart, the advisor closed 50% to 90% of the gap to the stronger model and you pay for the stronger model only on the consultations, which is what makes the cost cases possible: 
from line 599
The following table lists the levers in the order to try them: -| Lever | Saving in these runs | Quality cost | Latency | Where | -| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Prompt caching | Cost cut by a factor of 2.5 to 3.7 on agent loops; 83% on the triage run | None | Faster | [Cache repeated context](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#cache-repeated-context) | -| Input trimming | A further 5 percentage points on the triage run | None | Neutral | [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens) | -| Compaction | 38% on the long triage run; nothing on short loops | None measured | Neutral | [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens) | -| Batch API | 50% | None | Results within 24 hours | [Batch work that can wait](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#batch-work-that-can-wait) | -| Prompt audit against the current model | 14% on both migrations measured | None; a gain on one | Faster (fewer tool rounds) | [Audit prompts against the current model](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#audit-prompts-against-the-current-model) | -| Lower effort | Knowledge work: `medium` 15% to 30%, `low` a third to a half; long coding: `medium` about half, `low` about three quarters | 1 to 3 points on knowledge work, 2 to 8 on long coding | Faster | [Tune effort](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#tune-effort) | -| Re-run failures | About half, at the same pass rate | None | Two runs on the tasks that fail | [Re-run failures at higher effort](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#re-run-failures-at-higher-effort) | -| Task budget | 18% to 47% | 3 to 4 points | Faster | [Set budgets and output caps](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#set-budgets-and-output-caps) | -| Raising `max_tokens` | None per solved task, but more tasks solved | Gains of 2 to 18 points | Neutral | [Set budgets and output caps](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#set-budgets-and-output-caps) | -| Advisor | Depends on the capability gap and the consult rate; the chart-reading pairing scored above both models' effort curves, the coding pairing only marginally | Small gains | About two extra calls per task | [Advisor strategy](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#advisor-strategy-escalate-hard-decisions) | -| Orchestrator | More than 60% below the frontier model beyond one context window; about half on routine tails | 2 to 6 points below the frontier model | Much faster on large inputs | [Orchestrator strategy](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#orchestrator-strategy-delegate-bulk-work) | +| Lever | Saving in these runs | Quality cost | Latency | Where | +| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Prompt caching | Cost cut by a factor of 2.5 to 3.7 on agent loops; 83% on the triage run | None | Faster | [Cache repeated context](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#cache-repeated-context) | +| Input trimming | A further 5 percentage points on the triage run | None | Neutral | [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens) | +| Prune stale tool results at task boundaries | 39% on the long triage run (compaction 32%); nothing on short loops | None measured | Neutral | [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens) | +| Tool search | 45% with 500 tool definitions attached; 20% with a GitHub MCP server | None | Neutral | [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens) | +| Data files through code execution | 92% on a 25-question data task | A gain, 25 of 25 instead of 6 of 25 | Faster | [Trim input and context tokens](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#trim-input-and-context-tokens) | +| Batch API | 50% | None | Results within 24 hours | [Batch work that can wait](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#batch-work-that-can-wait) | +| Prompt audit against the current model | 14% on both migrations measured | None; a gain on one | Faster (fewer tool rounds) | [Audit prompts against the current model](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#audit-prompts-against-the-current-model) | +| Lower effort | Knowledge work: `medium` 15% to 30%, `low` a third to a half; long coding: `medium` about half, `low` about three quarters | 1 to 3 points on knowledge work, 2 to 8 on long coding | Faster | [Tune effort](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#tune-effort) | +| Re-run failures | About half, at the same pass rate | None | Two runs on the tasks that fail | [Re-run failures at higher effort](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#re-run-failures-at-higher-effort) | +| Task budget | 18% to 47% | 3 to 4 points | Faster | [Set budgets and output caps](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#set-budgets-and-output-caps) | +| Ask for shorter answers | 39% of output tokens, 14% of cost on the triage run | None | Faster | [Set budgets and output caps](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#set-budgets-and-output-caps) | +| Raising `max_tokens` | None per solved task, but more tasks solved | Gains of 2 to 18 points | Neutral | [Set budgets and output caps](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#set-budgets-and-output-caps) | +| Advisor | Depends on the capability gap and the consult rate; the chart-reading pairing scored above both models' effort curves, the coding pairing only marginally | Small gains | About two extra calls per task | [Advisor strategy](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#advisor-strategy-escalate-hard-decisions) | +| Orchestrator | More than 60% below the frontier model beyond one context window; about half on routine tails | 2 to 6 points below the frontier model | Much faster on large inputs | [Orchestrator strategy](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#orchestrator-strategy-delegate-bulk-work) | ## Benchmarks referenced All measurements are Anthropic-internal runs of these benchmarks. Unless noted, costs are USD at August 2026 list prices; Claude Sonnet 5 figures use $2 and $10 per million input and output tokens. Charts labeled "notional USD" price each request's token counts at those rates rather than reporting invoices. -1. **WideSearch:** Wong et al., "WideSearch: Benchmarking Agentic Broad Info-Seeking," arXiv.07999, 2025. Broad web-research tasks graded on a many-row table's completeness and accuracy; 200 problems, 3 runs per configuration. The caching and effort charts come from separate runs, so per-problem costs differ slightly. The cost-concentration chart is a separate 20-problem run, 3 runs per problem, costed from per-request billing records. +1. **WideSearch:** Wong et al., "WideSearch: Benchmarking Agentic Broad Info-Seeking," arXiv:2508.07999, 2025. Broad web-research tasks graded on a many-row table's completeness and accuracy; 200 problems, 3 runs per configuration. The caching and effort charts come from separate runs, so per-problem costs differ slightly. The cost-concentration chart is a separate 20-problem run, 3 runs per problem, costed from per-request billing records. 2. **GDPval:** OpenAI, "GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks," 2025. Knowledge-work deliverables graded against task rubrics; a 210-task run of the released gold set, one attempt per task. A Claude model grades, so absolute scores may differ from published results. -3. **SWE-bench Pro:** Scale AI, "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", 2025. A 482-problem subset selected for compatibility with Anthropic's evaluation harness; scores are not comparable to the public leaderboard. Claude Opus 5 at the default effort averages two runs; reduced-effort settings are single runs. Escalation figures come task by task from those runs: `low` first, then the default on its failures, solved 92.5% to 93.6% across run pairings for about $0.70; `medium` first, 93.8% to 94.2% for about $0.95; the default re-run on its own failures, 94.0% for $1.58; everything at the default, 90.9% to 92.5% for $1.39. The Claude Sonnet 5 executor pairings on the advisor chart come from the same August 2026 series on this subset: the Sonnet-plus-Opus pairing was run twice (a run and an exact replication) and the low-effort pairing once; the task-budget figures are one run per budget on the same subset. The Claude Fable 5 figure in [Compare models](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#compare-models-on-cost-per-task) is a single July 2026 run, also the task-budget chart's unbudgeted baseline; every budgeted run completed all 482 problems without harness errors. +3. **SWE-bench Pro:** Scale AI, "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", 2025. A 482-problem subset selected for compatibility with Anthropic's evaluation harness; scores are not comparable to the public leaderboard. Claude Opus 5 at the default effort averages two runs; reduced-effort settings are single runs. Escalation figures come task by task from those runs: `low` first, then the default on its failures, solved 92.5% to 93.6% across run pairings for about $0.70; `medium` first, 93.8% to 94.2% for about $0.95; the default re-run on its own failures, 94.0% for $1.58; everything at the default, 90.9% to 92.5% for $1.39. The Claude Sonnet 5 executor pairings on the advisor chart come from the same August 2026 series on this subset: the Sonnet-plus-Opus pairing was run twice (a run and an exact replication), the low-effort pairing once, and Claude Sonnet 5 alone twice (77.4%, the baseline for both Pro rows); the task-budget figures are one run per budget on the same subset. The Claude Fable 5 figure in [Compare models](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#compare-models-on-cost-per-task) is a single July 2026 run, also the task-budget chart's unbudgeted baseline; every budgeted run completed all 482 problems without harness errors. 4. **BrowseComp:** Wei et al., "BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents," OpenAI, 2025. Effort figures use a 500-problem cut, one to three runs per setting. The cost-insurance chart uses 10 reliably solved problems from a 26-problem slice, 50 delegated and 70 solo runs ($6.45 compared with $11.99 per run in expectation); delegated figures carry a measurement band of about 20%. -5. **Agent-architecture scaling:** Kim et al., "Towards a Science of Scaling Agent Systems," arXiv.08296, 2025. Independent external study, cited only for the direction of the finding on when delegation does not pay, not for any figure. -6. **DeepWideSearch:** "DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking," arXiv.20168, 2025. The 220 questions span 15 domains, each combining many-row collection with multi-hop retrieval; measured on the benchmark's standing row set, 3 runs per configuration. -7. **DeepResearch Bench II:** Li et al., "DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report," arXiv.08536, 2026. Its 132 research tasks across 22 domains are graded against expert-derived binary rubrics; measured on a 50-task subset stratified across all themes, one attempt per task, 3 runs, scored on tasks no configuration refused. Claude Opus 4.6 judges under the benchmark's rubric protocol; the original uses a different judge, and an Anthropic judge may favor the house style. The runs predate Claude Opus 5, hence its absence from the [Compare models](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#compare-models-on-cost-per-task) chart. The Sonnet 5 cost differs slightly between the caching chart (inference cost, with and without caching) and that chart (all-in cost, caching on); both come from the same runs. +5. **Agent-architecture scaling:** Kim et al., "Towards a Science of Scaling Agent Systems," arXiv:2512.08296, 2025. Independent external study, cited only for the direction of the finding on when delegation does not pay, not for any figure. +6. **DeepWideSearch:** "DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking," arXiv:2510.20168, 2025. The 220 questions span 15 domains, each combining many-row collection with multi-hop retrieval; measured on the benchmark's standing row set, 3 runs per configuration. +7. **DeepResearch Bench II:** Li et al., "DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report," arXiv:2601.08536, 2026. Its 132 research tasks across 22 domains are graded against expert-derived binary rubrics; measured on a 50-task subset stratified across all themes, one attempt per task, 3 runs, scored on tasks no configuration refused. Claude Opus 4.6 judges under the benchmark's rubric protocol; the original uses a different judge, and an Anthropic judge may favor the house style. The runs predate Claude Opus 5, hence its absence from the [Compare models](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#compare-models-on-cost-per-task) chart. The Sonnet 5 cost differs slightly between the caching chart (inference cost, with and without caching) and that chart (all-in cost, caching on); both come from the same runs. 8. **Corpus defect sweep:** Anthropic-internal, for work larger than one context window: a 21.6-million-token corpus from 14 public Python package sources with 130 planted defects and deterministic grading; protocol fixed before the runs and internally reviewed; three runs per configuration. Every configuration ran on Claude Managed Agents. The charted team configuration is an August 2026 run in which the Claude Fable 5 coordinator ran the whole sweep inside the platform at its documented limit of 25 concurrent Claude Sonnet 5 workers; its three episodes scored F1 0.842, 0.805, and 0.810 for $263, $299, and $261. The solo configurations are July 2026 runs on the same corpus build. Absolute F1 is specific to this corpus build, not comparable across benchmarks; configuration comparisons are like for like. 9. **GPQA Diamond:** Rein et al., "GPQA: A Graduate-Level Google-Proof Q\&A Benchmark," 2023. The 198-question Diamond subset, measured August 2026, two runs per configuration, model-graded against reference answers, advisor tokens metered per request. A platform safety check refused two biology questions on the Sonnet and Opus executors; excluding them changes no comparison by more than one point. -10. **DeepSWE:** Datacurve, "DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks," arXiv.07946, 2026. Measured August 2026: 113 original tasks across five languages with program-based verifiers. Pairings are two runs each with advisor tokens metered per request, and used a client-side advisor loop rather than the advisor tool, with identical accounting. Single-model effort sweeps are single runs priced from token counts, a cache-aware approximation. Costs per task are run totals divided by 113. +10. **DeepSWE:** Datacurve, "DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks," arXiv:2607.07946, 2026. Measured August 2026: 113 original tasks across five languages with program-based verifiers. Pairings are two runs each with advisor tokens metered per request, and used a client-side advisor loop rather than the advisor tool, with identical accounting. Single-model effort sweeps are single runs priced from token counts, a cache-aware approximation. Costs per task are run totals divided by 113. 11. **Internal agentic-coding benchmark:** Anthropic-internal: 370 repository tasks graded by the repositories' own tests. The API figures (Opus 5 alone, Fable 5 alone, and the pairing) were measured August 2026 at the default effort with a 128,000-token output cap, one run per configuration: five attempts per task at the default settings and for the pairing, one at `low` and `medium`; the pairing averaged about two advisor consultations per attempt; costs are per attempt. The Claude Code figures are July 2026 runs of the same tasks, one run per configuration, costs approximate. 12. **Internal repository-task benchmark (cap measurement):** A separate Anthropic-internal set of about 130 repository tasks, run August 2026 with a plain API agent loop, one attempt per task. The 16,384-token figures average two runs per model; the 64,000-token figures are single runs (124 tasks scored for Opus 5; 108 for Claude Fable 5, the environment having skipped the rest before the model ran). About half the Fable attempts the 16,384 cap had ended solved at 64,000; a further Fable run at 128,000 scored 56.1%, within noise of the 64,000 run. The SWE-bench Pro cap figures are one Claude Fable 5 run per cap at the default effort on a 100-problem subset stratified from reference 3's 482-problem set, not comparable to its scores. The chart's per-turn distributions come from the Opus run at 64,000 and the Fable run at 128,000, so neither is cut off by its own cap. 13. **Chartography:** Surge AI, "Chartography," 2026. The complete released 100-question set, measured August 2026 with Anthropic's implementation on Claude Managed Agents (standard cloud sandbox; advisor configurations use the Managed Agents advisor). Claude Sonnet 4.6 grades instead of the reference judge and the benchmark runs with tools, so scores compare across configurations here but not to the published leaderboard. Two runs per configuration, pooled; run-to-run spreads were 4 to 10 points. Costs exclude sandbox time, which added under 1%. The consult-rate comparison comes from rerunning the same configurations on the Messages API with a container tool set. 14. **Support-desk prompt-audit evaluation:** An Anthropic-constructed set of 44 support tickets with deterministic grading, run August 2026 under six system prompts, each adding to the same clean prompt one pattern common in prompts written for Claude Opus 4.8 and Claude Sonnet 4.6. Each chart point is one of three cases (older model, newer model on the same prompt, newer model after the audit) averaged over the six prompts and 44 tickets. The Opus 5 accuracy gain has a 95% confidence interval of 3 to 8 points; the Sonnet accuracy differences are within noise. +15. **Data-file question set:** An Anthropic-constructed set of 25 aggregate questions over a 1,862-row slice of a public liquor-sales CSV, with ground truth computed by pandas and exact-match grading, run August 2026 on Claude Sonnet 5 and Claude Opus 5 with thinking disabled (the in-context arm cannot complete at the default), a 4,000-token output cap, and no prompt caching, three runs per configuration. The file arm uploads the CSV through the Files API and uses the `code_execution_20260120` tool. ## Next steps