develop-tests changedtest-and-evaluate/develop-tests
Nearest release: v2.1.307, published an hour after this site recorded the change. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Recorded here
Lines+10added
Lines−10removed
From line
35
where the diff opens
First seen
14 Aug 2026
this site's first read of the page
Recorded edits11to this page, all time
The whole hunk
from line 35, old and new numbered
/
from line 35
3535 **Quantitative methods:**
3636
3737 * A/B testing: Compare performance against a baseline model or earlier version.
38 * User feedback: Implicit measures like task completion rates.
38 * User feedback: Implicit measures such as task completion rates.
3939 * Edge case analysis: Percentage of edge cases handled without errors.
4040
4141 **Qualitative scales:**
from line 63
6363
6464<AccordionGroup>
6565 <Accordion title="Task fidelity">
66 How well does the model need to perform on the task? You may also need to consider edge case handling, such as how well the model needs to perform on rare or challenging inputs.
66 How well does the model need to perform on the task? You might also need to consider edge case handling, such as how well the model needs to perform on rare or challenging inputs.
6767 </Accordion>
6868
6969 <Accordion title="Consistency">
from line 71
7171 </Accordion>
7272
7373 <Accordion title="Relevance and coherence">
74 How well does the model directly address the user's questions or instructions? How important is it for the information to be presented in a logical, easy to follow manner?
74 How well does the model directly address the user's questions or instructions? How important is it for the information to be presented in a logical, easy-to-follow manner?
7575 </Accordion>
7676
7777 <Accordion title="Tone and style">
from line 91
9191 </Accordion>
9292
9393 <Accordion title="Price">
94 What is your budget for running the model? Consider factors like the cost for each API call, the size of the model, and the frequency of usage.
94 What is your budget for running the model? Consider factors such as the cost for each API call, the size of the model, and the frequency of usage.
9595 </Accordion>
9696</AccordionGroup>
9797
from line 126
126126
127127<AccordionGroup>
128128 <Accordion title="Task fidelity (sentiment analysis) - exact match evaluation">
129 **What it measures:** Exact match evals measure whether the model's output matches a predefined correct answer, typically after normalizing whitespace and case. It's a simple, unambiguous metric that's perfect for tasks with clear-cut, categorical answers like sentiment analysis (positive, negative, neutral).
129 **What it measures:** Exact match evals measure whether the model's output matches a predefined correct answer, typically after normalizing whitespace and case. It's a simple, unambiguous metric that's perfect for tasks with clear-cut, categorical answers such as sentiment analysis (positive, negative, neutral).
130130
131131 **Example eval test cases:** 1,000 tweets with human-labeled sentiments.
132132
from line 1561
15611561 </Accordion>
15621562
15631563 <Accordion title="Tone and style (customer service) - LLM-based Likert scale">
1564 **What it measures:** The LLM-based Likert scale is a psychometric scale that uses an LLM to judge subjective attitudes or perceptions. Here, it's used to rate the tone of responses on a scale from 1 to 5. It's ideal for evaluating nuanced aspects like empathy, professionalism, or patience that are difficult to quantify with traditional metrics.
1564 **What it measures:** The LLM-based Likert scale is a psychometric scale that uses an LLM to judge subjective attitudes or perceptions. Here, it's used to rate the tone of responses on a scale from 1 to 5. It's ideal for evaluating nuanced aspects such as empathy, professionalism, or patience that are difficult to quantify with traditional metrics.
15651565
15661566 **Example eval test cases:** 100 customer inquiries with target tone (empathetic, patient, professional).
15671567
from line 2032
20322032 </Accordion>
20332033
20342034 <Accordion title="Privacy preservation (medical chatbot) - LLM-based binary classification">
2035 **What it measures:** Binary classification determines if an input belongs to one of two classes. Here, it's used to classify whether a response contains PHI or not. This method can account for context and identify subtle or implicit forms of PHI that rule-based systems might miss.
2035 **What it measures:** Binary classification determines if an input belongs to one of two classes. Here, it's used to classify whether a response contains PHI. This method can account for context and identify subtle or implicit forms of PHI that rule-based systems might miss.
20362036
20372037 **Example eval test cases:** 500 simulated patient queries, some with PHI.
20382038
from line 3232
32323232
323332332. **Human grading:** Most flexible and high quality, but slow and expensive. Avoid if possible.
32343234
32353. **LLM-based grading:** Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first then scale.
32353. **LLM-based grading:** Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.
32363236
32373237### Tips for LLM-based grading
32383238
32393239* **Have detailed, clear rubrics:** "The answer should always mention 'Acme Inc.' in the first sentence. If it does not, the answer is automatically graded as 'incorrect.'"
32403240 <Note>
3241 A given use case, or even a specific success criteria for that use case, might require several rubrics for holistic evaluation.
3241 A given use case, or even a specific success criterion for that use case, might require several rubrics for holistic evaluation.
32423242 </Note>
32433243* **Empirical or specific:** For example, instruct the LLM to output only 'correct' or 'incorrect', or to judge from a scale of 1–5. Purely qualitative evaluations are hard to assess quickly and at scale.
32443244* **Encourage reasoning:** Use a grader model with [thinking](https://platform.claude.com/docs/en/build-with-claude/thinking) on, so that it reasons before it produces an evaluation score. This increases evaluation performance, particularly for tasks requiring complex judgment.
from line 3716
37163716 **Tip:** Drop this page into the chat as guidance for Claude!
37173717 </Card>
37183718
3719 <Card title="Evals cookbook" icon="link" href="https://platform.claude.com/cookbook/misc-building-evals">
3719 <Card title="Evals recipe" icon="link" href="https://platform.claude.com/cookbook/misc-building-evals">
37203720 More code examples of human-, code-, and LLM-graded evals.
37213721 </Card>
37223722</CardGroup>
No line in this hunk matches that.