develop-tests changedtest-and-evaluate/develop-tests
Nearest release: v2.1.312, published under an hour before this site recorded the change. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Recorded here
Lines+18added
Lines−18removed
From line
14
where the diff opens
First seen
14 Aug 2026
this site's first read of the page
Recorded edits13to this page, all time
The whole hunk
from line 14, old and new numbered
/
from line 14
1414
1515Good success criteria are:
1616
17* **Specific:** Clearly define what you want to achieve. Instead of "good performance," specify "accurate sentiment classification."
17* **Specific:** Clearly define what you want to achieve. Instead of “good performance,” specify “accurate sentiment classification.”
1818
1919* **Measurable:** Use quantitative metrics or well-defined qualitative scales. Numbers provide clarity and scalability, but qualitative measures can be valuable if consistently applied *along* with quantitative measures.
2020
21 * Even "hazy" topics such as ethics and safety can be quantified:
21 * Even “hazy” topics such as ethics and safety can be quantified:
2222
2323 | | Safety criteria |
2424 | ---- | ------------------------------------------------------------------------------------------ |
from line 40
4040
4141 **Qualitative scales:**
4242
43 * Likert scales: "Rate coherence from 1 (nonsensical) to 5 (perfectly logical)"
43 * Likert scales: “Rate coherence from 1 (nonsensical) to 5 (perfectly logical)”
4444 * Expert rubrics: Linguists rating translation quality on defined criteria
4545 </Accordion>
4646
4747* **Achievable:** Base your targets on industry benchmarks, prior experiments, AI research, or expert knowledge. Your success metrics should not be unrealistic to current frontier model capabilities.
4848
49* **Relevant:** Align your criteria with your application's purpose and user needs. Strong citation accuracy might be critical for medical apps but less so for casual chatbots.
49* **Relevant:** Align your criteria with your application’s purpose and user needs. Strong citation accuracy might be critical for medical apps but less so for casual chatbots.
5050
5151<Accordion title="Example task fidelity criteria for sentiment analysis">
5252 | | Criteria |
from line 67
6767 </Accordion>
6868
6969 <Accordion title="Consistency">
70 How similar do the model's responses need to be for similar types of input? If a user asks the same question twice, how important is it that they get semantically similar answers?
70 How similar do the model’s responses need to be for similar types of input? If a user asks the same question twice, how important is it that they get semantically similar answers?
7171 </Accordion>
7272
7373 <Accordion title="Relevance and coherence">
74 How well does the model directly address the user's questions or instructions? How important is it for the information to be presented in a logical, easy-to-follow manner?
74 How well does the model directly address the user’s questions or instructions? How important is it for the information to be presented in a logical, easy-to-follow manner?
7575 </Accordion>
7676
7777 <Accordion title="Tone and style">
78 How well does the model's output style match expectations? How appropriate is its language for the target audience?
78 How well does the model’s output style match expectations? How appropriate is its language for the target audience?
7979 </Accordion>
8080
8181 <Accordion title="Privacy preservation">
from line 87
8787 </Accordion>
8888
8989 <Accordion title="Latency">
90 What is the acceptable response time for the model? This depends on your application's real-time requirements and user expectations.
90 What is the acceptable response time for the model? This depends on your application’s real-time requirements and user expectations.
9191 </Accordion>
9292
9393 <Accordion title="Price">
from line 103
103103 | Bad | The model should classify sentiments well |
104104 | Good | On a held-out test set of 10,000 diverse Twitter posts, the sentiment analysis model should achieve:<br />- an F1 score of at least 0.85<br />- 99.5% non-toxic outputs<br />- 90% of errors would cause inconvenience, not egregious error\*<br />- 95% response time \< 200ms |
105105
106 \*In reality, you would also define what "inconvenience" and "egregious" mean.
106 \*In reality, you would also define what “inconvenience” and “egregious” mean.
107107</Accordion>
108108
109109***
from line 112
112112
113113### Eval design principles
114114
1151. **Be task-specific:** Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!
1151. **Be task-specific:** Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!
116116 <Accordion title="Example edge cases">
117117 * Irrelevant or nonexistent input data
118118 * Overly long input data or user input
from line 126
126126
127127<AccordionGroup>
128128 <Accordion title="Task fidelity (sentiment analysis): exact match evaluation">
129 **What it measures:** Exact match evals measure whether the model's output matches a predefined correct answer, typically after normalizing whitespace and case. It's a simple, unambiguous metric that's perfect for tasks with clear-cut, categorical answers such as sentiment analysis (positive, negative, neutral).
129 **What it measures:** Exact match evals measure whether the model’s output matches a predefined correct answer, typically after normalizing whitespace and case. It’s a simple, unambiguous metric that’s perfect for tasks with clear-cut, categorical answers such as sentiment analysis (positive, negative, neutral).
130130
131131 **Example eval test cases:** 1,000 tweets with human-labeled sentiments.
132132
from line 488
488488 </Accordion>
489489
490490 <Accordion title="Consistency (FAQ bot): cosine similarity evaluation">
491 **What it measures:** Cosine similarity measures the similarity between two vectors (in this case, sentence embeddings of the model's output using [Sentence-BERT (SBERT)](https://sbert.net/)) by computing the cosine of the angle between them. Values closer to 1 indicate higher similarity. It's ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.
491 **What it measures:** Cosine similarity measures the similarity between two vectors (in this case, sentence embeddings of the model’s output using [Sentence-BERT (SBERT)](https://sbert.net/)) by computing the cosine of the angle between them. Values closer to 1 indicate higher similarity. It’s ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.
492492
493493 **Example eval test cases:** 50 groups with a few paraphrased versions each.
494494
from line 1561
15611561 </Accordion>
15621562
15631563 <Accordion title="Tone and style (customer service): LLM-based Likert scale">
1564 **What it measures:** The LLM-based Likert scale is a psychometric scale that uses an LLM to judge subjective attitudes or perceptions. Here, it's used to rate the tone of responses on a scale from 1 to 5. It's ideal for evaluating nuanced aspects such as empathy, professionalism, or patience that are difficult to quantify with traditional metrics.
1564 **What it measures:** The LLM-based Likert scale is a psychometric scale that uses an LLM to judge subjective attitudes or perceptions. Here, it’s used to rate the tone of responses on a scale from 1 to 5. It’s ideal for evaluating nuanced aspects such as empathy, professionalism, or patience that are difficult to quantify with traditional metrics.
15651565
15661566 **Example eval test cases:** 100 customer inquiries with target tone (empathetic, patient, professional).
15671567
from line 2032
20322032 </Accordion>
20332033
20342034 <Accordion title="Privacy preservation (medical chatbot): LLM-based binary classification">
2035 **What it measures:** Binary classification determines if an input belongs to one of two classes. Here, it's used to classify whether a response contains PHI. This method can account for context and identify subtle or implicit forms of PHI that rule-based systems might miss.
2035 **What it measures:** Binary classification determines if an input belongs to one of two classes. Here, it’s used to classify whether a response contains PHI. This method can account for context and identify subtle or implicit forms of PHI that rule-based systems might miss.
20362036
20372037 **Example eval test cases:** 500 simulated patient queries, some with PHI.
20382038
from line 2585
25852585 </Accordion>
25862586
25872587 <Accordion title="Context utilization (conversation assistant): LLM-based ordinal scale">
2588 **What it measures:** Similar to the Likert scale, the ordinal scale measures on a fixed, ordered scale (1–5). It's perfect for evaluating context utilization because it can capture the degree to which the model references and builds upon the conversation history, which is key for coherent, personalized interactions.
2588 **What it measures:** Similar to the Likert scale, the ordinal scale measures on a fixed, ordered scale (1–5). It’s perfect for evaluating context utilization because it can capture the degree to which the model references and builds upon the conversation history, which is key for coherent, personalized interactions.
25892589
25902590 **Example eval test cases:** 100 multi-turn conversations with context-dependent questions.
25912591
from line 3216
32163216</Tip>
32173217
32183218<Tip>
3219 If you don't know what eval methods might be useful to assess for your success criteria, you can also brainstorm with Claude!
3219 If you don’t know what eval methods might be useful to assess for your success criteria, you can also brainstorm with Claude!
32203220</Tip>
32213221
32223222***
from line 3236
32363236
32373237### Tips for LLM-based grading
32383238
3239* **Have detailed, clear rubrics:** "The answer should always mention 'Acme Inc.' in the first sentence. If it does not, the answer is automatically graded as 'incorrect.'"
3239* **Have detailed, clear rubrics:** “The answer should always mention ‘Acme Inc.’ in the first sentence. If it does not, the answer is automatically graded as ‘incorrect.’”
32403240 <Note>
32413241 A given use case, or even a specific success criterion for that use case, might require several rubrics for holistic evaluation.
32423242 </Note>
3243* **Be empirical or specific:** For example, instruct the LLM to output only 'correct' or 'incorrect', or to judge from a scale of 1–5. Purely qualitative evaluations are hard to assess quickly and at scale.
3243* **Be empirical or specific:** For example, instruct the LLM to output only ‘correct’ or ‘incorrect’, or to judge from a scale of 1–5. Purely qualitative evaluations are hard to assess quickly and at scale.
32443244* **Encourage reasoning:** Use a grader model with [thinking](https://platform.claude.com/docs/en/build-with-claude/thinking) on, so that it reasons before it produces an evaluation score. This increases evaluation performance, particularly for tasks requiring complex judgment.
32453245
32463246<Accordion title="Example: LLM-based grading">
No line in this hunk matches that.