develop-tests changedtest-and-evaluate/develop-tests
Nearest release: v2.1.308, published 2 hours after this site recorded the change. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Recorded here
Lines+11added
Lines−11removed
From line
98
where the diff opens
First seen
14 Aug 2026
this site's first read of the page
Recorded edits11to this page, all time
The whole hunk
from line 98, old and new numbered
/
from line 98
9898Most use cases need multidimensional evaluation along several success criteria.
9999
100100<Accordion title="Example multidimensional criteria for sentiment analysis">
101 | | Criteria |
102 | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
103 | Bad | The model should classify sentiments well |
104 | Good | On a held-out test set of 10,000 diverse Twitter posts, the sentiment analysis model should achieve:<br />- an F1 score of at least 0.85<br />- 99.5% of outputs are non-toxic<br />- 90% of errors would cause inconvenience, not egregious error\*<br />- 95% response time \< 200ms |
101 | | Criteria |
102 | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
103 | Bad | The model should classify sentiments well |
104 | Good | On a held-out test set of 10,000 diverse Twitter posts, the sentiment analysis model should achieve:<br />- an F1 score of at least 0.85<br />- 99.5% non-toxic outputs<br />- 90% of errors would cause inconvenience, not egregious error\*<br />- 95% response time \< 200ms |
105105
106106 \*In reality, you would also define what "inconvenience" and "egregious" mean.
107107</Accordion>
from line 125
125125### Example evals
126126
127127<AccordionGroup>
128 <Accordion title="Task fidelity (sentiment analysis) - exact match evaluation">
128 <Accordion title="Task fidelity (sentiment analysis): exact match evaluation">
129129 **What it measures:** Exact match evals measure whether the model's output matches a predefined correct answer, typically after normalizing whitespace and case. It's a simple, unambiguous metric that's perfect for tasks with clear-cut, categorical answers such as sentiment analysis (positive, negative, neutral).
130130
131131 **Example eval test cases:** 1,000 tweets with human-labeled sentiments.
from line 487
487487 </CodeGroup>
488488 </Accordion>
489489
490 <Accordion title="Consistency (FAQ bot) - cosine similarity evaluation">
490 <Accordion title="Consistency (FAQ bot): cosine similarity evaluation">
491491 **What it measures:** Cosine similarity measures the similarity between two vectors (in this case, sentence embeddings of the model's output using [Sentence-BERT (SBERT)](https://sbert.net/)) by computing the cosine of the angle between them. Values closer to 1 indicate higher similarity. It's ideal for evaluating consistency because similar questions should yield semantically similar answers, even if the wording varies.
492492
493493 **Example eval test cases:** 50 groups with a few paraphrased versions each.
from line 1033
10331033 </CodeGroup>
10341034 </Accordion>
10351035
1036 <Accordion title="Relevance and coherence (summarization) - ROUGE-L evaluation">
1036 <Accordion title="Relevance and coherence (summarization): ROUGE-L evaluation">
10371037 **What it measures:** ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence) evaluates the quality of generated summaries. It measures the length of the longest common subsequence between the candidate and reference summaries. High ROUGE-L scores indicate that the generated summary captures key information in a coherent order.
10381038
10391039 **Example eval test cases:** 200 articles with reference summaries.
from line 1560
15601560 </CodeGroup>
15611561 </Accordion>
15621562
1563 <Accordion title="Tone and style (customer service) - LLM-based Likert scale">
1563 <Accordion title="Tone and style (customer service): LLM-based Likert scale">
15641564 **What it measures:** The LLM-based Likert scale is a psychometric scale that uses an LLM to judge subjective attitudes or perceptions. Here, it's used to rate the tone of responses on a scale from 1 to 5. It's ideal for evaluating nuanced aspects such as empathy, professionalism, or patience that are difficult to quantify with traditional metrics.
15651565
15661566 **Example eval test cases:** 100 customer inquiries with target tone (empathetic, patient, professional).
from line 2031
20312031 </CodeGroup>
20322032 </Accordion>
20332033
2034 <Accordion title="Privacy preservation (medical chatbot) - LLM-based binary classification">
2034 <Accordion title="Privacy preservation (medical chatbot): LLM-based binary classification">
20352035 **What it measures:** Binary classification determines if an input belongs to one of two classes. Here, it's used to classify whether a response contains PHI. This method can account for context and identify subtle or implicit forms of PHI that rule-based systems might miss.
20362036
20372037 **Example eval test cases:** 500 simulated patient queries, some with PHI.
from line 2584
25842584 </CodeGroup>
25852585 </Accordion>
25862586
2587 <Accordion title="Context utilization (conversation assistant) - LLM-based ordinal scale">
2587 <Accordion title="Context utilization (conversation assistant): LLM-based ordinal scale">
25882588 **What it measures:** Similar to the Likert scale, the ordinal scale measures on a fixed, ordered scale (1–5). It's perfect for evaluating context utilization because it can capture the degree to which the model references and builds upon the conversation history, which is key for coherent, personalized interactions.
25892589
25902590 **Example eval test cases:** 100 multi-turn conversations with context-dependent questions.
from line 3240
32403240 <Note>
32413241 A given use case, or even a specific success criterion for that use case, might require several rubrics for holistic evaluation.
32423242 </Note>
3243* **Empirical or specific:** For example, instruct the LLM to output only 'correct' or 'incorrect', or to judge from a scale of 1–5. Purely qualitative evaluations are hard to assess quickly and at scale.
3243* **Be empirical or specific:** For example, instruct the LLM to output only 'correct' or 'incorrect', or to judge from a scale of 1–5. Purely qualitative evaluations are hard to assess quickly and at scale.
32443244* **Encourage reasoning:** Use a grader model with [thinking](https://platform.claude.com/docs/en/build-with-claude/thinking) on, so that it reasons before it produces an evaluation score. This increases evaluation performance, particularly for tasks requiring complex judgment.
32453245
32463246<Accordion title="Example: LLM-based grading">
No line in this hunk matches that.