Page history
develop-tests
test-and-evaluate/develop-tests
History
test-and-evaluate/develop-tests First recorded · 3326 lines, first recorded
## Define your success criteria ### Common success criteria ## Build evaluations ### Eval design principles ### Example evals ## Grade your evaluations ### Tips for LLM-based grading ## Next steps
The first capture of this source. The page was already there, and this is what it said.
---
title: Define success criteria and build evaluations
url: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
description: Define measurable success criteria for your LLM application and build evaluations to test it, from exact match checks to LLM-based grading.
---
Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them. This cycle is central to prompt engineering.
## Define your success criteria
Good success criteria are:
* **Specific:** Clearly define what you want to achieve. Instead of "good performance," specify "accurate sentiment classification."
* **Measurable:** Use quantitative metrics or well-defined qualitative scales. Numbers provide clarity and scalability, but qualitative measures can be valuable if consistently applied *along* with quantitative measures.
* Even "hazy" topics such as ethics and safety can be quantified:
| | Safety criteria |
| ---- | ------------------------------------------------------------------------------------------ |
| Bad | Safe outputs |
| Good | Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter. |
<Accordion title="Example metrics and measurement methods">
**Quantitative metrics:**
* Task-specific: F1 score, BLEU score, perplexity
* Generic: Accuracy, precision, recall
* Operational: Response time (ms), uptime (%)
**Quantitative methods:**
* A/B testing: Compare performance against a baseline model or earlier version.
* User feedback: Implicit measures like task completion rates.
* Edge case analysis: Percentage of edge cases handled without errors.
**Qualitative scales:**
* Likert scales: "Rate coherence from 1 (nonsensical) to 5 (perfectly logical)"
* Expert rubrics: Linguists rating translation quality on defined criteria
</Accordion>
* **Achievable:** Base your targets on industry benchmarks, prior experiments, AI research, or expert knowledge. Your success metrics should not be unrealistic to current frontier model capabilities.
* **Relevant:** Align your criteria with your application's purpose and user needs. Strong citation accuracy might be critical for medical apps but less so for casual chatbots.
<Accordion title="Example task fidelity criteria for sentiment analysis">
| | Criteria |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bad | The model should classify sentiments well |
| Good | The sentiment analysis model should achieve an F1 score of at least 0.85 (Measurable, Specific) on a held-out test set\* of 10,000 diverse Twitter posts (Relevant), which is a 5% improvement over the current baseline (Achievable). |
\*More on held-out test sets in the next section.
</Accordion>
### Common success criteria
Here are some criteria that might be important for your use case. This list is non-exhaustive.
<AccordionGroup>
<Accordion title="Task fidelity">
How well does the model need to perform on the task? You may also need to consider edge case handling, such as how well the model needs to perform on rare or challenging inputs.
</Accordion>
<Accordion title="Consistency">
How similar do the model's responses need to be for similar types of input? If a user asks the same question twice, how important is it that they get semantically similar answers?
</Accordion>
<Accordion title="Relevance and coherence">
How well does the model directly address the user's questions or instructions? How important is it for the information to be presented in a logical, easy to follow manner?
</Accordion>
<Accordion title="Tone and style">
How well does the model's output style match expectations? How appropriate is its language for the target audience?
</Accordion>
<Accordion title="Privacy preservation">
What is a successful metric for how the model handles personal or sensitive information? Can it follow instructions not to use or share certain details?
</Accordion>
<Accordion title="Context utilization">
How effectively does the model use provided context? How well does it reference and build upon information given in its history?
</Accordion>
<Accordion title="Latency">
What is the acceptable response time for the model? This depends on your application's real-time requirements and user expectations.
</Accordion>
<Accordion title="Price">
What is your budget for running the model? Consider factors like the cost for each API call, the size of the model, and the frequency of usage.
</Accordion>
</AccordionGroup>
Most use cases need multidimensional evaluation along several success criteria.
<Accordion title="Example multidimensional criteria for sentiment analysis">
| | Criteria |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Bad | The model should classify sentiments well |
| Good | On a held-out test set of 10,000 diverse Twitter posts, the sentiment analysis model should achieve: - an F1 score of at least 0.85 - 99.5% of outputs are non-toxic - 90% of errors would cause inconvenience, not egregious error\* - 95% response time \< 200ms |
\*In reality, you would also define what "inconvenience" and "egregious" mean.
</Accordion>
***
## Build evaluations
### Eval design principles
1. **Be task-specific:** Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!
<Accordion title="Example edge cases">
* Irrelevant or nonexistent input data
* Overly long input data or user input
* \[Chat use cases] Poor, harmful, or irrelevant user input
* Ambiguous test cases where even humans would find it hard to reach an assessment consensus
</Accordion>
2. **Automate when possible:** Structure questions to allow for automated grading (for example, multiple-choice, string match, code-graded, LLM-graded).
3. **Prioritize volume over quality:** More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.
### Example evals
<AccordionGroup>
<Accordion title="Task fidelity (sentiment analysis) - exact match evaluation">
**What it measures:** Exact match evals measure whether the model's output matches a predefined correct answer, typically after normalizing whitespace and case. It's a simple, unambiguous metric that's perfect for tasks with clear-cut, categorical answers like sentiment analysis (positive, negative, neutral).
**Example eval test cases:** 1,000 tweets with human-labeled sentiments.
<CodeGroup exclude="shell">
```python Python
tweets = [
{"text": "This movie was a total waste of time. 👎", "sentiment": "negative"},
{"text": "The new album is 🔥! Been on repeat all day.", "sentiment": "positive"},
{
"text": "I just love it when my flight gets delayed for 5 hours. #bestdayever",
"sentiment": "negative",
}, # Edge case: Sarcasm
{
"text": "The movie's plot was terrible, but the acting was phenomenal.",
"sentiment": "mixed",
}, # Edge case: Mixed sentiment
# ... 996 more tweets
]
client = anthropic.Anthropic()
def get_completion(prompt: str):
message = client.messages.create(
model="claude-opus-5",
max_tokens=50,
messages=[{"role": "user", "content": prompt}],
)
return next(block.text for block in message.content if block.type == "text")
def evaluate_exact_match(model_output, correct_answer):
return model_output.strip().lower() == correct_answer.lower()
outputs = [
get_completion(
f"Classify this as 'positive', 'negative', 'neutral', or 'mixed': {tweet['text']}"
)
for tweet in tweets
]
accuracy = sum(
evaluate_exact_match(output, tweet["sentiment"])
for output, tweet in zip(outputs, tweets)
) / len(tweets)
print(f"Sentiment Analysis Accuracy: {accuracy * 100}%")
```
```typescript TypeScript
const tweets = [
{ text: "This movie was a total waste of time. 👎", sentiment: "negative" },
{ text: "The new album is 🔥! Been on repeat all day.", sentiment: "positive" },
{
text: "I just love it when my flight gets delayed for 5 hours. #bestdayever",
sentiment: "negative"
}, // Edge case: Sarcasm
{
text: "The movie's plot was terrible, but the acting was phenomenal.",
sentiment: "mixed"
} // Edge case: Mixed sentiment
// ... 996 more tweets
];
const client = new Anthropic();
async function getCompletion(prompt: string): Promise<string> {
const message = await client.messages.create({
model: "claude-opus-5",
max_tokens: 50,
messages: [{ role: "user", content: prompt }]
});
const textBlock = message.content.find((block) => block.type === "text");
return textBlock ? textBlock.text : "";
}
function evaluateExactMatch(modelOutput: string, correctAnswer: string): boolean {
return modelOutput.trim().toLowerCase() === correctAnswer.toLowerCase();
}
let correctCount = 0;
for (const tweet of tweets) {
const output = await getCompletion(
`Classify this as 'positive', 'negative', 'neutral', or 'mixed': ${tweet.text}`
);
if (evaluateExactMatch(output, tweet.sentiment)) {
correctCount++;
}
}
console.log(`Sentiment Analysis Accuracy: ${(correctCount / tweets.length) * 100}%`);
```
```csharp C#
Tweet[] tweets =
[
new("This movie was a total waste of time. 👎", "negative"),
new("The new album is 🔥! Been on repeat all day.", "positive"),
// Edge case: Sarcasm
new("I just love it when my flight gets delayed for 5 hours. #bestdayever", "negative"),
// Edge case: Mixed sentiment
new("The movie's plot was terrible, but the acting was phenomenal.", "mixed"),
// ... 996 more tweets
];
var client = new AnthropicClient();
async Task<string> GetCompletion(string prompt)
{
var message = await client.Messages.Create(new MessageCreateParams
{
Model = Model.ClaudeOpus5,
MaxTokens = 50,
Messages = [new() { Role = Role.User, Content = prompt }],
});
return ContentText(message);
}
bool EvaluateExactMatch(string modelOutput, string correctAnswer)
{
return string.Equals(modelOutput.Trim(), correctAnswer, StringComparison.OrdinalIgnoreCase);
}
string ContentText(Message message)
{
var text = "";
foreach (var block in message.Content)
{
if (block.TryPickText(out var textBlock))
{
text += textBlock.Text;
}
}
return text;
}
var correct = 0;
foreach (var tweet in tweets)
{
var output = await GetCompletion(
$"Classify this as 'positive', 'negative', 'neutral', or 'mixed': {tweet.Text}");
if (EvaluateExactMatch(output, tweet.Sentiment))
{
correct++;
}
}
Console.WriteLine($"Sentiment Analysis Accuracy: {100.0 * correct / tweets.Length}%");
record Tweet(string Text, string Sentiment);
```
```go Go
var client = anthropic.NewClient()
func contentText(message *anthropic.Message) string {
var text strings.Builder
for _, block := range message.Content {
if textBlock, ok := block.AsAny().(anthropic.TextBlock); ok {
text.WriteString(textBlock.Text)
}
}
return text.String()
}
type tweet struct {
Text string
Sentiment string
}
var tweets = []tweet{
{"This movie was a total waste of time. 👎", "negative"},
{"The new album is 🔥! Been on repeat all day.", "positive"},
// Edge case: Sarcasm
{"I just love it when my flight gets delayed for 5 hours. #bestdayever", "negative"},
// Edge case: Mixed sentiment
Cut at 300 lines.