content-moderation changedabout-claude/use-case-guides/content-moderation
Nearest release: v2.1.312, published under an hour before this site recorded the change. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Recorded here
Lines+10added
Lines−10removed
From line
7
where the diff opens
First seen
14 Aug 2026
this site's first read of the page
Recorded edits12to this page, all time
The whole hunk
from line 7, old and new numbered
/
from line 7
77> Visit the [content moderation recipe](https://platform.claude.com/cookbook/misc-building-moderation-filter) in the Claude Cookbook to see an example content moderation implementation using Claude.
88
99<Tip>
10 This guide is focused on moderating user-generated content within your application. If you're looking for guidance on moderating interactions with Claude, refer to
10 This guide is focused on moderating user-generated content within your application. If you’re looking for guidance on moderating interactions with Claude, refer to
1111
1212 [Mitigate jailbreaks and prompt injections](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks)
1313
from line 46
4646 </Accordion>
4747
4848 <Accordion title="You need multimodal support">
49 Claude's multimodal capabilities allow it to analyze and interpret content across both text and images. This makes it a versatile tool for comprehensive content moderation in environments where different media types need to be evaluated together.
49 Claude’s multimodal capabilities allow it to analyze and interpret content across both text and images. This makes it a versatile tool for comprehensive content moderation in environments where different media types need to be evaluated together.
5050 </Accordion>
5151</AccordionGroup>
5252
from line 310
310310 ```
311311</CodeGroup>
312312
313Effectively moderating these examples requires a nuanced understanding of language. In the comment, `This movie was great, I really enjoyed it. The main actor really killed it!`, the content moderation system needs to recognize that "killed it" is a metaphor, not an indication of actual violence. Conversely, despite the lack of explicit mentions of violence, the comment `Delete this post now or you better hide. I am coming after you and your family.` should be flagged by the content moderation system.
313Effectively moderating these examples requires a nuanced understanding of language. In the comment, `This movie was great, I really enjoyed it. The main actor really killed it!`, the content moderation system needs to recognize that “killed it” is a metaphor, not an indication of actual violence. Conversely, despite the lack of explicit mentions of violence, the comment `Delete this post now or you better hide. I am coming after you and your family.` should be flagged by the content moderation system.
314314
315The unsafe categories can be customized to fit your specific needs. For example, if you want to prevent minors from creating content on your website, you could add "Underage Posting" to the categories.
315The unsafe categories can be customized to fit your specific needs. For example, if you want to prevent minors from creating content on your website, you could add “Underage Posting” to the categories.
316316
317317***
318318
from line 873
873873
874874In this example, the `moderate_message` function contains an assessment prompt that includes the unsafe content categories and the message to evaluate. The prompt asks Claude to assess whether the message should be moderated, based on the unsafe categories you defined earlier.
875875
876The model's assessment is then parsed to determine if there is a violation. If there is a violation, Claude also returns a list of violated categories and an explanation as to why the message is unsafe.
876The model’s assessment is then parsed to determine if there is a violation. If there is a violation, Claude also returns a list of violated categories and an explanation as to why the message is unsafe.
877877
878878### Evaluate your prompt
879879
880Content moderation is a classification problem. Thus, you can use the same techniques outlined in the Claude Cookbook's [classification recipe](https://platform.claude.com/cookbook/capabilities-classification-guide) to determine the accuracy of your content moderation system.
880Content moderation is a classification problem. Thus, you can use the same techniques outlined in the Claude Cookbook’s [classification recipe](https://platform.claude.com/cookbook/capabilities-classification-guide) to determine the accuracy of your content moderation system.
881881
882882One additional consideration is that instead of treating content moderation as a binary classification problem, you can create multiple categories to represent various risk levels. Creating multiple risk levels allows you to adjust the aggressiveness of your moderation. For example, you might want to automatically block user queries that are deemed high risk, whereas users with many medium risk queries are flagged for human review.
883883
from line 1433
14331433
14341434### Deploy your prompt
14351435
1436Once you are confident in the quality of your solution, it's time to deploy it to production. Here are some best practices to follow when using content moderation in production:
1436Once you are confident in the quality of your solution, it’s time to deploy it to production. Here are some best practices to follow when using content moderation in production:
14371437
143814381. **Provide clear feedback to users:** When user input is blocked or a response is flagged because of content moderation, provide informative and constructive feedback to help users understand why their message was flagged and how they can rephrase it appropriately. In the earlier coding examples, this is done through the `explanation` field in the Claude response.
14391439
from line 2190
21902190
21912191### Consider batch processing
21922192
2193To reduce costs in situations where real-time moderation isn't necessary, consider moderating messages in batches. Include multiple messages within the prompt's context, and ask Claude to assess which messages should be moderated.
2193To reduce costs in situations where real-time moderation isn’t necessary, consider moderating messages in batches. Include multiple messages within the prompt’s context, and ask Claude to assess which messages should be moderated.
21942194
21952195<CodeGroup exclude="shell">
21962196 ```python Python
from line 2728
27282728 ```
27292729</CodeGroup>
27302730
2731In this example, the `batch_moderate_messages` function handles the moderation of an entire batch of messages with a single Claude API call. Inside the function, a prompt is created that includes the list of messages to evaluate and the unsafe content categories. The prompt directs Claude to return a JSON object listing all messages that contain violations. Each message in the response is identified by its `id`, which corresponds to the message's position in the batch. Keep in mind that finding the optimal batch size for your specific needs might require some experimentation. Although larger batch sizes can lower costs, they might also lead to a slight decrease in quality. Additionally, you might need to increase the `max_tokens` parameter in the Claude API call to accommodate longer responses. For details on the maximum number of tokens your chosen model can output, refer to the [model comparison table](https://platform.claude.com/docs/en/models/overview#latest-models-comparison).
2731In this example, the `batch_moderate_messages` function handles the moderation of an entire batch of messages with a single Claude API call. Inside the function, a prompt is created that includes the list of messages to evaluate and the unsafe content categories. The prompt directs Claude to return a JSON object listing all messages that contain violations. Each message in the response is identified by its `id`, which corresponds to the message’s position in the batch. Keep in mind that finding the optimal batch size for your specific needs might require some experimentation. Although larger batch sizes can lower costs, they might also lead to a slight decrease in quality. Additionally, you might need to increase the `max_tokens` parameter in the Claude API call to accommodate longer responses. For details on the maximum number of tokens your chosen model can output, refer to the [model comparison table](https://platform.claude.com/docs/en/models/overview#latest-models-comparison).
27322732
27332733<CardGroup cols={2}>
27342734 <Card title="Content moderation recipe" icon="link" href="https://platform.claude.com/cookbook/misc-building-moderation-filter">
No line in this hunk matches that.