content-moderation changedabout-claude/use-case-guides/content-moderation
Nearest release: v2.1.307, published an hour after this site recorded the change. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Recorded here
Lines+21added
Lines−21removed
From line
4
where the diff opens
First seen
14 Aug 2026
this site's first read of the page
Recorded edits10to this page, all time
The whole hunk
from line 4, old and new numbered
/
from line 4
44description: Content moderation is a critical aspect of maintaining a safe, respectful, and productive environment in digital applications. This guide discusses how Claude can be used to moderate content within your digital application.
55---
66
7> Visit the [content moderation cookbook](https://platform.claude.com/cookbook/misc-building-moderation-filter) to see an example content moderation implementation using Claude.
7> Visit the [content moderation recipe](https://platform.claude.com/cookbook/misc-building-moderation-filter) in the Claude Cookbook to see an example content moderation implementation using Claude.
88
99<Tip>
1010 This guide is focused on moderating user-generated content within your application. If you're looking for guidance on moderating interactions with Claude, refer to
from line 18
1818
1919### Decide whether to use Claude for content moderation
2020
21Here are some key indicators that you should use an LLM like Claude instead of a traditional ML or rules-based approach for content moderation:
21Here are some key indicators that you should use an LLM such as Claude instead of a traditional ML or rules-based approach for content moderation:
2222
2323<AccordionGroup>
2424 <Accordion title="You want a cost-effective and rapid implementation">
from line 26
2626 </Accordion>
2727
2828 <Accordion title="You want both semantic understanding and quick decisions">
29 Traditional ML approaches, such as bag-of-words models or simple pattern matching, often struggle to understand the tone, intent, and context of the content. While human moderation systems excel at understanding semantic meaning, they require time for content to be reviewed. Claude addresses both needs by combining semantic understanding with the ability to deliver moderation decisions quickly.
29 Traditional ML approaches, such as bag-of-words models or simple pattern matching, often have difficulty capturing the tone, intent, and context of the content. Although human moderation systems excel at understanding semantic meaning, they require time for content to be reviewed. Claude addresses both needs by combining semantic understanding with the ability to deliver moderation decisions quickly.
3030 </Accordion>
3131
3232 <Accordion title="You need consistent policy decisions">
33 By leveraging its advanced reasoning capabilities, Claude can interpret and apply complex moderation guidelines uniformly. This consistency helps ensure fair treatment of all content, reducing the risk of inconsistent or biased moderation decisions that can undermine user trust.
33 With its advanced reasoning capabilities, Claude can interpret and apply complex moderation guidelines uniformly. This consistency helps ensure fair treatment of all content, reducing the risk of inconsistent or biased moderation decisions that can undermine user trust.
3434 </Accordion>
3535
3636 <Accordion title="Your moderation policies are likely to change or evolve over time">
from line 42
4242 </Accordion>
4343
4444 <Accordion title="You need multilingual support without maintaining separate models">
45 Traditional ML approaches typically require separate models or extensive translation processes for each supported language. Human moderation requires hiring a workforce fluent in each supported language. Claude’s multilingual capabilities allow it to classify tickets in various languages without the need for separate models or extensive translation processes, streamlining moderation for global customer bases.
45 Traditional ML approaches typically require separate models or extensive translation processes for each supported language. Human moderation requires hiring a workforce fluent in each supported language. Claude’s multilingual capabilities allow it to classify content in various languages without the need for separate models or extensive translation processes, streamlining moderation for global customer bases.
4646 </Accordion>
4747
4848 <Accordion title="You require multimodal support">
from line 51
5151</AccordionGroup>
5252
5353<Note>
54 All Claude models are trained with built-in safety behaviors. This may result in Claude moderating content deemed particularly dangerous (in line with the
54 All Claude models are trained with built-in safety behaviors. This might result in Claude moderating content deemed particularly dangerous (in line with the
5555
56 [Acceptable Use Policy](https://www.anthropic.com/legal/aup)
56 [Usage Policy](https://www.anthropic.com/legal/aup)
5757
58 ), regardless of the prompt used. For example, an adult website that wants to allow users to post explicit sexual content may find that Claude still flags explicit content as requiring moderation, even if they specify in their prompt not to moderate explicit sexual content. Consider reviewing the AUP in advance of building a moderation solution.
58 ), regardless of the prompt used. For example, an adult website that wants to allow users to post explicit sexual content might find that Claude still flags explicit content as requiring moderation, even if they specify in their prompt not to moderate explicit sexual content. Consider reviewing the Usage Policy in advance of building a moderation solution.
5959</Note>
6060
6161### Generate examples of content to moderate
6262
63Before developing a content moderation solution, first create examples of content that should be flagged and content that should not be flagged. Ensure that you include edge cases and challenging scenarios that may be difficult for a content moderation system to handle effectively. Afterward, review your examples to create a well-defined list of moderation categories. For instance, the examples generated by a social media platform might include the following:
63Before developing a content moderation solution, first create examples of content that should be flagged and content that should not be flagged. Ensure that you include edge cases and challenging scenarios that might be difficult for a content moderation system to handle effectively. Afterward, review your examples to create a well-defined list of moderation categories. For instance, the examples generated by a social media platform might include the following:
6464
6565<CodeGroup exclude="shell">
6666 ```python Python
from line 320
320320
321321### Select the right Claude model
322322
323When selecting a model, it’s important to consider the size of your data. If costs are a concern, a smaller model such as Claude Haiku 5.5 is an excellent choice because of its cost-effectiveness. The following is an estimate of the cost to moderate text for a social media platform that receives one billion posts per month:
323When selecting a model, it’s important to consider the size of your data. If costs are a concern, a smaller model such as Claude Haiku 5.5 is an excellent choice because of its cost-effectiveness. The following is an estimate of the cost to moderate text for a social media platform that receives 1 billion posts per month:
324324
325325* **Content size**
326326
from line 354
354354 * Monthly cost: $143,000 + $37,500 = $180,500 USD
355355
356356<Tip>
357 Actual costs may differ from these estimates. These estimates are based on the prompt highlighted in the section on
357 Actual costs might differ from these estimates. These estimates are based on the prompt highlighted in the section on
358358
359359 [batch processing](https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation#consider-batch-processing)
360360
from line 367
367367
368368### Build a strong prompt
369369
370To use Claude for content moderation, Claude must understand the moderation requirements of your application. Start by writing a prompt that allows you to define your moderation needs:
370To use Claude for content moderation, Claude must have information about the moderation requirements of your application. Start by writing a prompt that allows you to define your moderation needs:
371371
372372<CodeGroup exclude="shell">
373373 ```python Python
from line 877
877877
878878### Evaluate your prompt
879879
880Content moderation is a classification problem. Thus, you can use the same techniques outlined in the [classification cookbook](https://platform.claude.com/cookbook/capabilities-classification-guide) to determine the accuracy of your content moderation system.
880Content moderation is a classification problem. Thus, you can use the same techniques outlined in the Claude Cookbook's [classification recipe](https://platform.claude.com/cookbook/capabilities-classification-guide) to determine the accuracy of your content moderation system.
881881
882One additional consideration is that instead of treating content moderation as a binary classification problem, you may instead create multiple categories to represent various risk levels. Creating multiple risk levels allows you to adjust the aggressiveness of your moderation. For example, you might want to automatically block user queries that are deemed high risk, while users with many medium risk queries are flagged for human review.
882One additional consideration is that instead of treating content moderation as a binary classification problem, you can instead create multiple categories to represent various risk levels. Creating multiple risk levels allows you to adjust the aggressiveness of your moderation. For example, you might want to automatically block user queries that are deemed high risk, whereas users with many medium risk queries are flagged for human review.
883883
884884<CodeGroup exclude="shell">
885885 ```python Python
from line 1429
14291429
14301430Within the function, a prompt is generated for Claude, including the message to be assessed, the unsafe categories, and specific instructions for evaluating the risk level. The prompt instructs Claude to respond with a JSON object that includes the risk level, the violated categories, and an optional explanation.
14311431
1432This approach enables flexible content moderation by assigning risk levels. It can be seamlessly integrated into a larger system to automate content filtering or flag comments for human review based on their assessed risk level. For instance, when running this code, the comment `Delete this post now or you better hide. I am coming after you and your family.` is identified as high risk because of its dangerous threat. Conversely, the comment `Stay away from the 5G cellphones!! They are using 5G to control you.` is categorized as medium risk.
1432This approach allows flexible content moderation by assigning risk levels. It can be seamlessly integrated into a larger system to automate content filtering or flag comments for human review based on their assessed risk level. For instance, when you run this code, the comment `Delete this post now or you better hide. I am coming after you and your family.` is identified as high risk because of its dangerous threat. Conversely, the comment `Stay away from the 5G cellphones!! They are using 5G to control you.` is categorized as medium risk.
14331433
14341434### Deploy your prompt
14351435
from line 1445
14451445
14461446## Improve performance
14471447
1448In complex scenarios, it may be helpful to consider additional strategies to improve performance beyond standard [prompt engineering techniques](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview). Here are some advanced strategies:
1448In complex scenarios, it might be helpful to consider additional strategies to improve performance beyond standard [prompt engineering techniques](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview). Here are some advanced strategies:
14491449
14501450### Define topics and provide examples
14511451
from line 2728
27282728 ```
27292729</CodeGroup>
27302730
2731In this example, the `batch_moderate_messages` function handles the moderation of an entire batch of messages with a single Claude API call. Inside the function, a prompt is created that includes the list of messages to evaluate and the unsafe content categories. The prompt directs Claude to return a JSON object listing all messages that contain violations. Each message in the response is identified by its `id`, which corresponds to the message's position in the batch. Keep in mind that finding the optimal batch size for your specific needs may require some experimentation. While larger batch sizes can lower costs, they might also lead to a slight decrease in quality. Additionally, you may need to increase the `max_tokens` parameter in the Claude API call to accommodate longer responses. For details on the maximum number of tokens your chosen model can output, refer to the [model comparison table](https://platform.claude.com/docs/en/models/overview#latest-models-comparison).
2731In this example, the `batch_moderate_messages` function handles the moderation of an entire batch of messages with a single Claude API call. Inside the function, a prompt is created that includes the list of messages to evaluate and the unsafe content categories. The prompt directs Claude to return a JSON object listing all messages that contain violations. Each message in the response is identified by its `id`, which corresponds to the message's position in the batch. Keep in mind that finding the optimal batch size for your specific needs might require some experimentation. Although larger batch sizes can lower costs, they might also lead to a slight decrease in quality. Additionally, you might need to increase the `max_tokens` parameter in the Claude API call to accommodate longer responses. For details on the maximum number of tokens your chosen model can output, refer to the [model comparison table](https://platform.claude.com/docs/en/models/overview#latest-models-comparison).
27322732
27332733<CardGroup cols={2}>
2734 <Card title="Content moderation cookbook" icon="link" href="https://platform.claude.com/cookbook/misc-building-moderation-filter">
2734 <Card title="Content moderation recipe" icon="link" href="https://platform.claude.com/cookbook/misc-building-moderation-filter">
27352735 View a fully implemented code-based example of how to use Claude for content moderation.
27362736 </Card>
27372737
No line in this hunk matches that.