content-moderation

about-claude/use-case-guides/content-moderation First recorded · 2747 lines, first recorded

## Before building with Claude ### Decide whether to use Claude for content moderation ### Generate examples of content to moderate ## How to moderate content using Claude ### Select the right Claude model ### Build a strong prompt ### Evaluate your prompt ### Deploy your prompt ## Improve performance ### Define topics and provide examples ### Consider batch processing

The first capture of this source. The page was already there, and this is what it said.

---
title: Content moderation
url: https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation
description: Content moderation is a critical aspect of maintaining a safe, respectful, and productive environment in digital applications. This guide discusses how Claude can be used to moderate content within your digital application.
---

> Visit the [content moderation cookbook](https://platform.claude.com/cookbook/misc-building-moderation-filter) to see an example content moderation implementation using Claude.

<Tip>
  This guide is focused on moderating user-generated content within your application. If you're looking for guidance on moderating interactions with Claude, refer to 

  [Mitigate jailbreaks and prompt injections](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks)

  .
</Tip>

## Before building with Claude

### Decide whether to use Claude for content moderation

Here are some key indicators that you should use an LLM like Claude instead of a traditional ML or rules-based approach for content moderation:

<AccordionGroup>
  <Accordion title="You want a cost-effective and rapid implementation">
    Traditional ML methods require significant engineering resources, ML expertise, and infrastructure costs. Human moderation systems incur even higher costs. With Claude, you can have a sophisticated moderation system operational in significantly less time and at a much lower cost.
  </Accordion>

  <Accordion title="You want both semantic understanding and quick decisions">
    Traditional ML approaches, such as bag-of-words models or simple pattern matching, often struggle to understand the tone, intent, and context of the content. While human moderation systems excel at understanding semantic meaning, they require time for content to be reviewed. Claude addresses both needs by combining semantic understanding with the ability to deliver moderation decisions quickly.
  </Accordion>

  <Accordion title="You need consistent policy decisions">
    By leveraging its advanced reasoning capabilities, Claude can interpret and apply complex moderation guidelines uniformly. This consistency helps ensure fair treatment of all content, reducing the risk of inconsistent or biased moderation decisions that can undermine user trust.
  </Accordion>

  <Accordion title="Your moderation policies are likely to change or evolve over time">
    Once a traditional ML approach has been established, changing it is a laborious and data-intensive undertaking. On the other hand, as your product or customer needs evolve, Claude can easily adapt to changes or additions to moderation policies without extensive relabeling of training data.
  </Accordion>

  <Accordion title="You require interpretable reasoning for your moderation decisions">
    If you want to provide users or regulators with clear explanations behind moderation decisions, Claude can generate detailed and coherent justifications. This transparency is important for building trust and ensuring accountability in content moderation practices.
  </Accordion>

  <Accordion title="You need multilingual support without maintaining separate models">
    Traditional ML approaches typically require separate models or extensive translation processes for each supported language. Human moderation requires hiring a workforce fluent in each supported language. Claude’s multilingual capabilities allow it to classify tickets in various languages without the need for separate models or extensive translation processes, streamlining moderation for global customer bases.
  </Accordion>

  <Accordion title="You require multimodal support">
    Claude's multimodal capabilities allow it to analyze and interpret content across both text and images. This makes it a versatile tool for comprehensive content moderation in environments where different media types need to be evaluated together.
  </Accordion>
</AccordionGroup>

<Note>
  All Claude models are trained with built-in safety behaviors. This may result in Claude moderating content deemed particularly dangerous (in line with the 

  [Acceptable Use Policy](https://www.anthropic.com/legal/aup)

  ), regardless of the prompt used. For example, an adult website that wants to allow users to post explicit sexual content may find that Claude still flags explicit content as requiring moderation, even if they specify in their prompt not to moderate explicit sexual content. Consider reviewing the AUP in advance of building a moderation solution.
</Note>

### Generate examples of content to moderate

Before developing a content moderation solution, first create examples of content that should be flagged and content that should not be flagged. Ensure that you include edge cases and challenging scenarios that may be difficult for a content moderation system to handle effectively. Afterward, review your examples to create a well-defined list of moderation categories. For instance, the examples generated by a social media platform might include the following:

<CodeGroup exclude="shell">
  ```python Python
  client = anthropic.Anthropic()

  allowed_user_comments = [
      "This movie was great, I really enjoyed it. The main actor really killed it!",
      "I hate Mondays.",
      "It is a great time to invest in gold!",
  ]

  disallowed_user_comments = [
      "Delete this post now or you better hide. I am coming after you and your family.",
      "Stay away from the 5G cellphones!! They are using 5G to control you.",
      "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
  ]

  # Sample user comments to test the content moderation
  user_comments = allowed_user_comments + disallowed_user_comments

  # Categories considered unsafe for content moderation
  unsafe_categories = [
      "Child Exploitation",
      "Conspiracy Theories",
      "Hate",
      "Indiscriminate Weapons",
      "Intellectual Property",
      "Non-Violent Crimes",
      "Privacy",
      "Self-Harm",
      "Sex Crimes",
      "Sexual Content",
      "Specialized Advice",
      "Violent Crimes",
  ]
  ```

  ```typescript TypeScript
  const client = new Anthropic();

  const allowedUserComments = [
    "This movie was great, I really enjoyed it. The main actor really killed it!",
    "I hate Mondays.",
    "It is a great time to invest in gold!"
  ];

  const disallowedUserComments = [
    "Delete this post now or you better hide. I am coming after you and your family.",
    "Stay away from the 5G cellphones!! They are using 5G to control you.",
    "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
  ];

  // Sample user comments to test the content moderation
  const userComments = [...allowedUserComments, ...disallowedUserComments];

  // Categories considered unsafe for content moderation
  const unsafeCategories = [
    "Child Exploitation",
    "Conspiracy Theories",
    "Hate",
    "Indiscriminate Weapons",
    "Intellectual Property",
    "Non-Violent Crimes",
    "Privacy",
    "Self-Harm",
    "Sex Crimes",
    "Sexual Content",
    "Specialized Advice",
    "Violent Crimes"
  ];
  ```

  ```csharp C#
  var client = new AnthropicClient();

  string[] allowedUserComments =
  [
      "This movie was great, I really enjoyed it. The main actor really killed it!",
      "I hate Mondays.",
      "It is a great time to invest in gold!",
  ];

  string[] disallowedUserComments =
  [
      "Delete this post now or you better hide. I am coming after you and your family.",
      "Stay away from the 5G cellphones!! They are using 5G to control you.",
      "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
  ];

  // Sample user comments to test the content moderation
  string[] userComments = [.. allowedUserComments, .. disallowedUserComments];

  // Categories considered unsafe for content moderation
  string[] unsafeCategories =
  [
      "Child Exploitation",
      "Conspiracy Theories",
      "Hate",
      "Indiscriminate Weapons",
      "Intellectual Property",
      "Non-Violent Crimes",
      "Privacy",
      "Self-Harm",
      "Sex Crimes",
      "Sexual Content",
      "Specialized Advice",
      "Violent Crimes",
  ];
  ```

  ```go Go
  var client = anthropic.NewClient()

  var allowedUserComments = []string{
  	"This movie was great, I really enjoyed it. The main actor really killed it!",
  	"I hate Mondays.",
  	"It is a great time to invest in gold!",
  }

  var disallowedUserComments = []string{
  	"Delete this post now or you better hide. I am coming after you and your family.",
  	"Stay away from the 5G cellphones!! They are using 5G to control you.",
  	"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
  }

  // Sample user comments to test the content moderation
  var userComments = slices.Concat(allowedUserComments, disallowedUserComments)

  // Categories considered unsafe for content moderation
  var unsafeCategories = []string{
  	"Child Exploitation",
  	"Conspiracy Theories",
  	"Hate",
  	"Indiscriminate Weapons",
  	"Intellectual Property",
  	"Non-Violent Crimes",
  	"Privacy",
  	"Self-Harm",
  	"Sex Crimes",
  	"Sexual Content",
  	"Specialized Advice",
  	"Violent Crimes",
  }

  ```

  ```java Java
  final AnthropicClient client = AnthropicOkHttpClient.fromEnv();

  final List<String> allowedUserComments = List.of(
          "This movie was great, I really enjoyed it. The main actor really killed it!",
          "I hate Mondays.",
          "It is a great time to invest in gold!");

  final List<String> disallowedUserComments = List.of(
          "Delete this post now or you better hide. I am coming after you and your family.",
          "Stay away from the 5G cellphones!! They are using 5G to control you.",
          "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!");

  // Sample user comments to test the content moderation
  final List<String> userComments =
          Stream.concat(allowedUserComments.stream(), disallowedUserComments.stream()).toList();

  // Categories considered unsafe for content moderation
  final List<String> unsafeCategories = List.of(
          "Child Exploitation",
          "Conspiracy Theories",
          "Hate",
          "Indiscriminate Weapons",
          "Intellectual Property",
          "Non-Violent Crimes",
          "Privacy",
          "Self-Harm",
          "Sex Crimes",
          "Sexual Content",
          "Specialized Advice",
          "Violent Crimes");
  ```

  ```php PHP
  $client = new Client();

  $allowedUserComments = [
      'This movie was great, I really enjoyed it. The main actor really killed it!',
      'I hate Mondays.',
      'It is a great time to invest in gold!',
  ];

  $disallowedUserComments = [
      'Delete this post now or you better hide. I am coming after you and your family.',
      'Stay away from the 5G cellphones!! They are using 5G to control you.',
      'Congratulations! You have won a $1,000 gift card. Click here to claim your prize!',
  ];

  // Sample user comments to test the content moderation
  $userComments = [...$allowedUserComments, ...$disallowedUserComments];

  // Categories considered unsafe for content moderation
  $unsafeCategories = [
      'Child Exploitation',
      'Conspiracy Theories',
      'Hate',
      'Indiscriminate Weapons',
      'Intellectual Property',
      'Non-Violent Crimes',
      'Privacy',
      'Self-Harm',
      'Sex Crimes',
      'Sexual Content',
      'Specialized Advice',
      'Violent Crimes',
  ];
  ```

  ```ruby Ruby
  CLIENT = Anthropic::Client.new

  ALLOWED_USER_COMMENTS = [
    "This movie was great, I really enjoyed it. The main actor really killed it!",
    "I hate Mondays.",
    "It is a great time to invest in gold!"
  ]

  DISALLOWED_USER_COMMENTS = [
    "Delete this post now or you better hide. I am coming after you and your family.",
    "Stay away from the 5G cellphones!! They are using 5G to control you.",
    "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
  ]

  # Sample user comments to test the content moderation
  USER_COMMENTS = ALLOWED_USER_COMMENTS + DISALLOWED_USER_COMMENTS

  # Categories considered unsafe for content moderation
  UNSAFE_CATEGORIES = [
    "Child Exploitation",
    "Conspiracy Theories",
    "Hate",

Cut at 300 lines.