Corey Roth [MVP]

On any consumer-facing platform where users can submit content that will be seen by the public, it's a good idea to have some level of content moderation.  Manual content moderation using humans is tedious and labor-intensive.  Luckily, Azure Content Moderator, part of Azure Cognitive Services, provides us with some AI capability to potentially detect language that might be offensive.  It's actually quite easy to get started with as well.

First, you'll want to create a new Content Moderator in your Azure Portal.  The options are fairly straight forward based upon region, resource groups, and scale.  For this example, I used the F0 Plan which comes with 5000 moderation transactions a month (full pricing details).  This should be more than enough to prove out the concept.

When your Content Moderator finishes provisioning, go to the Keys tab and make note of Key 1 and Key 2 here.  You'll use this later.  Keep these keys secure so that others don't use your API calls.

In this example, we are using Ionic Framework 4.0 which just hit general availability fairly recently.  We'll build on a simple out-of-the-box app.  Full source code is available at the link at the end of this post.  If you don't have Ionic Framework installed, you can install it from here.   To start a new project in Ionic Framework, issue the following command.

ionic start IonicAzureContentModerator

After the project has been created, you can see it in the browser by running the following command:

ionic serve

This will launch the app in a web browser with live reload.  Now let's add the components we need to test this out.  First, let's create a service to make our call to Azure.  Normally this service would call our own API or function which would proxy the call to Azure Content Moderator.  We do this so that our API key is not stored in the client application.  For simplicity though, we are going to call Azure directly because this is only an example.  To create the service issue the following command in the Ionic CLI.

ionic g service services/ContentModeratorService

This will create a stub for a service. We'll look at what goes in here shortly.  However, since we are going to be making an Http call, we need to add the angular HttpClientModule.  Open app.module.ts and add the following import:

import { HttpClientModule, HttpClient } from '@angular/common/http';

const httpOptions = {

headers: new HttpHeaders({

'Ocp-Apim-Subscription-Key': Constants.apiKey

})

};

Finally we make the HTTP POST while passing the content that we want to moderate in the body.

Now, we can build a simple form to collect the user's input and make the call to our service.  I add an ion-textarea to collect the user's input and add a toggle to choose whether or not to scan for PII.  Then I just make use of some ion-badge components to display the results.  Here is what our interface looks like in the browser.

Here is the HTML code.

Clicking the button simply executes the moderateContent method in the service we created.  Let's look at the data that comes back when we execute a call. 

In the Content Moderator response, we receive a Classification object and in that object there are three categories returned: Category1, Category2, and Category3 along with a boolean ReviewRecommended which will return true if the text is likely an issue.  The categories are defined as follows:

• Catagory1 - refers to potential presence of language that may be considered sexually explicit or adult in certain situations.
• Category2 - refers to potential presence of language that may be considered sexually suggestive or mature in certain situations.
• Category3 - refers to potential presence of language that may be considered offensive in certain situations.

Each category contains a decimal value between 0 and 1.  The higher it is, the more likely the content applies to that category.  In this case, the phrase we sent is more of a general profanity as opposed to being sexual of nature. 

In our app, we simply bind these values using the ion-note so that you can easily see how the phrase was interpreted.

If you use the toggle for PII, you can look for things such as national id numbers, phone numbers, addresses, and IP addresses.  Here is an example of the output.  You can find more information in the docs.

You can observe the values of potential PII using the PII object.  You'll notice in this case as well that the Classification values were significantly lower because there wasn't any offensive text in the string we sent.

Azure Content Moderator part of Cognitive Services is a great way to test content to see if it's potentially offensive.  While you will still want to incorporate a human review element into any content moderation process. This should help automate some of the process for you.  In addition, Cognitive Services has Content Moderation for images which can identify potentially suggestive images.

If you want to dive deeper, you can take a look at my code sample in GitHub.