Last year we talked about (gasp!) the dirty little secret that no one talks about: you either die an MVP or live long enough to build a content moderation system. You liked that blog post so much that it became one of our most popular posts ever. Deft writing plus Dark Knight references — what’s not to like? However, it may also be that you, too, have content moderation concerns. Fear not; in this follow-up post, we will show you how to implement a content moderation system into your application in five minutes…without code…Seriously. 😮
Dealing with abusive users and thinking about content moderation systems may be the last thing you want to spend your time on, but it is a necessary evil. Bad actors are everywhere, and it’s only a matter of time before they find their way to your rapidly growing product.
We on the partnerships team at Mux want to make it easier to add Artificial Intelligence (AI) powered content moderation to your application. Our partner AVflow makes it simple to build a low-code workflow that adds content analysis and safety detection from Hive to your Mux video processing and streaming.
We will use Hive to analyze the videos, AVflow to build out the logic of the workflow using their drag-and-drop GUI, and Mux to ingest, encode, store, and deliver your videos at scale, to all devices, and with the highest quality.
Once you have these accounts, the easiest way to get started is to:
Here’s how it works:
This guide assumes that you’ve already built the means for end-users to upload a video through your website or application (i.e., user-generated content) and that you are storing this uploaded video in S3. Once the videos are stored, these are the steps that will follow:
AVflow lets you drag and drop APIs, hooks, and services into your desired, ordered flow.
Add the S3 icon to set the Flow trigger:
You can also use a webhook from your app to trigger the Flow:
Add the Mux icon to enable Asset creation from Mux from the uploaded video. You want to turn on Master Access by entering the value “temporary” in the field. Master Access is where Mux generates a high-resolution MP4, which we will send to Hive to analyze in a subsequent step.
Add the Mux icon for Step 2 as well, since we need the action “Create playback.”
Drop in the Hive icon and select the action “Classify content.” For the Video Source field, click the ➕ icon and then choose “Master File URL” from Step 1. Choose the types of classes you want Hive to analyze for, such as “nsfw.”
Hive will extract sample images from the MP4 and analyze each thumbnail, providing the results as a JSON payload.
For example: If your user uploaded the opening sequence to “Baywatch,” Hive would extract and analyze the thumbnails, and it would look something like this in your account in Hive:
The results are sent back as JSON data to AVflow, where they need to be saved to your AWS S3 bucket with the “Transfer to Storage” step.
AVflow has a nifty feature called Functions, where it can run code as a step in a Flow. AVflow provides a sample script that parses the JSON results from Hive to determine whether a video should be categorized as safe or not safe based on a threshold you set. Generally, the 0.9 threshold works well whereby, if Hive flags any video with a probability of 90% or greater for a class you asked it to scan, then that video should be considered not safe. You may choose to lower the threshold if your platform has a low risk tolerance and it’s more important for you to prevent false negatives than risk blocking content that’s actually safe.
You’ll add this function as a step in your Flow, like this:
Note how you can determine your own probability in the threshold field in the “Input data” section.
There are a variety of options you may choose to take on the content that falls into your not safe category. Maybe you want to queue it for a human moderator to review, unpublish it in your content management system, ban the contributor from your app, or even delete the Mux playback ID. To accomplish whatever action you want to take, you can add a webhook as a final step that will send the binary output of the function — safe or not safe — back to your application, which can then handle this event in any way you see fit.
If you are a Mux customer and you have a platform that allows user-generated content, we hope this post illustrates how, using AVflow and Hive, you can quickly add content moderation to your application in a few minutes without writing code. You can adjust the thresholds (in the AVflow script) per Hive category to best suit your needs and requirements. Although this post focuses on NSFW video analysis, you may wish to add OCR-based text analysis, or audio analysis too. Check out Hive’s Content Moderation Dashboard for an even more turnkey approach.
As your UGC platform grows, bad actors will get ever more creative in trying to circumvent your detection tools; inevitably, your solutions will need to evolve too. Hopefully our post will help you get started building a content moderation system quickly, easily, and flexibly; we can’t wait to see what you do with it! Want to learn more about how Mux and Hive work together to solve content moderation problems? Watch this online event of Mux’s @Dylan Jhaveri and Hive’s Max Chang as they discuss the art of the possible along with pragmatic approaches to content moderation.
No credit card to start. $20 in free credits when you're ready.
50 Beale Street, Floor 9
San Francisco, CA, 94105
34-37 Liverpool Street
Unit 4.06, 4th Floor
London, EC2M 7PP