Automatically moderate video content using AI to detect violence or nudity in your Mux assets
This guide demonstrates how to integrate OpenAI's moderation models with Mux Video to automatically screen video content for inappropriate material. When a video is uploaded and processed by Mux, the workflow will extract thumbnails at regular intervals and analyze them using OpenAI's vision-based moderation API. If the content exceeds your defined thresholds for sexual or violent content, the workflow automatically removes the playback ID, effectively blocking access to the video.
This approach provides an automated first line of defense against inappropriate content, helping you maintain content standards at scale without manual review of every upload.
video.asset.ready eventsBefore starting this guide, make sure you have:
First, you'll need to install the following dependencies:
npm install @mux/mux-node openai dotenvAs we're using dotenv, create a .env file in your project root with your credentials:
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
OPENAI_API_KEY=your_openai_api_keyFirst, let's create the core moderation functionality that selects thumbnails from an asset, and analyzes them using OpenAI's moderation API:
// moderation.js
import { OpenAI } from 'openai';
const openaiClient = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
// Moderation thresholds, you can adjust these as you refine your moderation logic
const THRESHOLDS = {
sexual: 0.7,
violence: 0.8
};
// Generates a list of thumbnail URLs at regular intervals, based on the asset's duration
export function getThumbnailUrls({ playbackId, duration }) {
const timestamps = [];
if (duration <= 50) {
// Short videos less than 50 seconds: 5 evenly spaced thumbnails
const interval = duration / 6;
for (let i = 1; i <= 5; i++) {
timestamps.push(Math.round(i * interval));
}
} else {
// Longer videos: one thumbnail every 10 seconds
for (let time = 0; time < duration; time += 10) {
timestamps.push(time);
}
}
return timestamps.map(
(time) => `https://image.mux.com/${playbackId}/thumbnail.png?time=${time}&width=640`
);
}
// Sends all the thumbnail URLs to OpenAI's moderation API concurrently, and returns the scores for each thumbnail
export async function requestModeration(imageUrls) {
const moderationPromises = imageUrls.map(async (url) => {
console.log(`Moderating image: ${url}`);
try {
const moderation = await openaiClient.moderations.create({
model: "omni-moderation-latest",
input: [
{
type: "image_url",
image_url: {
url: url,
},
},
],
});
const categoryScores = moderation.results[0].category_scores;
return {
url,
sexual: categoryScores.sexual || 0,
violence: categoryScores.violence || 0,
error: false
};
} catch (error) {
console.error("Failed to moderate image:", error);
return {
url,
sexual: 0,
violence: 0,
error: true,
};
}
});
const scores = await Promise.all(moderationPromises);
// Find highest scores across all thumbnails
const maxSexual = Math.max(...scores.map(s => s.sexual));
const maxViolence = Math.max(...scores.map(s => s.violence));
return {
scores,
maxScores: {
sexual: maxSexual,
violence: maxViolence
},
exceedsThreshold: maxSexual > THRESHOLDS.sexual || maxViolence > THRESHOLDS.violence
};
}Now let's create the webhook handler that processes Mux Video's video.asset.ready events and uses our moderation logic:
// webhook.js
import Mux from '@mux/mux-node';
import { getThumbnailUrls, requestModeration } from './moderation.js';
const mux = new Mux();
// Called when a video asset is ready, and starts the moderation process
async function handleAssetReady(assetData) {
const { id: assetId, duration, playback_ids } = assetData;
if (!playback_ids || playback_ids.length === 0) {
console.log(`No playback IDs for asset ${assetId}, skipping moderation`);
return;
}
// Filter for public playback IDs only
const publicPlaybackIds = playback_ids.filter(pid => pid.policy === 'public');
if (publicPlaybackIds.length === 0) {
console.log(`Asset ${assetId} has only signed playback IDs, skipping moderation`);
return;
}
const playbackId = publicPlaybackIds[0].id;
console.log(`Starting moderation for asset ${assetId}`);
const thumbnailUrls = getThumbnailUrls({ playbackId, duration });
console.log(`Generated ${thumbnailUrls.length} thumbnails for moderation`);
const moderationResult = await requestModeration(thumbnailUrls);
console.log(`Moderation scores - Sexual: ${moderationResult.maxScores.sexual}, Violence: ${moderationResult.maxScores.violence}`);
if (moderationResult.exceedsThreshold) {
// Delete playback ID if content violates thresholds
console.log(`Content exceeds thresholds, removing access to asset`);
try {
await mux.video.assets.deletePlaybackId(assetId, playbackId);
console.log(`Asset ${assetId} has been restricted`);
} catch (error) {
console.error(`Failed to delete playback ID for asset ${assetId}:`, error);
}
} else {
console.log(`Asset ${assetId} passed moderation`);
}
}
// Main webhook handler
export async function muxWebhook(req, res) {
const event = req.body;
if (event.type === 'video.asset.ready') {
try {
await handleAssetReady(event.data);
res.status(200).json({ message: 'Moderation complete' });
} catch (error) {
res.status(500).json({ error: 'Moderation failed' });
}
} else {
res.status(200).json({ message: 'Skipped webhook event' });
}
}Asset Ready Event: When a video finishes processing in Mux, we send a video.asset.ready webhook to your endpoint. This event contains all the information about the newly processed video, including its duration and playback IDs, which we need to moderate the content.
Thumbnail Extraction: We then select a subset of frames to analyze based on asset's length. For assets under 50 seconds, we extract 5 evenly-spaced thumbnails to get a representative sample. For longer videos, we take one thumbnail every 10 seconds.
OpenAI Moderation: Each thumbnail is then sent to OpenAI's omni-moderation-latest model, which uses computer vision to detect potentially inappropriate content. The API returns confidence scores for various categories, with our implementation focusing on sexual and violent content.
Threshold Evaluation: We then compare the highest scores across all thumbnails against the configured thresholds.
Plaback ID removal: If any thumbnail exceeds the thresholds, we immediately delete the playback ID from the asset. This removes public access to the video while preserving the original asset for potential manual review or appeals.
To test this out, you'll need a little web server to receive the webook, here's a simple one that uses express:
// server.js
import 'dotenv/config';
import express from 'express';
import { muxWebhook } from './webhook.js';
const app = express();
const PORT = process.env.PORT || 3000;
app.use(express.json());
app.post('/webhook', muxWebhook);
app.listen(PORT, () => {
console.log(`Server running on port ${PORT}`);
console.log(`Webhook endpoint: http://localhost:${PORT}/webhook`);
});If you're developing locally, you can use ngrok to expose your local server to the internet:
ngrok http 3000This will give you a public URL that you can use to test the webhook. Finally, in your Mux Dashboard, you can create a new webhook endpoint pointing to your public URL - don't forget to append /webhook to the end of your URL.
Obviously, you can also make simple changes to this code to deploy it into your cloud provider or serverless platform of choice.
The moderation thresholds use a 0-1 scale where higher values mean stricter moderation. The default values provide a balanced approach:
const THRESHOLDS = {
sexual: 0.7, // Flags content with 70%+ confidence of sexual content
violence: 0.8 // Flags content with 80%+ confidence of violence
};You should adjust these based on your content policies and user base.
Once you have basic moderation working, you should consider these enhancements: