On this page

Automatic subtitle translations with AI

Use the @mux/ai library to automatically translate video captions into different languages via LLMs

This guide uses @mux/ai, our open-source library that provides prebuilt workflows for common video AI tasks. It works with your favorite LLM provider (OpenAI, Anthropic, or Google). Check out the GitHub repository for more details!

Mux uses OpenAI's Whisper model to create auto-generated captions, which must be generated in the same language as your source audio. To make your content accessible globally, you can use AI to translate captions into other languages. The @mux/ai library makes this straightforward by handling caption fetching, translation, and re-uploading to Mux.

Prerequisites

Before starting, make sure you have:

A Mux account with API credentials (token ID and token secret)
An API key for your preferred AI provider (OpenAI, Anthropic, or Google)
An S3-compatible storage bucket (required for caption file hosting during upload)
Node.js installed
Videos with captions enabled (human-generated captions are best, but auto-generated captions work great too)

Installation

npm install @mux/ai

Configuration

Set your environment variables:

# Required for Mux
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
# You only need the API key for the provider you're using
OPENAI_API_KEY=your_openai_api_key # OR
ANTHROPIC_API_KEY=your_anthropic_api_key # OR
GOOGLE_GENERATIVE_AI_API_KEY=your_google_api_key
# Required for uploading translated captions back to Mux
S3_ENDPOINT=https://your-s3-endpoint.com
S3_REGION=auto
S3_BUCKET=your-bucket-name
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key

Basic usage

import { translateCaptions } from "@mux/ai/workflows";

// Translate English captions to Spanish
const result = await translateCaptions(
  "your-mux-asset-id",
  "en",  // source language
  "es",  // target language
  {
    provider: "anthropic" // or "openai" or "google"
  }
);

console.log(result.uploadedTrackId);
// The new Mux track ID for the translated captions

The function automatically:

Fetches the source captions from Mux
Translates them using your chosen AI provider
Uploads the translated VTT file to your S3 bucket
Creates a new caption track on your Mux asset
Returns the new track ID

Language support

@mux/ai uses ISO 639-1 language codes and automatically converts them to full language names. It supports all standard language codes, but the translation capability of your chosen AI provider may vary.

// Common translations
await translateCaptions("your-mux-asset-id", "en", "es");  // English → Spanish
await translateCaptions("your-mux-asset-id", "en", "fr");  // English → French
await translateCaptions("your-mux-asset-id", "en", "de");  // English → German
await translateCaptions("your-mux-asset-id", "en", "ja");  // English → Japanese
await translateCaptions("your-mux-asset-id", "en", "zh");  // English → Chinese
await translateCaptions("your-mux-asset-id", "en", "ar");  // English → Arabic
// etc.

Provider options

@mux/ai supports three AI providers:

OpenAI (default): Uses gpt-5-mini model - Fast and cost-effective
Anthropic: Uses claude-sonnet-4-5 model - Excellent for nuanced translations
Google: Uses gemini-2.5-flash model - Balance of speed and quality

const result = await translateCaptions("your-mux-asset-id", "en", "es", {
  provider: "anthropic",
  model: "claude-opus-4-5" // Optional: override default model
});

Translate without uploading

If you want to handle the upload yourself or just get the translated file:

const result = await translateCaptions("your-mux-asset-id", "en", "es", {
  uploadToMux: false
});

console.log(result.presignedUrl);
// URL to download the translated VTT file for review before uploading to Mux

Webhook integration

For automated translation when videos are uploaded, you should trigger the call to translate captions from the video.asset.track.ready webhook for your source language:

export async function handleWebhook(req, res) {
  const event = req.body;

  if (event.type === 'video.asset.track.ready' &&
      event.data.type === 'text' &&
      event.data.language_code === 'en') {
    const result = await translateCaptions(event.data.asset_id, "en", "es");
    await db.saveTranslationTrack(event.data.asset_id, result.uploadedTrackId);
  }
}

Using with Mux Player

Mux Player automatically detects multiple caption tracks and shows a language selector:

<mux-player
  playback-id="your-playback-id"
  metadata-video-title="My Video"
></mux-player>

Users can switch between languages using the captions menu in the player controls.

Complete example

Here's a complete webhook handler that translates captions:

import express from 'express';
import { translateCaptions } from "@mux/ai/workflows";

const app = express();
app.use(express.json());

app.post('/webhook', async (req, res) => {
  const event = req.body;

  if (event.type === 'video.asset.track.ready' &&
      event.data.type === 'text' &&
      event.data.language_code === 'en') {

    const assetId = event.data.asset_id;

    try {
      // Translate to Spanish
      const result = await translateCaptions(assetId, "en", "es");

      console.log(`Spanish captions created: ${result.uploadedTrackId}`);

      res.status(200).json({ success: true });
    } catch (error) {
      console.error('Translation error:', error);
      res.status(500).json({ error: error.message });
    }
  } else {
    res.status(200).json({ message: 'Event ignored' });
  }
});

app.listen(3000, () => {
  console.log('Webhook server running on port 3000');
});

How it works

Under the hood, @mux/ai handles:

Fetching source captions: Downloads the VTT file from Mux
Translation: Sends the captions to your chosen AI provider with optimized prompts
VTT preservation: Maintains timing information and formatting
S3 upload: Uploads the translated file to your S3 bucket with a presigned URL
Mux track creation: Creates a new caption track on your asset
Cleanup: Optionally cleans up temporary files

Mux features used

Auto-generated captions - Source captions for translation
Webhooks - Trigger translations automatically
Mux Player - Display translated captions with language switching

Best practices

Validate translations and review quality: AI translations are generally accurate but may miss context-specific nuances - for critical content, consider human review
Handle errors gracefully: Translation may fail for very long videos or stability issues with LLMs
Consider costs: Translating to many languages increases LLM costs

Resources

On this page

Automatic subtitle translations with AI

Use the @mux/ai library to automatically translate video captions into different languages via LLMs

Prerequisites

Before starting, make sure you have:

A Mux account with API credentials (token ID and token secret)
An API key for your preferred AI provider (OpenAI, Anthropic, or Google)
An S3-compatible storage bucket (required for caption file hosting during upload)
Node.js installed
Videos with captions enabled (human-generated captions are best, but auto-generated captions work great too)

Installation

npm install @mux/ai

Configuration

Set your environment variables:

# Required for Mux
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
# You only need the API key for the provider you're using
OPENAI_API_KEY=your_openai_api_key # OR
ANTHROPIC_API_KEY=your_anthropic_api_key # OR
GOOGLE_GENERATIVE_AI_API_KEY=your_google_api_key
# Required for uploading translated captions back to Mux
S3_ENDPOINT=https://your-s3-endpoint.com
S3_REGION=auto
S3_BUCKET=your-bucket-name
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key

Basic usage

import { translateCaptions } from "@mux/ai/workflows";

// Translate English captions to Spanish
const result = await translateCaptions(
  "your-mux-asset-id",
  "en",  // source language
  "es",  // target language
  {
    provider: "anthropic" // or "openai" or "google"
  }
);

console.log(result.uploadedTrackId);
// The new Mux track ID for the translated captions

The function automatically:

Fetches the source captions from Mux
Translates them using your chosen AI provider
Uploads the translated VTT file to your S3 bucket
Creates a new caption track on your Mux asset
Returns the new track ID

Language support

// Common translations
await translateCaptions("your-mux-asset-id", "en", "es");  // English → Spanish
await translateCaptions("your-mux-asset-id", "en", "fr");  // English → French
await translateCaptions("your-mux-asset-id", "en", "de");  // English → German
await translateCaptions("your-mux-asset-id", "en", "ja");  // English → Japanese
await translateCaptions("your-mux-asset-id", "en", "zh");  // English → Chinese
await translateCaptions("your-mux-asset-id", "en", "ar");  // English → Arabic
// etc.

Provider options

@mux/ai supports three AI providers:

OpenAI (default): Uses gpt-5-mini model - Fast and cost-effective
Anthropic: Uses claude-sonnet-4-5 model - Excellent for nuanced translations
Google: Uses gemini-2.5-flash model - Balance of speed and quality

const result = await translateCaptions("your-mux-asset-id", "en", "es", {
  provider: "anthropic",
  model: "claude-opus-4-5" // Optional: override default model
});

Translate without uploading

If you want to handle the upload yourself or just get the translated file:

const result = await translateCaptions("your-mux-asset-id", "en", "es", {
  uploadToMux: false
});

console.log(result.presignedUrl);
// URL to download the translated VTT file for review before uploading to Mux

Webhook integration

For automated translation when videos are uploaded, you should trigger the call to translate captions from the video.asset.track.ready webhook for your source language:

export async function handleWebhook(req, res) {
  const event = req.body;

  if (event.type === 'video.asset.track.ready' &&
      event.data.type === 'text' &&
      event.data.language_code === 'en') {
    const result = await translateCaptions(event.data.asset_id, "en", "es");
    await db.saveTranslationTrack(event.data.asset_id, result.uploadedTrackId);
  }
}

Using with Mux Player

Mux Player automatically detects multiple caption tracks and shows a language selector:

<mux-player
  playback-id="your-playback-id"
  metadata-video-title="My Video"
></mux-player>

Users can switch between languages using the captions menu in the player controls.

Complete example

Here's a complete webhook handler that translates captions:

import express from 'express';
import { translateCaptions } from "@mux/ai/workflows";

const app = express();
app.use(express.json());

app.post('/webhook', async (req, res) => {
  const event = req.body;

  if (event.type === 'video.asset.track.ready' &&
      event.data.type === 'text' &&
      event.data.language_code === 'en') {

    const assetId = event.data.asset_id;

    try {
      // Translate to Spanish
      const result = await translateCaptions(assetId, "en", "es");

      console.log(`Spanish captions created: ${result.uploadedTrackId}`);

      res.status(200).json({ success: true });
    } catch (error) {
      console.error('Translation error:', error);
      res.status(500).json({ error: error.message });
    }
  } else {
    res.status(200).json({ message: 'Event ignored' });
  }
});

app.listen(3000, () => {
  console.log('Webhook server running on port 3000');
});

How it works

Under the hood, @mux/ai handles:

Fetching source captions: Downloads the VTT file from Mux
Translation: Sends the captions to your chosen AI provider with optimized prompts
VTT preservation: Maintains timing information and formatting
S3 upload: Uploads the translated file to your S3 bucket with a presigned URL
Mux track creation: Creates a new caption track on your asset
Cleanup: Optionally cleans up temporary files

Mux features used

Auto-generated captions - Source captions for translation
Webhooks - Trigger translations automatically
Mux Player - Display translated captions with language switching

Best practices

Validate translations and review quality: AI translations are generally accurate but may miss context-specific nuances - for critical content, consider human review
Handle errors gracefully: Translation may fail for very long videos or stability issues with LLMs
Consider costs: Translating to many languages increases LLM costs