While Mux's auto-generated captions can produce transcripts in the original language, you may want to translate and dub the audio itself into different languages. The @mux/ai library integrates with ElevenLabs' dubbing API to automatically create translated audio tracks and add them to your videos as multi-track audio.

Prerequisites

Before starting, make sure you have:

• A Mux account with API credentials (token ID and token secret)
• An ElevenLabs API key with dubbing access
• An S3-compatible storage bucket (required for audio file hosting during upload)
• Node.js installed

Installation

npm install @mux/ai

Configuration

Set your environment variables:

# Required for Mux
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
# Required for ElevenLabs dubbing
ELEVENLABS_API_KEY=your_elevenlabs_api_key
# Required for uploading dubbed audio back to Mux
S3_ENDPOINT=https://your-s3-endpoint.com
S3_REGION=auto
S3_BUCKET=your-bucket-name
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key

Enabling audio static renditions

The @mux/ai library automatically requests audio-only static renditions if they don't already exist on your asset. However, you can pre-emptively enable them when creating videos to avoid waiting for rendition generation during the dubbing workflow.

To enable when creating a video:

import Mux from '@mux/mux-node';

const mux = new Mux();

const asset = await mux.video.assets.create({
  input: "https://example.com/video.mp4",
  playback_policy: ['public'],
  static_renditions: [
    { resolution: 'audio-only' }  // Enable audio.m4a rendition
  ]
});

Or add to an existing asset:

await mux.video.assets.createStaticRendition("your-mux-asset-id", {
  resolution: 'audio-only'
});

Basic usage

import { translateAudio } from "@mux/ai/workflows";

// Dub video audio to Spanish
const result = await translateAudio(
  "your-mux-asset-id",
  "es"  // target language (source language is auto-detected)
);

console.log(result.uploadedTrackId);
// The new Mux audio track ID for the dubbed audio

console.log(result.dubbingId);
// ElevenLabs dubbing ID for tracking

console.log(result.targetLanguageCode);  // "es"

The function automatically:

1. Fetches the audio.m4a static rendition from Mux
2. Sends it to ElevenLabs for dubbing (source language auto-detected)
3. Waits for the dubbing to complete
4. Uploads the dubbed audio to your S3 bucket
5. Creates a new audio track on your Mux asset
6. Returns the new track ID

Language support

The library uses ISO 639-1 language codes. Common target languages include:

await translateAudio("your-mux-asset-id", "es");  // Spanish
await translateAudio("your-mux-asset-id", "fr");  // French
await translateAudio("your-mux-asset-id", "de");  // German
await translateAudio("your-mux-asset-id", "ja");  // Japanese
await translateAudio("your-mux-asset-id", "zh");  // Chinese
await translateAudio("your-mux-asset-id", "pt");  // Portuguese
await translateAudio("your-mux-asset-id", "it");  // Italian
// etc.

Speaker detection

You can specify the number of speakers for better dubbing quality:

// Auto-detect number of speakers (default)
const result = await translateAudio("your-mux-asset-id", "es", {
  numSpeakers: 0
});

// Specify exact number of speakers
const result = await translateAudio("your-mux-asset-id", "es", {
  numSpeakers: 2  // For videos with 2 distinct speakers
});

Download without uploading

If you want to handle the upload yourself or just get the dubbed audio file:

const result = await translateAudio("your-mux-asset-id", "es", {
  uploadToMux: false
});

console.log(result.presignedUrl);
// URL to download the dubbed audio file for manual review before uploading to Mux

Webhook integration

For automated dubbing when videos are uploaded, you should trigger the call to translate audio from the video.asset.static_rendition.ready webhook:

export async function handleWebhook(req, res) {
  const event = req.body;

  if (event.type === 'video.asset.static_rendition.ready') {
    const result = await translateAudio(event.data.id, "es");
    await db.saveDubbedTrack(event.data.id, result.uploadedTrackId);
  }
}

Playing multi-language content

Mux Player (and most other common video players), automatically detects multiple audio tracks and shows an audio selector. Users can switch between audio languages using the audio menu in the player controls.

How it works

Under the hood, @mux/ai handles:

1. Fetching source audio: Downloads the audio.m4a static rendition from Mux
2. ElevenLabs dubbing: Submits the audio to ElevenLabs with language parameters
3. Polling: Waits for the dubbing job to complete (can take several minutes)
4. Download: Retrieves the dubbed audio file
5. S3 upload: Uploads the dubbed file to your S3 bucket with a presigned URL
6. Mux track creation: Creates a new audio track on your asset

Mux features used

• Audio-only static renditions - Source audio for dubbing
• Multi-track audio - Adding dubbed tracks
• Webhooks - Trigger dubbing automatically
• Mux Player - Play videos with language-selectable audio

Best practices

• Enable audio-only renditions: Required for the dubbing workflow
• Sequential processing: Process one language at a time to avoid rate limits
• Error handling: Dubbing can fail or take time; implement retries and timeouts
• Cost management: Dubbing is more expensive than caption translation and takes several minutes per video
• Quality review: AI dubbing quality varies - voices may not match the original tone, lip sync can be off, and nuances like humor or cultural references may be lost. Consider human review for important or high-visibility content
• Set user expectations: Add labels like "Auto-dubbed" in your UI to indicate the content is AI-generated

Video demo

Here's an example of AI-dubbed audio in action:

Resources

• @mux/ai GitHub Repository
• @mux/ai Workflows Documentation
• ElevenLabs Dubbing API
• Mux Multi-track Audio
• Mux Static Renditions