On this page

Building a video recommendation engine with AI

Use the @mux/ai library to generate video embeddings and build a recommendation engine

This guide uses @mux/ai, our open-source library that provides prebuilt workflows for common video AI tasks. It works with your favorite LLM provider (OpenAI, Anthropic, or Google). Check out the GitHub repository for more details!

You can build a content-based recommendation system that suggests similar videos by converting video transcripts into AI embeddings and performing vector similarity search. The @mux/ai library makes this straightforward by handling transcript fetching, chunking, and embedding generation.

Overview

The core concept is to convert text (your video transcripts) into high-dimensional vectors (embeddings) that capture semantic meaning. Videos with similar content will have embeddings that are close together in vector space, allowing you to find and recommend similar content.

Prerequisites

Before starting, make sure you have:

A Mux account with API credentials (token ID and token secret)
An API key for OpenAI or Google (the providers that support embeddings)
A vector database (Pinecone, Supabase with pgvector, etc.)
Node.js installed
Videos with captions enabled (human-generated captions are best, but auto-generated captions work great too)

Installation

npm install @mux/ai

Configuration

Set your environment variables:

# Required
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
# You only need the API key for the provider you're using
OPENAI_API_KEY=your_openai_api_key # OR
GOOGLE_GENERATIVE_AI_API_KEY=your_google_api_key

Basic usage

import { generateVideoEmbeddings } from "@mux/ai/workflows";

const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "openai",  // or "google"
  languageCode: "en"
});

// Use the averaged embedding for video-level search
console.log(result.averagedEmbedding);
// Array of 1536 numbers (for OpenAI's text-embedding-3-small)

// Or use individual chunks for timestamp-accurate search
console.log(result.chunks.length);
console.log(result.chunks[0].embedding);

The function returns:

{
  "assetId": "your-asset-id",
  "averagedEmbedding": [0.123, -0.456, ...],  // Single vector representing the whole video
  "chunks": [
    {
      "chunkId": "chunk_0",
      "embedding": [0.234, -0.567, ...],
      "metadata": {
        "tokenCount": 450,
        "startTime": 0,
        "endTime": 30.5,
        "text": "Welcome to our tutorial..."
      }
    }
    // ... more chunks
  ],
  "metadata": {
    "totalChunks": 12,
    "totalTokens": 5432,
    "embeddingDimensions": 1536,
    "languageCode": "en"
  }
}

Provider options

@mux/ai supports two embedding providers:

OpenAI (default): Uses text-embedding-3-small model (default 1536 dimensions) - Fast and cost-effective
Google: Uses text-embedding-004 model (768 dimensions) - Alternative option

// Using OpenAI (default)
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "openai"
});

// Using Google
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "google"
});

// Override the default model
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "openai",
  model: "text-embedding-3-large"  // 3072 dimensions, higher quality
});

Chunking strategies

For long videos, transcripts are split into chunks to fit within embedding model token limits. Chunking strategy affects the granularity of your search results:

Token-based chunking: Splits text by token count, maximizing information density. Best for general video-level recommendations.
VTT-based chunking: Preserves caption boundaries and timing metadata. Best when you need timestamp-accurate search or want to recommend specific video segments.

// Token-based chunking (default)
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  chunkingStrategy: {
    type: "token",
    maxTokens: 500,
    overlap: 100  // Tokens of overlap between chunks
  }
});

// VTT-based chunking (preserves caption boundaries)
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  chunkingStrategy: {
    type: "vtt",
    maxTokens: 500,
    overlapCues: 2  // Number of caption cues to overlap
  }
});

Storing embeddings in a vector database

Using Pinecone

import { Pinecone } from '@pinecone-database/pinecone';
import { generateVideoEmbeddings } from "@mux/ai/workflows";

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pinecone.index('video-recommendations');

// Generate and store embeddings
const result = await generateVideoEmbeddings("your-mux-asset-id");

// Store the averaged embedding for video-level search
await index.upsert([{
  id: "your-mux-asset-id",
  values: result.averagedEmbedding,
  metadata: {
    title: "Video Title",
    duration: 300,
    totalChunks: result.metadata.totalChunks
  }
}]);

Using Supabase with pgvector

import { createClient } from '@supabase/supabase-js';
import { generateVideoEmbeddings } from "@mux/ai/workflows";

const supabase = createClient(
  process.env.SUPABASE_URL,
  process.env.SUPABASE_KEY
);

// Generate embeddings
const result = await generateVideoEmbeddings("your-mux-asset-id");

// Store in Supabase
await supabase.from('video_embeddings').insert({
  asset_id: "your-mux-asset-id",
  embedding: result.averagedEmbedding,
  metadata: result.metadata
});

Finding similar videos

Once embeddings are stored, use vector similarity search:

// Generate embedding for the query video
const queryResult = await generateVideoEmbeddings(queryAssetId);

// Search for similar videos in Pinecone
const searchResults = await index.query({
  vector: queryResult.averagedEmbedding,
  topK: 5,  // Return 5 most similar videos
  includeMetadata: true
});

// Display recommendations
searchResults.matches.forEach(match => {
  console.log(`Similar video: ${match.id}`);
  console.log(`Similarity score: ${match.score}`);
});

Webhook integration

For automated embedding generation when videos are uploaded, you should trigger the call to generate video embeddings from the video.asset.track.ready webhook:

export async function handleWebhook(req, res) {
  const event = req.body;

  if (event.type === 'video.asset.track.ready' &&
      event.data.type === 'text' &&
      event.data.language_code === 'en') {
    const result = await generateVideoEmbeddings(event.data.asset_id);
    await vectorDB.upsert({ id: event.data.asset_id, embedding: result.averagedEmbedding });
  }
}

How it works

Under the hood, @mux/ai handles:

Fetching transcript: Downloads the VTT file from Mux
Chunking: Splits long transcripts into manageable pieces
Token counting: Ensures chunks fit within model limits
Batch processing: Sends chunks to the embedding API efficiently
Averaging: Computes a single vector for the whole video
Metadata tracking: Preserves timing and token information

Mux features used

Auto-generated captions - Source transcripts for embeddings

Best practices

Enable captions: Human-generated captions provide the best results, but auto-generated captions work great too
Choose appropriate chunking: Token-based for most cases, VTT-based for caption-aligned chunks
Use averaged embeddings for video-level search: Faster and simpler for basic recommendations
Use chunk embeddings for precise matching: Better for finding specific segments within videos
Consistent embedding model: Always use the same model for queries and storage
Set quality thresholds: Only recommend videos with similarity scores above a minimum threshold

Vector database options

Here are a few popular options for storing embeddings:

Pinecone: Managed vector database, easy to use
Supabase with pgvector: PostgreSQL extension, good for existing Postgres users
Weaviate: Open-source vector database
Milvus: Scalable vector database
Qdrant: High-performance vector search

Resources

On this page

Building a video recommendation engine with AI

Use the @mux/ai library to generate video embeddings and build a recommendation engine

Overview

Prerequisites

Before starting, make sure you have:

A Mux account with API credentials (token ID and token secret)
An API key for OpenAI or Google (the providers that support embeddings)
A vector database (Pinecone, Supabase with pgvector, etc.)
Node.js installed
Videos with captions enabled (human-generated captions are best, but auto-generated captions work great too)

Installation

npm install @mux/ai

Configuration

Set your environment variables:

# Required
MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
# You only need the API key for the provider you're using
OPENAI_API_KEY=your_openai_api_key # OR
GOOGLE_GENERATIVE_AI_API_KEY=your_google_api_key

Basic usage

import { generateVideoEmbeddings } from "@mux/ai/workflows";

const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "openai",  // or "google"
  languageCode: "en"
});

// Use the averaged embedding for video-level search
console.log(result.averagedEmbedding);
// Array of 1536 numbers (for OpenAI's text-embedding-3-small)

// Or use individual chunks for timestamp-accurate search
console.log(result.chunks.length);
console.log(result.chunks[0].embedding);

The function returns:

{
  "assetId": "your-asset-id",
  "averagedEmbedding": [0.123, -0.456, ...],  // Single vector representing the whole video
  "chunks": [
    {
      "chunkId": "chunk_0",
      "embedding": [0.234, -0.567, ...],
      "metadata": {
        "tokenCount": 450,
        "startTime": 0,
        "endTime": 30.5,
        "text": "Welcome to our tutorial..."
      }
    }
    // ... more chunks
  ],
  "metadata": {
    "totalChunks": 12,
    "totalTokens": 5432,
    "embeddingDimensions": 1536,
    "languageCode": "en"
  }
}

Provider options

@mux/ai supports two embedding providers:

OpenAI (default): Uses text-embedding-3-small model (default 1536 dimensions) - Fast and cost-effective
Google: Uses text-embedding-004 model (768 dimensions) - Alternative option

// Using OpenAI (default)
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "openai"
});

// Using Google
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "google"
});

// Override the default model
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  provider: "openai",
  model: "text-embedding-3-large"  // 3072 dimensions, higher quality
});

Chunking strategies

For long videos, transcripts are split into chunks to fit within embedding model token limits. Chunking strategy affects the granularity of your search results:

Token-based chunking: Splits text by token count, maximizing information density. Best for general video-level recommendations.
VTT-based chunking: Preserves caption boundaries and timing metadata. Best when you need timestamp-accurate search or want to recommend specific video segments.

// Token-based chunking (default)
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  chunkingStrategy: {
    type: "token",
    maxTokens: 500,
    overlap: 100  // Tokens of overlap between chunks
  }
});

// VTT-based chunking (preserves caption boundaries)
const result = await generateVideoEmbeddings("your-mux-asset-id", {
  chunkingStrategy: {
    type: "vtt",
    maxTokens: 500,
    overlapCues: 2  // Number of caption cues to overlap
  }
});

Storing embeddings in a vector database

Using Pinecone

import { Pinecone } from '@pinecone-database/pinecone';
import { generateVideoEmbeddings } from "@mux/ai/workflows";

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pinecone.index('video-recommendations');

// Generate and store embeddings
const result = await generateVideoEmbeddings("your-mux-asset-id");

// Store the averaged embedding for video-level search
await index.upsert([{
  id: "your-mux-asset-id",
  values: result.averagedEmbedding,
  metadata: {
    title: "Video Title",
    duration: 300,
    totalChunks: result.metadata.totalChunks
  }
}]);

Using Supabase with pgvector

import { createClient } from '@supabase/supabase-js';
import { generateVideoEmbeddings } from "@mux/ai/workflows";

const supabase = createClient(
  process.env.SUPABASE_URL,
  process.env.SUPABASE_KEY
);

// Generate embeddings
const result = await generateVideoEmbeddings("your-mux-asset-id");

// Store in Supabase
await supabase.from('video_embeddings').insert({
  asset_id: "your-mux-asset-id",
  embedding: result.averagedEmbedding,
  metadata: result.metadata
});

Finding similar videos

Once embeddings are stored, use vector similarity search:

// Generate embedding for the query video
const queryResult = await generateVideoEmbeddings(queryAssetId);

// Search for similar videos in Pinecone
const searchResults = await index.query({
  vector: queryResult.averagedEmbedding,
  topK: 5,  // Return 5 most similar videos
  includeMetadata: true
});

// Display recommendations
searchResults.matches.forEach(match => {
  console.log(`Similar video: ${match.id}`);
  console.log(`Similarity score: ${match.score}`);
});

Webhook integration

For automated embedding generation when videos are uploaded, you should trigger the call to generate video embeddings from the video.asset.track.ready webhook:

export async function handleWebhook(req, res) {
  const event = req.body;

  if (event.type === 'video.asset.track.ready' &&
      event.data.type === 'text' &&
      event.data.language_code === 'en') {
    const result = await generateVideoEmbeddings(event.data.asset_id);
    await vectorDB.upsert({ id: event.data.asset_id, embedding: result.averagedEmbedding });
  }
}

How it works

Under the hood, @mux/ai handles:

Fetching transcript: Downloads the VTT file from Mux
Chunking: Splits long transcripts into manageable pieces
Token counting: Ensures chunks fit within model limits
Batch processing: Sends chunks to the embedding API efficiently
Averaging: Computes a single vector for the whole video
Metadata tracking: Preserves timing and token information

Mux features used

Auto-generated captions - Source transcripts for embeddings

Best practices

Enable captions: Human-generated captions provide the best results, but auto-generated captions work great too
Choose appropriate chunking: Token-based for most cases, VTT-based for caption-aligned chunks
Use averaged embeddings for video-level search: Faster and simpler for basic recommendations
Use chunk embeddings for precise matching: Better for finding specific segments within videos
Consistent embedding model: Always use the same model for queries and storage
Set quality thresholds: Only recommend videos with similarity scores above a minimum threshold

Vector database options

Here are a few popular options for storing embeddings:

Pinecone: Managed vector database, easy to use
Supabase with pgvector: PostgreSQL extension, good for existing Postgres users
Weaviate: Open-source vector database
Milvus: Scalable vector database
Qdrant: High-performance vector search