Use any* LLM with @mux/ai

*Almost any.

Back in May, Joshua on our Community Engineering team forked @mux/ai to fine-tune his own video intelligence model. He added Ollama so he could experiment locally without paying for tokens, added Baseten so he could train and host a LoRA fine-tune of Mistral Small 3.1, and wired the result into @mux/ai's workflows. It's a great post, but he needed a fork of the SDK to do it.

When we launched @mux/ai in December last year, we built it as a BYO-LLM toolkit so you'd stay in control of which providers see your content. But "bring your own" has so far meant bring your own hosted provider: primarily OpenAI, Anthropic, or Google. If you're running your own inference infrastructure, or your policies decide where your video is allowed to go, that list probably wasn't much use to you, and we wanted @mux/ai to be an option for you too.

So why are we adding support for wider LLMs now? It’s pretty simple - open-weight models have improved dramatically, especially multimodal ones. Deploying them has also got much easier: providers like Baseten will serve open models behind an OpenAI-compatible endpoint, and if you'd rather self-host, vLLM and friends speak that same OpenAI-compatible protocol.

A couple of weeks ago we shipped @mux/ai support for both (#205): a first-class baseten provider, and a general-purpose openai-compatible provider that works with any endpoint speaking the OpenAI protocol. Both are built on the AI SDK's @ai-sdk/openai-compatible package under the hood.

Bring your own endpoint

Configuration is the same shape for every provider: a URL, a key, and a model. Here's the general-purpose one pointed at a local Ollama:

OPENAI_COMPATIBLE_BASE_URL=http://localhost:11434/v1
OPENAI_COMPATIBLE_API_KEY=      # optional! local endpoints are allowed to be keyless
OPENAI_COMPATIBLE_MODEL=qwen3-vl:30b

And then the workflows just work, exactly as they do with the hosted providers:

import { getSummaryAndTags } from "@mux/ai/workflows";

const result = await getSummaryAndTags("your-mux-asset-id", {
  provider: "openai-compatible",
});

Or, running my favourite question from stream.new's moderation pipeline against Kimi K3 on Baseten's Model APIs:

import { askQuestions } from "@mux/ai/workflows";

const result = await askQuestions("your-mux-asset-id", [
  { question: "Is this video mostly of feet?" },
], {
  provider: "baseten",
  model: "moonshotai/Kimi-K3",
});

One difference from the hosted providers: there's no default model. We can't know what you've deployed on your own infrastructure, so you always tell us, either in code or via BASETEN_MODEL / OPENAI_COMPATIBLE_MODEL.

Why two providers?

So the original plan was just Baseten. Joshua's fork had proven the integration worked (and that it was easy to build something with real value using it), Baseten's Model APIs give you a catalogue of open models behind one endpoint, and their dedicated deployments are a great home for a fine-tuned model.

Partway through the work we noticed we weren't really building a Baseten integration. Baseten's endpoints speak the OpenAI protocol, and so does more or less everything else in this space: vLLM, SGLang, Ollama, Together AI, Fireworks, OpenRouter, and whatever gateway your platform team deployed last quarter. Building on the OpenAI-compatible protocol directly meant one integration covered the lot, so we shipped it as its own provider, with Baseten kept as a preset on top. The Baseten preset knows about Baseten's URL shapes and validates them, so a misconfigured deployment fails immediately with a useful error.

So about that asterisk…

Here's the fine print: @mux/ai's workflows ask more of a model than a chat app does.

First, the four workflows that look at video frames or storyboards (getSummaryAndTags, askQuestions, hasBurnedInCaptions, and generateEngagementInsights) need a vision-capable model. By default we hand the model a URL to the storyboard and the endpoint fetches it server-side; if your endpoint can't fetch remote URLs passed via the image_url param (Ollama historically can't, for example), set imageSubmissionMode: "base64" and we'll inline the image instead, unlocking many more models.

Second, everything requests structured output via response_format: json_schema. That's how the workflows return typed results you can actually use, and endpoints without json_schema support fail loudly rather than producing unparseable output.

The text-only workflows (generateChapters, translateCaptions, editCaptions) are much less picky: almost any model with structured outputs and a decent-sized context window will do, including text-only open models like DeepSeek V4 or gpt-oss.

To save you some trial and error, we've published notes on which models we've verified and which should work. Kimi K3, both Inkling variants, and a couple of Kimi K2 models are verified against @mux/ai on Baseten's Model APIs (as of August 2026). The Mistral family, Qwen3-VL, Llama 4, and a handful of others are documented as vision-capable by their vendors but we haven't verified them ourselves, so we ship a script that checks both requirements against a live endpoint before you bet a pipeline on one:

npx tsx scripts/verify-vision-models.ts -p openai-compatible -m "<your-model>"

We can't eval your model for you, either. Every hosted provider in @mux/ai ships with eval coverage so you know what quality you're getting; your fine-tuned Mistral on your own GPU is invisible to us, so these providers are excluded from eval runs, and cost estimation returns zero (we don't know your GPU bill, or how fast your GPU is for calculating latency, and we're not going to guess). The verification script tells you a model can run the workflows; whether it runs them well is between you and your evals.

In our testing, open models also have some reliability quirks we haven't seen on the closed models, and we hit one with Inkling: roughly one request in twelve degenerated into output truncated at the token cap, which then failed schema validation, even with json_schema requested and the endpoint otherwise healthy. The failure isn't deterministic: retrying the same request almost always succeeds, so withRetry (which already wraps every workflow's model call) now treats a no-output response as retryable. Content-policy refusals are never retried, for hopefully obvious reasons.

And finally, fine-tuning

If you want to take this further, Joshua's post walks through fine-tuning your own model for these workflows: LoRA-tuning Mistral Small 3.1 on synthetic @mux/ai workflow data with Baseten's training SDK, then hosting it on a dedicated deployment. When he wrote it, that needed a fork; now his deployment plugs straight into the baseten provider. In his comparison, the fine-tune produced more concise, opinionated summaries than the default LLM output. Go read it. It's fun!

Try it out

@mux/ai is fully open source under Apache 2.0, if you've forgotten. This feature started as Joshua's fork, and we'd love the next one to start as yours:

npm install @mux/ai

The provider docs cover configuration for both new providers, and VISION-MODELS.md is the place to check before choosing a model. If you get a model working that we haven't verified, feel free to drop us a pull request. And as always, we want to hear your feedback, and to see what you build.