September 10, 2020 (over 3 years ago)

Survive CDN failures with Redundant Streams

Phil Cluff
By Phil Cluff4 min readProduct

When you're live streaming, you don't get second chances. Everything has to go off perfectly, every time.

Sometimes though, the internet just doesn't want that to be the case. At Mux we work really hard to make sure that our delivery platform is as reliable as possible, but there exist fundamental risks - BGP leaks, large-scale CDN outages, or even a backhoe through a fiber connection, which present significant problems to even the largest streaming video platforms.

Recently we spent time investigating what the fastest, highest value change we could make to Mux Video to improve our resilience to larger, internet wide service outages.

While today Mux already uses multiple CDNs to deliver your content, selected based on the quality of experience, as of right now, we make that decision when the end viewer starts watching the stream. This is acceptable for short-form VoD content, but starts to run into limitations for long-form content, or live streams.

The solution to this problem is usually referred to "mid stream CDN switching". There's a few approaches that we could have taken to enable mid-stream CDN switching for our customers, but as our first step to get something ready as soon as possible for your high value events, we've chosen to leverage a client side technology known as "HLS Redundant Streams".

LinkHLS Redundant Streams

HLS redundant streams are an often forgotten about part of HLS's functionality. This feature allows you to define multiple HLS renditions in your manifest that have identical characteristics, other than the URL which they point to, allowing the player to treat these as a redundant set, for failover purposes. In the HLS manifest, it looks something like this:

#EXTM3U #EXT-X-STREAM-INF: BANDWIDTH=2000000, RESOLUTION=1280x720 http://primary-cdn.cdn.mux.com/2mbps/index.m3u8 #EXT-X-STREAM-INF: BANDWIDTH=2000000, RESOLUTION=1280x720 http://secondary-cdn.cdn.mux.com/2mbps/index.m3u8 #EXT-X-STREAM-INF: BANDWIDTH=5000000, RESOLUTION=1920x1080 http://primary-cdn.cdn.mux.com/5mbps/index.m3u8 #EXT-X-STREAM-INF: BANDWIDTH=5000000, RESOLUTION=1920x1080 http://secondary-cdn.cdn.mux.com/5mbps/index.m3u8

The redundant renditions in this manifest are identified by the player by checking for an identical EXT-X-STREAM-INF line.

Since failover between the HLS redundant streams is implemented on the client side, the behavior and support does vary between players.

In general, video players look for 4XX or 5XX status codes on either segment or manifest downloads, in order to trigger a failover. Some players will fail over immediately to the same rendition on the alternate CDN, while other players will try other renditions on the primary CDN first, before trying the redundant streams.

Let's take a look at a demo of how one of the most popular video players, HLS.js deals with a major CDN outage when redundant streams are enabled.

LinkLooks great! How do I turn it on?

LinkFirst, check your player is compatible

We've spent a lot of time testing the compatibility of the most popular HLS players with a variety of failure scenarios, focusing on failures in both manifest, and media delivery. You can check the video players that you're using in the compatibility grid below:

Player

Tested Version

Manifest 4XX

Manifest 5XX

Media 4XX

Media 5XX

Video.js

>= 7.6.6

HLS.js

>= 0.14.11

JWPlayer

Production Release Channel

Safari iOS (AVPlayer)

iOS >= 13.6.1

Safari MacOS

Safari > 13.1.2

ExoPlayer

>= r2.12.0

You'll notice there's some missing common players on this list; We're actively working with both open and closed source player vendors to make sure that this functionality is available where you need it.

LinkAdd the required playback modifier to your playback URL

Enabling redundant streams is really easy, all you have to do is set a query parameter on your playback URL:

https://stream.mux.com/{PLAYBACK_ID}.m3u8?redundant_streams=true

If you're using signed URLs, you instead need to add a claim inside your JWT body, setting the claim redundant_streams to true - your JWT payload should look something like this:

{ "exp": 1599583520, "aud": "v", "sub": "{PLAYBACK_ID}", "redundant_streams" : true }

Check the playback documentation for more details.

LinkDeploy to some users, and monitor for stability

When you deploy redundant streams, you'll want to make sure that your player is behaving as expected with the new manifests we're delivering. We strongly recommend deploying to a subset of users, and using Mux Data's experiment field to run some A/B testing on your viewers. In normal conditions you wouldn't expect to see any change in the viewer experience scores, only in exceptional circumstances, such as a CDN failure, or routing issues to particular CDNs, so don't be surprised if your viewer scores don't immediately improve.

LinkIn Summary

Leveraging redundant streams in Mux Video gives you access to a more resilient multi-CDN delivery strategy, allowing you more protection against CDN, or internet infrastructure outages. You can enable redundant streams today for free, on both live and on-demand streams.

This isn't the end for our reliability efforts at Mux, we've got a lot more projects in the pipeline to make sure that your live events are bulletproof - keep your eye on the blog for more updates in the future!

Written By

Phil Cluff

Phil Cluff – Director of Product Management - Mux Video Group Product Manager

Phil has spent the last 10 years building some of the biggest AVOD, SVOD, and public service streaming platforms in the world at the BBC and Brightcove. He’s here to chew gum and stream video, and he’s all out of gum.

Leave your wallet where it is

No credit card required to get started.