Published on February 12, 2019 (about 5 years ago)

Multi-CDN support in Mux Video for improved performance and reliability

Scott Kidder
By Scott Kidder10 min readEngineering

Delivering the optimal video viewing experience has always been the highest priority at Mux. In some instances, this means optimizing at the CDN level. While our Mux Data product makes it easy to identify network bottlenecks by ASN or CDN using historical data or in real-time, you may need an advanced CDN strategy such as introducing additional CDNs and performing dynamic CDN-selection.

This blog post will explain why you should care about CDNs, and share how Mux Video automatically leverages multiple CDNs to deliver the best network performance and viewing experience.

LinkWhat is a CDN?

Content Delivery Networks (CDNs) consist of numerous points-of-presence (POPs) in Internet-connected datacenters around the world. A CDN POP typically consists of one or more machines belonging to the CDN racked in an ISPs datacenter. Some POPs use their own Internet backbone connections out of the ISPs datacenter, while others just tie into the ISPs own backbone connection. This is a mutually beneficial arrangement, as it reduces the amount of traffic leaving the ISPs datacenter (lower operational expenses), improves the CDNs performance in the region (better value proposition), and provides the end-user with a shorter path to the content (faster-loading cat videos!).

Artur Bergman, CEO/Founder of Fastly, described CDNs as simple reverse-proxies, with the major differences being that a CDN is distributed around the world, and caches a whole lot more content.

In addition to caching, the benefits of using a CDN can include:

  • Termination of SSL connections close to the client, leading to smaller TCP congestion windows and better throughput
  • Load-balancing of requests to the origin
  • Long-running persistent connections to the origin
  • Rule-based dispatch of requests
  • DDOS protection

These benefits apply even to content not intended to be cached at the CDN. Many companies choose to route all their traffic through CDNs to make use of these benefits and to simplify their network architectures.

LinkWhy Use Multiple CDNs?

An increasing number of video streaming services of all sizes are making use of multiple CDNs. (Dan Rayburn, Streaming Media). Let's explore some reasons for doing so.

LinkPerformance

The round-trip-time (RTT) between the viewer and CDN POP is a major indicator of CDN performance, and for good reason: choosing a nearby CDN POP reduces the connection setup time, as well as the amount of data that must be backhauled by the viewer’s ISP. Even if the request to a local CDN POP ends up being a cache-miss, it stands a good chance of being delivered more quickly than if it were sent to a distant CDN POP with the content cached. Many multi-CDN setups use RTT as a criteria for CDN selection.

“By maintaining servers inside internet exchange points and by having preferred relationships with Internet service providers and other network carriers, a CDN is able to optimize network pathways between locations, resulting in reduced RTT and improved latency for visitors accessing content cached inside the CDN.” (Cloudflare)

Video rebuffering and slow startup times rank among the top reasons for abandoning video playback. These conditions are heavily influenced by the networks used by clients to retrieve content, particularly for latency-sensitive content like live video.

LinkReliability

Uttering the phrase “single-point-of-failure” is enough to raise the blood-pressure of most engineers. Despite the distributed nature of CDNs, they can still present a single-point-of-failure. And CDNs do fail. Cedexis provided a detailed write-up of a July 2014 outage affecting a major CDN that lasted up to 70 minutes. When CDN outages occur, it becomes immediately clear who is using multiple CDNs and who is not.

When CDN outages occur, it becomes immediately clear who is using multiple CDNs and who is not.

Popular live-streaming events also have the potential to saturate the link of CDN POPs for a region. The shared nature of CDNs leads to competition for limited resources, in a variation of the noisy-neighbor problem. This can be avoided by distributing traffic across multiple CDN POPs in the area.

LinkAvoiding Vendor Lock-in

Supporting multiple interchangeable CDN vendors increases your position during pricing negotiations. It’s also possible to strategically manage monthly commits by shifting traffic between CDNs as the cost-profile changes.

LinkDifferences among CDNs

Likening CDNs to glorified reverse-proxies might be an oversimplification, as there are actually many differences among CDN vendors.

CDN peering agreements with ISPs have a huge impact on performance. A CDN establishes a POP in an ISP’s datacenter as part of a peering agreement. This generally leads to lower round-trip-times for requests coming to the nearby CDN POP. Choosing CDNs with a large number of global POPs and peering agreements with many ISPs helps ensure a short hop for every viewer.

Security features also vary among CDNs. Fastly supports any URL signing scheme that can be implemented in VCL. Highwinds and Stackpath have explicit CDN signing schemes that cannot be customized. Amazon CloudFront uses its own custom policy-based URL signing scheme. This presents a challenge, as the CDN must be known prior to signing the content URLs, and signed content is specific to a CDN. It’s also desirable to use different signing keys for each CDN to limit the impact of a key being compromised.

Raw log access also varies. Fastly provides real-time streaming access-logs over Syslog. Highwinds provides logs via FTP with a 15 minute delay. Stackpath provides logs through a REST API that has delay on the order of several hours. Obviously real-time access from all CDNs would be ideal; but slow log delivery is not a user-facing problem, and can be tolerated by most log-processing systems. Still, processing access-logs from multiple CDNs is a chore that few people enjoy.

LinkHow Mux Video Leverages Multiple CDNs

You should now be convinced of the value of CDNs and why using multiple CDNs is desirable. Fortunately, all users of Mux Video are already using multiple CDNs! Here’s how we did it.

LinkPerformance-based CDN Selection

Mux uses Cedexis OpenMix to dynamically select the optimal CDN for every viewer at the start of playback. Cedexis gathers Real-User Monitoring (RUM) metrics for CDNs around the world, as well as Mux Data performance data for video playback.

Video playback is initiated by a request for the HLS master manifest. Mux generates dynamic manifests for every viewer. While the Mux manifest server is retrieving details about the video asset, a parallel request is made to Cedexis OpenMix to find the optimal CDN given the IP address of the viewer. The master manifest response includes rendition manifest URLs with the optimal CDN specified as a query-parameter. When the rendition manifest is requested, the response will include segment URLs pointing at the optimal CDN.

Mux Video uses HTTP chunked-transfer-encoding to ensure low time-to-first-byte, and request-collapsing to reduce the number of duplicate requests to the origin in the case of a CDN cache-miss. All CDNs used by Mux support these features and have been tested to deliver optimal performance. We currently use the Fastly and Highwinds (now Stackpath) CDNs, and are looking to add more soon.

LinkLog Processing

Mux Video has already integrated the access logs for multiple CDNs into our usage and billing pipelines. In the case of Fastly, we receive streaming CDN logs over Syslog. And for Highwinds, we pull periodically logs by FTP. All CDN logs are written to Apache Kafka and funnel into our Apache Flink-powered CDN log-processing pipeline. The logs are accumulated into hourly windows for customer usage billing. This shows up as a single number of minutes delivered without customers ever having to worry about which CDN was used to serve the content.

My talk from the Scale by the Bay 2018 conference explains our CDN log processing system in detail.

LinkURL Signing

Security will always be a concern for video content owners. Mux supports signed URLs as a way of restricting access to all types of content. For HTTP Live Streaming (HLS) resources, the signing restrictions must propagate all the way down to the transport-stream (TS) segments. All Mux Video TS segments are signed using CDN-specific URL-signing schemes, and all requests are verified at the CDN edge.

Requiring that all segment requests be signed offers an additional layer of DDOS protection, as only signed requests for cache-misses reach our origin.

LinkFuture Work

We’d like to support mid-stream-switching between CDNs. This ensures automatic and seamless transitions between CDNs in the event of a CDN outage or degraded performance. This is difficult to solve in a way that works across most video players.

Support for additional CDNs is also in the works. An expanded global footprint will provide access to even more POPs for increased performance and reliability.

LinkConclusion

CDNs provide significant value to video content providers and end-users. This value is further amplified by the addition of automatic CDN selection using real-time performance metrics. The multi-CDN capabilities in Mux Video ensure that we’ll continue to provide the best viewing experience for all of our customers and their viewers, no matter where they are in the world.

We encourage you to create a free account to see how Mux Video makes video streaming simple!

Header image is from "Bell Telephone Magazine", circa 1922

Written By

Scott Kidder

Scott Kidder – Sr. Engineering Manager

10 years of video experience. Previously solved hard problems at Brightcove.

Leave your wallet where it is

No credit card required to get started.