Are Your Ads Worth It? - How to Measure the Effect Your Preroll Ads Have on Viewer Experience

We’ve all experienced it before: sitting down in front of your computer to watch the latest news clip, TV episode, or cat video and being frustrated by how long it takes for the player to start playing an ad. It’s bad enough that there’s an ad taking up precious time, but it often feels like an eternity before the ad even starts playing. This delay has a large impact on how we feel watching videos online, and, at its worst, can cause us to spend our time elsewhere.

I’m not advocating for removing ads completely; we live in a world where ads are necessary, and arguably even beneficial to end users (how else can we expect to watch content for free?). However, the experience of watching ads is fraught with technical challenges, a lot of which result in viewer frustration. Until this point, it’s been difficult to measure an ad’s impact on viewer experience, and a focus of ours here at Mux is to help our customers provide their viewers the best experience possible.

Link42IconThe Ins and Outs of Video Advertising (kind of)

Before I dig into the metrics Mux provides to diagnose ad issues, a little bit of technical detail on the world of video advertising is in order. Feel free to skip ahead if you just want to read about how Mux is helping diagnose issues with ad playback.

Video advertising is an immensely complicated technology, with many different systems and components working together to decide which ad to deliver to which user, and then play back that ad.

This is going to be a relatively high-level overview of these systems, components and their interactions, in an attempt to make this comprehensible to most people. As such, some components may not be depicted here; only the most critical are included.

Link42IconThe Simplest Possible Scenario (nothing’s this easy)

FullScreen42Icon

The above diagram shows the simplest client-side ad insertion architecture. The ad-adept of you will note the lack of redirects and waterfall requests, but I’ll get to those shortly. At its core, this workflow for client-side ad insertion includes the following steps:

• Ad request - The video player’s ad integration decides that it is time to play an ad, pauses playback of the content (if it is currently playing), and makes a request to the configured ad server.
  • This typically happens at the beginning of the content (preroll), at a specific timestamp during playback (midroll), or at the end (postroll).
  • The URL for this request is often called an ad tag URL, and includes parameters passed from the client to help the ad server decide which ad to deliver back to the client.
• Ad Response - The primary ad server takes this request and — based on information that it knows about the currently running ad campaigns, the user watching the video, and the playback capabilities of the device — it returns a document that describes what ad or ads to play, and when.
  • This document is typically either a VAST document for single ad placements, or a VMAP document describing all ads to play back across the timeline of a single video. However, certain ad servers may use proprietary formats for these ad requests/responses, rather than using VAST/VMAP.
• Ad Load - The player’s ad integration parses this document to determine the playback URL of the ad, if one exists, and requests the ad in order to play it back.
  • Note: the ad may be hosted by the ad server rather than a different server, but this is not required.
• Ad Playback - Once the player has enough of the ad asset buffered, playback of the asset begins, much in the same way that playback of the content is handled

While those steps may seem simple, let me assure you that they are not. There are many details that are not mentioned here. For example, inserting a different video seamlessly into the middle of another video is difficult anywhere, but some devices make this much harder than others.

In addition, exactly how the ad server decides which ad to return to the player could be the sole subject of a longer post than this one. Even taking those details for granted, you can start to see how inserting an ad into the playback for a video can start to introduce those frustrating delays.

However, it gets worse.

Link42IconMultiple Requests, Waterfalls, and Redirects (more like reality)

FullScreen42Icon

Remember above, where we were dealing in a world where the ad server just “knows” what ad to return to the client? Unfortunately, due to various business and technical reasons, this is not always the case. This leaves us in a world where both the client and ad server may make multiple requests to one or more ad servers, networks, or exchanges. The reasons for this are many, and may include (but are not limited to):

• Targeting rules - wanting to deliver ad X to viewer set A, but ad Y to viewer set B.
• Frequency capping rules - not wanting to deliver the same ad to viewer A multiple times in a row.
• Value optimization - technologies like Real Time Bidding can allow publishers to optimize the money they receive for a given ad slot that is available to a viewer.

The end result is that the single request/single response model is not applicable for a large portion of video ad playback on the internet. Instead, the following scenarios may happen:

• For a given ad request from a client video player, the ad server could make calls to other ad servers or networks to decide which ad to return to the player.
• The ad response that an ad server/network sends back may include a new ad tag URL to be requested, instead of a playable ad. It is the responsibility of the client making the initial ad request to follow these redirects, by requesting the new URL.
• A client video player may have the ability to configure multiple ad tag URLs, to be used in a waterfall, where if the first returns no ad, it moves on to the second with a new request that may follow a very similar path through multiple servers, and on from there.
• Any combination of these, as well as a handful of other less-frequent scenarios.

With all these possible ad requests, even if every one of them completes in milliseconds, you can see how the time can add up. And this is only to inform the player which ad asset to play. Once it knows which ad to play, it may take some time to retrieve enough of the ad to begin playback, further adding to the amount of time you sit waiting for anything to start playing.

Link42IconTechnical Details, VPAID Ads, etc (yup, it gets worse)

I’m not going to go deep on this topic here, but I’d be remiss if I didn’t point out that what I laid out above does not cover everything that makes video ad delivery what it is today. For instance, there are many details about what an ad server does, as well as the difference between ad servers, networks, and exchanges.

One more important thing to call out: the above implicitly assumes that the ad in question is a simple video ad (like an MP4 or WebM file). For interactive ads, this is not the case. The IAB has defined a format called VPAID which, from a layman’s perspective, allows the ad to control the user’s experience on the video player. These VPAID ads may make their own requests for the video to play back, so it’s possible for you to get penalized twice in startup time.

Also, while most advertisers are migrating to using JavaScript VPAID ads, some still require Flash, which may fail outright or may require loading an entire Flash SWF and the Flash runtime, which is not instantaneous, to say the least.

Link42IconHow Can You Tell What’s Happening? - What Mux Can Do For You

Link42IconThe Past

Previously, Mux’s Quality of Experience metrics were what we called “ad aware.” This means that all of the scores and metrics were calculated in a way that ads would be taken into account, but that there were no ad-specific metrics. For instance, Video Startup Time (a.k.a. time-to-first-frame) was calculated as the time it takes from the first attempt of playback to the point when moving video was shown in the player, be that the actual content or the first preroll ad.

This methodology allowed us to provide our customers a holistic view of the overall experience, as we reported the amount of time the viewer waited with a loading spinner at the beginning of playback. This information is useful, for sure, but it fell short on actionability, which is another focus we have here at Mux. Imagine you ran preroll ads on some videos, but not on others; how could you tell exactly the impact your ads had on your viewer experience? There were ways to accomplish this (for instance, using different experiment names for ad vs no ad), but it was not straightforward.

Link42IconThe Present

Keeping the goal of actionability in mind, we chose to make our initial focus on preroll ads and, more specifically, how they affect startup times. In order to do this, we introduced three new metrics under the Startup Time category:

• Video Startup Preroll Request Time
  • Measures the portion of Video Startup Time during which the player making ad tag URL requests. This can also be thought of as the time that it takes for the player to determine which ad to play.
  • To optimize for this, you should work with your ad server to lower ad response times, or limit the number of waterfall/fallback requests that are made to determine the ad to play.
• Video Startup Preroll Load Time
  • Measures the portion of Video Startup Time during which the player is loading the first preroll ad asset.
  • To optimize for this, you should work with the CDN and origin that are responsible for serving your ad assets.
• Requests for First Preroll
  • Measures the number of individual requests that the player made in order to determine which ad it is supposed to play.
  • Note that this will not count any additional requests your ad server makes on behalf of the client.

As a reminder, Video Startup Time is defined as the wall-clock time from when playback is initiated (either by the user or via auto-play) until the first moving frame of either content or ad. So, for a situation where a preroll ad is played:

Video Startup Preroll Request Time + Video Startup Preroll Load Time = Video Startup Time

In order to provide more context for these ad metrics (and all other metrics), we have also included four new dimensions:

• Preroll Ad Asset Hostname - reflects the hostname of the first ad tag URL that was requested, which can be used to pinpoint ad servers that are performing poorly.
• Preroll Ad Tag Hostname - reflects the hostname of the first ad asset that is loaded by the player, which can be used to pinpoint the CDN/origin serving your ads that is underperforming.
• Preroll Requested - reflects whether a preroll ad was requested for this view or not, which can help you understand how much ads are affecting your performance, even if no ad is returned to be played.
• Preroll Played - reflects whether a preroll ad was played (i.e. the player had a preroll to play, and at least tried to load it), which can help you understand how much the delivery of your ad assets may be affecting performance.

Link42IconThe Future

These first ad metrics will enable people to determine the impact ads are having on their overall viewer experience, and guide them towards actions to improve their experience. However, there are many more places that we want to take our ad metrics, including (but not limited to):

• Midroll ad startup times, and ad-to-content resume times
• VPAID vs non-VPAID ad performance
• Ad error rates, and impact of ads to overall playback failures
• Specific ad creatives that have poor performance

Ads are a necessary part of video on the internet, and up until now, there has been a major lack of information about how ads affect the overall viewer experience. Our goal at Mux, as with all of our other metrics, is to provide actionable information on the various aspects of your video architecture, and we believe these metrics are a great start to empowering publishers to make data-driven decisions on how they approach ads.

As always, if you have any questions don’t hesitate to email us at info@mux.com, or if you’re interested, you can starting a free trial of Mux.