Matt Ward

I can’t believe that three years ago was the last time I wrote on our blog and I shared how we are Monitoring and Architecting for Failure at Mux. In that post, I shared a story of a failure in our Mux Data product and some high-level reliability concepts. In this post, I’ll dig into a couple of topics in that post with more details related to two Mux Video outages.

Before we get into those topics, everyone knows that outages suck. We’d rather be transparent and honest and learn from our mistakes than hide incidents when they happen. These are opportunities to learn, and we’re lucky enough to be afforded these opportunities because of our recent growth. As others have helped us get to where we are, we hope that sharing some of what we’ve learned along the way will help you. The following learnings and outages are related to real incidents we’ve had, but this is not a fully detailed public postmortem. See my previous post for questions we ask during postmortems to extract these kinds of learnings and other high-level strategies we consider when designing our systems.

Outage 1 - Connection pools and timeouts

If you have written software that stored state, chances are you have interacted with a database. To interact with that database, you likely used a connection pool to reduce that interaction’s latency. How do they help? Let’s quickly take a look at an example timeline of incoming requests to a connection pool. In this diagram, the lanes (like swimming lanes in a pool) are database connections and our x-axis will represent time. There are four total Each of the boxes represents a request that the pool is handling.

The first request comes in and we have no active connections, so we pay a little overhead to open our connection and start a session with the database. This is the orange bit at the front of the first request.

Another request comes in while the first one is underway, so we need a new connection. We repeat the process: open a connection and start a session.

We receive another request. We have 1 idle connection that we can reuse in our pool so this request saves that bit of overhead and runs a bit faster, even though the actual database query timing is exactly the same.

Traffic is ramping up and now all of our connections are in use. Throughput relies on our  connections freeing up so we can handle new requests.

It happened. Maybe a new user just went live without you knowing what load they could add to your system or maybe it was something else. In this example, our tiny connection pool of 4 was using 2 connections and it just received 3 more requests! There are now more requests in flight than the connection pool can handle.

What happens now?

The application server starts taking on requests that it has to hold on to in memory while it waits for the connection pool to give it a chance to run the database query it needs. The application server starts chewing up RAM holding on to these requests. Is it going to stop? If you configured your requests with timeouts and the load doesn’t continue to build on your server forever, hopefully it will indeed stop and you look at your graphs to find something like we saw:

So what about that outage? Well the timeouts were working and the server didn’t run out-of- memory (OOM). Awesome! Sadly this is not an awesome story. The underlying database queries that were consuming all of the connections never completed. All the requests to this application server just timed out and were returned as errors due to our 99.99 percentile latency increasing. These few slow requests ended up consuming all of the requests in our connection pool because the driver we were using to interact with CockroachDB from our go application didn’t cancel the query to the database. Make sure your query timeouts propagate to your database and actually free the resources there too!

Outage 2 - Timeout. Retry. Give up?

Suppose a server ran out of memory in the middle of handling your request and crashed. Bummer. Maybe a timeout was reached somewhere in your client application and you really needed that reply from the server (video segment maybe?). Slap a retry on it?

Careful.

If this is not done carefully you might end up with a thundering herd that completely overloads your system in sharp intervals.

Retries come in layers in many applications. In video players, the client will frequently retry requests for segments due to the occasional 503 where the CDN to origin network has issues, for example. We’ve seen this kind of network issue coupled with client retires add a ton of load to our application in the past. In order to help our customers’ players retry better, Phil recently explained how a retry can actually go to a different CDN so you can survive everything from the occasional network hiccup to complete CDN failures.

I am not going to go into retry theory because that is well explained by others out there. In this case, our retries were configured well enough such that we were able to recover reasonably quickly without overloading the system.

Quick recoveries don’t cut it in live video streaming. They cut the stream.

Forget about those 99.99% uptime conversations; our customers expect 100% uptime for their live streams. We are always handling live streams. Anyone that is familiar with the concept of measuring reliability will tell you 100% uptime is simply not possible, but we need to do better than 99.99% because our customers do not want their streams to stop for 8 seconds every day. In order to accomplish this, as much as possible, we need to design our systems to tolerate failures at every layer.

In this case, we actually found a layer with a misconfigured timeout. If it had been tuned correctly, the system could have failed open. Failing open would result in a fallback to stale data and maybe some retries in the services downstream of the storage system, but we could have sufficiently prevented the outage and instead only had about a 1 second increase in median latency.

The scenario was a call that downstream systems have a 2 second timeout on. The internal calls must return within a total of 2 seconds or the downstream system will error the call, try to do some retries of that call, and then fail the request. Let’s dig into the details of this complex layered timeout scenario.

In this diagram, the client places a request to our server that has two backends. One backend is critical; we should be able to tolerate the failure of the other. We’ll call that non-critical backend the expediting backend as its job is to speed up the call and reply faster than the parallel call to the critical system. Most of the time the expediting backend does this well, but right now is not one of those times. It’s stalling. The server’s request (1) to the expediting backend takes longer than the server’s request (2) to the critical backend. The expediting backend’s timeout is configured for longer than the 2 second time between the request and the client canceling the request from its own timeout expiring.

Our old timeout values would have made sense if we were dealing with something like this simple “one call to each” situation, but the server isn’t this simple. It has looping calls to the expediting backend based on replies from the critical server. It looks more like this:

The critical backend returns a list of 2 items that should be decorated with data from the expediting backend if available. It is available for call 3, but it is slow. When the first call (3a) runs it exhausts the timeout of the client. The next item that wanted to be decorated by the expediting service (3b) succeeds but it is too late, the client has considered this an error.

Still with us? Let’s dive in even deeper. We can write and solve a formula that expresses this functions’ timeouts: max(call_1, call_2) + (call_2_items * call_3)

In order to correctly solve for the timeout values here, we need to understand call_2_items and then we need to make sure this expression evaluates to < 2 seconds. We effectively used Prometheus histograms to re-tune these timeouts after our recent incident. In this case we look at normal behavior to determine appropriate timeout values and can evaluate what % of requests would succeed for a timeout by using the “Apdex score” approach described in the Prometheus docs.

This PromQL expression helps us understand the % of requests that completed successfully in < 300ms. If this expression is near 100%, then maybe we want to use that 300ms setting for a timeout on the expedite service as it would not result in an unnecessary number of requests being “degraded” by not having the data. In our example, if we use this timeout for all calls then we can support only 5 items being returned as the loop call 3s’ to the expediting service would result in a total time of 1.8 seconds, just beating that client’s timeout and allowing a little bit of headroom for other application bits and pieces as well as potentially network timings.

Note that it makes a lot of sense to align your histogram configurations with your ideal/target timeouts values! Picking these values throughout your application requires careful thought, a little bit of math, a solid monitoring set up, and continuous review. This same kind timeout tuning may also be required in clients that retry calls to backend services. In these cases, rather than looping over response items to place calls, the loop that consumes our request’s time is attempts to get a response. With retries that are following best practices, it is much harder to solve for the timeout values because of the complexity of the function timing formula.

We’re always working to make our systems more reliable and carefully tuning these critical aspects of our system help to bring us closer to our goal of a perceived 100% uptime. If you like working through the details of complex operations and distributed systems problems and are interested in helping us hit that lofty goal, we’re hiring.

I can’t believe that three years ago was the last time I wrote on our blog and I shared how we are 

Monitoring and Architecting for Failure at Mux

. In that post, I shared a story of a failure in our Mux Data product and some high-level reliability concepts. In this post, I’ll dig into a couple of topics in that post with more details related to two Mux Video outages.

Before we get into those topics, everyone knows that outages suck. We’d rather be transparent and honest and learn from our mistakes than hide incidents when they happen. These are opportunities to learn, and we’re lucky enough to be afforded these opportunities because of 

. As others have helped us get to where we are, we hope that sharing some of what we’ve learned along the way will help you. The following learnings and outages are related to real incidents we’ve had, but this is not a fully detailed public postmortem. See my previous post for questions we ask during postmortems to extract these kinds of learnings and other high-level strategies we consider when designing our systems.

If you have written software that stored state, chances are you have interacted with a database. To interact with that database, you likely used a connection pool to reduce that interaction’s latency. How do they help? Let’s quickly take a look at an example timeline of incoming requests to a connection pool. In this diagram, the lanes (like swimming lanes in a pool) are database connections and our x-axis will represent time. There are four total Each of the boxes represents a request that the pool is handling.

The first request comes in and we have no active connections, so we pay a little overhead to open our connection and start a session with the database. This is the orange bit at the front of the first request.

Another request comes in while the first one is underway, so we need a new connection. We repeat the process: open a connection and start a session.

We receive another request. We have 1 idle connection that we can reuse in our pool so this request saves that bit of overhead and runs a bit faster, even though the actual database query timing is exactly the same.

Traffic is ramping up and now all of our connections are in use. Throughput relies on our  connections freeing up so we can handle new requests.

. Maybe a new user just went live without you knowing what load they could add to your system or maybe it was something else. In this example, our tiny connection pool of 4 was using 2 connections and it just received 3 more requests! There are now more requests in flight than the connection pool can handle.

The application server starts taking on requests that it has to hold on to in memory while it waits for the connection pool to give it a chance to run the database query it needs. The application server starts chewing up RAM holding on to these requests. Is it going to stop? If you configured your requests with timeouts and the load doesn’t continue to build on your server forever, hopefully it will indeed stop and you look at your graphs to find something like we saw:

So what about that outage? Well the timeouts were working and the server didn’t run out-of- memory (OOM). Awesome! Sadly this is not an awesome story. The underlying database queries that were consuming all of the connections never completed. All the requests to this application server just timed out and were returned as errors due to our 99.99 percentile latency increasing. These few slow requests ended up consuming all of the requests in our connection pool because the driver we were using to interact with CockroachDB from our go application didn’t 

. Make sure your query timeouts propagate to your database and actually free the resources there too!

Suppose a server ran out of memory in the middle of handling your request and crashed. Bummer. Maybe a timeout was reached somewhere in your client application and you really needed that reply from the server (video segment maybe?). Slap a retry on it?

If this is not done carefully you might end up with a thundering herd that completely overloads your system in sharp intervals.

Retries come in layers in many applications. In video players, the client will frequently retry requests for segments due to the occasional 503 where the 

 to origin network has issues, for example. We’ve seen this kind of network issue coupled with client retires add a ton of load to our application in the past. In order to help our customers’ players retry better, Phil recently explained how a retry can actually go to a different CDN so 

you can survive everything from the occasional network hiccup to complete CDN failures

I am not going to go into retry theory because that is 

. In this case, our retries were configured well enough such that we were able to recover reasonably quickly without overloading the system.

Quick recoveries don’t cut it in live video streaming. They cut the stream.

Forget about those 99.99% uptime conversations; our customers expect 100% uptime for their live streams. We are always handling live streams. Anyone that is familiar with the concept of measuring reliability will tell you 100% uptime is simply not possible, but we need to do better than 99.99% because our customers do not want their streams to 

. In order to accomplish this, as much as possible, we need to design our systems to tolerate failures at every layer.

In this case, we actually found a layer with a misconfigured timeout. If it had been tuned correctly, 

 Failing open would result in a fallback to stale data and maybe some retries in the services downstream of the storage system, but we could have sufficiently prevented the outage and instead only had about a 1 second increase in median latency.

The scenario was a call that downstream systems have a 2 second timeout on. The internal calls must return within a total of 2 seconds or the downstream system will error the call, try to do some retries of that call, and then fail the request. Let’s dig into the details of this complex layered timeout scenario.

A sequence diagram with 2 parallel calls and one that takes too long.

In this diagram, the client places a request to our server that has two backends. One backend is critical; we should be able to tolerate the failure of the other. We’ll call that non-critical backend the 

 as its job is to speed up the call and reply faster than the parallel call to the critical system. Most of the time the expediting backend does this well, but right now is not one of those times. It’s stalling. The server’s request (1) to the expediting backend takes longer than the server’s request (2) to the critical backend. The expediting backend’s timeout is configured for longer than the 2 second time between the request and the client canceling the request from its own timeout expiring.

Our old timeout values would have made sense if we were dealing with something like this simple “one call to each” situation, but the server isn’t this simple. It has looping calls to the expediting backend based on replies from the critical server. It looks more like this:

A sequence diagram with 2 parallel calls then 2 more calls quickly.

The critical backend returns a list of 2 items that should be decorated with data from the expediting backend if available. It is available for call 3, but it is slow. When the first call (3a) runs it exhausts the timeout of the client. The next item that wanted to be decorated by the expediting service (3b) succeeds but it is too late, the client has considered this an error.

Still with us? Let’s dive in even deeper. We can write and solve a formula that expresses this functions’ timeouts: 

max(call_1, call_2) + (call_2_items * call_3)

In order to correctly solve for the timeout values here, we need to understand 

 and then we need to make sure this expression evaluates to < 2 seconds. We effectively used 

 to re-tune these timeouts after our recent incident. In this case we look at normal behavior to determine appropriate timeout values and can evaluate what % of requests would succeed for a timeout by using the “Apdex score” approach described in the Prometheus docs.

sum(rate(http_request_duration_seconds_bucket{le="0.3",job=”expedite”}[5m]))
/
sum(rate(http_request_duration_seconds_count{job=”expedite”}[5m]))

Apdex score query

This PromQL expression helps us understand the % of requests that completed successfully in < 300ms. If this expression is near 100%, then maybe we want to use that 300ms setting for a timeout on the expedite service as it would not result in an unnecessary number of requests being “degraded” by not having the data. In our example, if we use this timeout for all calls then we can support only 5 items being returned as the loop call 3s’ to the expediting service would result in a total time of 1.8 seconds, just beating that client’s timeout and allowing a little bit of headroom for other application bits and pieces as well as potentially network timings.

Note that it makes a lot of sense to align your histogram configurations with your ideal/target timeouts values! Picking these values throughout your application requires careful thought, a little bit of math, a solid monitoring set up, and continuous review. This same kind timeout tuning may also be required in clients that retry calls to backend services. In these cases, rather than looping over response items to place calls, the loop that consumes our request’s time is attempts to get a response. With retries that are following best practices, it is much harder to solve for the timeout values because of the complexity of the function timing formula.

We’re always working to make our systems more reliable and carefully tuning these critical aspects of our system help to bring us closer to our goal of a perceived 100% uptime. If you like working through the details of complex operations and distributed systems problems and are interested in helping us hit that lofty goal, 

Timeout. Let’s try this again. Tuning timeouts and retries at scale.

Everyone knows that outages suck. We’re always learning from them. Let's dive into the details of some changes we made to our system that have helped bring us closer to our goal of a perceived 100% uptime.

If you are lucky enough to have ever run software at scale, then having things fail is probably something you're familiar with. If not, you probably at least remember the first time your external hard drive or flash drive with that super important file failed to be read by your computer. Regardless of your background with hardware or software, you probably have experienced some kind of failure. Did you know exactly when and how that failure happened? Were you prepared for it?

The postmortem process is not a punishment, but rather a mode of learning that you should be excited to participate in.

Maybe this all sounds crazy to you and you are thinking "I run my software in the magical cloud with durable hard drives and live migrating virtual machines." Depending on your situation, you may still be vulnerable to physical failures such as a hard drive head crash or a network fiber cut, but the majority of software outages are actually caused while engineers are taking action on the running systems, frequently during updating or repairing systems.

Before we get too deep into the technical discussion about failure detection and architecture, let's stop for a second and remember that not everyone is YouTube (where I used to work) or Google where minutes of downtimes seriously impacts revenue. If your website is down for 5 minutes, what is the cost of that outage? If the cost is near $0, you probably should spend nearly $0 on your monitoring maybe by using one of the many free uptime check services. When the cost becomes significant you will not only want to consider more complex monitoring systems in order to assess your SLOs and ensure you meet your SLAs, but you should also consider how to design your system to handle transient failures of dependent services.

Mux's Recent Failure

The first full data collection outage I have seen since I joined Mux in August of 2016 occurred on April 19th 2017 around 4:30 PM PST during an upgrade of our API server which was ironically adding additional monitoring capabilities.

A component of that API server which our internal services rely on was misconfigured during the upgrade. This misconfiguration resulted in our data collection agents rejecting all requests once their in-memory caches expired. The cache we had designed to mitigate such outages actually resulted in a slow decay of traffic, as designed.

However, the alerts were configured to only notify in a 0 data collection scenario because the higher values for our "too little data collection" threshold were misbehaving a couple days earlier and had been subsequently disabled. The relationship between these components in our staging environment is not the same as in production, which led to a false positive during our release process. We resolved the issue around 7:30 PM PST on the same day, and are working to improve both the monitoring and reliability of our data collection.

Active and Reactive Monitoring

I admit I can be a bit arrogant when it comes to the awesomeness of the systems I have built, but there is nothing more humbling than having it fail, silently. On multiple occasions, I have been looking at the data from a system I built and suddenly realized it was failing in a way I did not anticipate. I am a human. I make mistakes. Today, eyeballs looking at graphs or logs are frequently better at catching anomalies than algorithms. For this reason, I encourage everyone running a system they feel is very important to keep a literal eyeball on their key metrics. YouTube had TVs showing nearby engineers critical metrics and on more than one occasion a real live human caught a regression in a well known graph before any system did. On the day of our outage, our Raspberry Pi that we were using to show off various metrics via Grafana Playlists was not running and our eyeballs missed a regression.

Active monitoring like that requires human eyeballs looking at graphs 24/7, and that doesn't just suck, it is completely impractical for so many reasons. Reactive monitoring is what most people are relying on these days. This is some software that continually inspects the state of the system, evaluates some (normally human defined) rules which determine if the state of the system is irregular, and then alerts a human if required. The uptime services I mentioned earlier are the simplest example of a reactive monitoring system. More complex systems might look at metrics such as high disk usage and alert a human who will delete unused data. Setting alerting thresholds isn't always obvious or easy. Remembering to set both "too high" and "too low" levels for various metrics is important, but so often it is easy to forget one of these as we did that day. It doesn't matter what monitoring infrastructure you run if you don't have good alerts defined.

Alerting is hard to do right. The only alert that is worse than an alert that doesn't fire is the alert that makes you ask, "is this really a problem?" One technique you can use when defining alerts is to have different tiers. Separate the "this is urgent" alerts which should actually set off a pager from the "you should look at this soon" alerts which can email the whole team. Promote and demote alerts appropriately between the different tiers you define. If an alert that pages someone is being too noisy but it is super critical to operations, consider demoting that alert to just an email instead of deleting it entirely. It actually isn't fair to say we forgot an alert, we had removed an alert that was being too noisy when we probably should have just demoted it.

Learning from Data

After you see an issue in the system, or an alert tells you there is a problem with it, what do you do? Fix it, obviously. Good job. Can I go back to sleep now? Maybe, but did you understand the problem? Do you know how to prevent that problem from happening again? I know some people that dread writing postmortems. The postmortem process is not a punishment, but rather a mode of learning that you should be excited to participate in. I believe that your company is destined for failure the second this tool has become a burden or just another part of "the process" to resolve an outage. Take time to think about what happened and what else could happen given all of that new information you just acquired.

Here's a few things I think about when assessing an outage:

Do we have all the information we need to understand all of the details of what happened and why?

Could we have detected this outage in another way? (Was the alert we received as specific as possible?)

Was the action that needed to be taken obvious to all members of the oncall team?

Can we automate the action that was taken?

What was the root cause of this outage?

Can/should we architect our systems to be able to fail more gracefully in this (and similar) circumstance(s)?

Architecting Against Failure

When you are operating at a certain scale, failure is bound to happen. Through the lens of the postmortem, you become a better architect against possible failures. There is not a one-size-fits-all approach to failure mitigation. Each of the decisions you make, both small and large, should go through a simple cost-benefit analysis in order to assess whether or not you should actually implement them. Here's a few examples from my experiences:

Retry, Backoff and Rate Limit - Sometimes just simply adding a retry will go a long way, but that is dangerous as it may add additional load when the system is under pressure. In that scenario, it is important to have a backoff strategy that is appropriate for your workload such as linear, exponential or other special snowflake rate limiting designs.

Use a Cache - If the server doesn't respond, can you reuse the last response to fulfill the current request? If this is feasible, is the cost of the additional resources worth it?

Add Redundancy - If a single server fails, does the whole system stop? Can you introduce redundancy to eliminate this kind of failure and again, is that cost worth it?

Build a Buffer - Can handling of the data wait? If it can, consider adding a component such as Amazon Kinesis or Apache Kafka to enqueue the data for later processing.

Reconsider Dependencies - Does the system actually need the successful operation of this other component? Can the dependency on the other component be removed entirely or can it be replaced with something that has a more appropriate set of guarantees?

Introduce Isolation - Were a large set of well behaved users affected by a single abuser? Can you isolate the resources that known well behaved users rely on from those which are more likely to encounter abusers? Can you do a better job isolating internal and external traffic?

Improve Test and Release Practices - Sometimes, the hardest part of architecting a system is actually not architecting the system itself, but rather the continual updating of that system. How much value can you get from improved testing and release practices? The variance in both the cost and benefit here is probably the highest.

We hope you learned as much from this as we did. Please reach out to us on Twitter @MuxHQ or email us at info@mux.com with any questions!

If you are lucky enough to have ever run software at scale, then having things fail is probably something you're familiar with. If not, you probably at least remember the first time your external hard drive or flash drive with that super important file failed to be read by your computer. Regardless of your background with hardware or software, you probably have experienced some kind of failure. Did you know exactly when and how that failure happened? Were you prepared for it?

The postmortem process is not a punishment, but rather a mode of learning that you should be excited to participate in.

Maybe this all sounds crazy to you and you are thinking "I run my software in the magical cloud with 

." Depending on your situation, you may still be vulnerable to physical failures such as a hard drive head crash or a network fiber cut, but the majority of software outages are actually caused while engineers are taking action on the running systems, frequently during 

Before we get too deep into the technical discussion about failure detection and architecture, let's stop for a second and remember that not everyone is YouTube (where I used to work) or Google where minutes of downtimes seriously impacts revenue. If your website is down for 5 minutes, what is the cost of that outage? If the cost is near $0, you probably should spend nearly $0 on your monitoring maybe by using one of the 

. When the cost becomes significant you will not only want to consider more complex monitoring systems in order to 

assess your SLOs and ensure you meet your SLAs

, but you should also consider how to design your system to handle transient failures of dependent services.

The first full data collection outage I have seen since I 

 in August of 2016 occurred on April 19th 2017 around 4:30 PM PST during an upgrade of our API server which was ironically adding additional monitoring capabilities.

A component of that API server which our internal services rely on was misconfigured during the upgrade. This misconfiguration resulted in our data collection agents rejecting all requests once their in-memory caches expired. The cache we had designed to mitigate such outages actually resulted in a slow decay of traffic, as designed.

However, the alerts were configured to only notify in a 0 data collection scenario because the higher values for our "too little data collection" threshold were misbehaving a couple days earlier and had been subsequently disabled. The relationship between these components in our staging environment is not the same as in production, which led to a false positive during our release process. We resolved the issue around 7:30 PM PST on the same day, and are working to improve both the monitoring and reliability of our data collection.

I admit I can be a bit arrogant when it comes to the awesomeness of the systems I have built, but there is nothing more humbling than having it fail, 

 On multiple occasions, I have been looking at the data from a system I built and suddenly realized it was failing in a way I did not anticipate. I am a human. I make mistakes. Today, eyeballs looking at graphs or logs are frequently better at catching anomalies than algorithms. For this reason, I encourage everyone running a system they feel is very important to keep a literal eyeball on their key metrics. YouTube had TVs showing nearby engineers critical metrics and on more than one occasion a real live human caught a regression in a well known graph before any system did. On the day of our outage, our Raspberry Pi that we were using to show off various metrics via 

 was not running and our eyeballs missed a regression.

Active monitoring like that requires human eyeballs looking at graphs 24/7, and that doesn't just suck, it is completely impractical for so many reasons. Reactive monitoring is what most people are relying on these days. This is 

 that continually inspects the state of the system, evaluates some (normally human defined) rules which determine if the state of the system is irregular, and then alerts a human if required. The uptime services I mentioned earlier are the simplest example of a reactive monitoring system. More complex systems might look at metrics such as high disk usage and alert a human who will delete unused data. Setting alerting thresholds isn't always obvious or easy. Remembering to set both "too high" and "too low" levels for various metrics is important, but so often it is easy to forget one of these as we did that day. It doesn't matter what monitoring infrastructure you run if you don't have good alerts defined.

Alerting is hard to do right. The only alert that is worse than an alert that doesn't fire is the alert that makes you ask, "

 One technique you can use when defining alerts is to have different tiers. Separate the 

 alerts which should actually set off a pager from the 

 alerts which can email the whole team. Promote and demote alerts appropriately between the different tiers you define. If an alert that pages someone is being too noisy but it is super critical to operations, consider demoting that alert to just an email instead of deleting it entirely. It actually isn't fair to say we forgot an alert, 

we had removed an alert that was being too noisy

 when we probably should have just demoted it.

After you see an issue in the system, or an alert tells you there is a problem with it, what do you do? Fix it, obviously. Good job. Can I go back to sleep now? Maybe, but did you 

 the problem? Do you know how to prevent that problem from happening again? I know some people that dread writing postmortems. 

 I believe that your company is destined for failure the second this tool has become a burden or just another part of "the process" to resolve an outage. Take time to think about what happened and what else could happen given all of that new information you just acquired.

Here's a few things I think about when assessing an outage:

Do we have all the information we need to understand all of the details of what happened and why?

Could we have detected this outage in another way? (Was the alert we received as specific as possible?)

Was the action that needed to be taken obvious to all members of the oncall team?

Can we automate the action that was taken?

Can/should we architect our systems to be able to fail more gracefully in this (and similar) circumstance(s)?

When you are operating at a certain scale, failure is bound to happen. 

Through the lens of the postmortem, you become a better architect against possible failures.

 There is not a one-size-fits-all approach to failure mitigation. Each of the decisions you make, both small and large, should go through a simple cost-benefit analysis in order to assess whether or not you should actually implement them. Here's a few examples from my experiences:

 - Sometimes just simply adding a retry will go a long way, but that is dangerous as it may add additional load when the system is under pressure. In that scenario, it is important to have a backoff strategy that is appropriate for your workload such as linear, exponential or other 

 - If the server doesn't respond, can you reuse the last response to fulfill the current request? If this is feasible, is the cost of the additional resources worth it?

 - If a single server fails, does the whole system stop? Can you introduce redundancy to eliminate this kind of failure and again, is that cost worth it?

 - Can handling of the data wait? If it can, consider adding a component such as 

 to enqueue the data for later processing.

 - Does the system actually need the successful operation of this other component? Can the dependency on the other component be removed entirely or can it be replaced with something that has a more appropriate set of guarantees?

 - Were a large set of well behaved users affected by a single abuser? Can you isolate the resources that known well behaved users rely on from those which are more likely to encounter abusers? Can you do a better job isolating internal and external traffic?

 - Sometimes, the hardest part of architecting a system is actually not architecting the system itself, but rather the continual updating of that system. How much value can you get from improved testing and release practices? The variance in both the cost and benefit here is probably the highest.

We hope you learned as much from this as we did. Please reach out to us on Twitter 

Regardless of your background, you probably have experienced some kind of failure, but did you know exactly when and how that failure happened?

Matt Ward

Timeout. Let’s try this again. Tuning timeouts and retries at scale.

Monitoring and Architecting for Failure at Mux

United States

United Kingdom

Products

Mux Video

Mux Data

Docs & Tools

API SDKs

Company

Learn about video