Skip to main content

How To Hedge Against Amazon Web Services RDS Outages

Posted on October 23, 2012 by Anthony James Founder/CEOAnthony James Founder/CEO

If you’re familiar with Amazon Web Services, then you know that it is supposed to have enough redundancy at Amazon’s data centers to protect against such an issue. Still it’s possible, as we’ve seen, for Amazon Web Services to have an unavoidable outage. Here’s a few simple methods to hedge against these outages.
Difficulty: Basic

If we take a look at Amazon’s health status, we can see that one of the reasons for the outage was that the RDS (Relational Database Service) went offline.
Screen Shot 2012-10-22 at 7.58.03 PM
This is, however, avoidable depending on your budget and need for redundancy. There are several ways to set up Amazon RDS instances, but for the sake of this, let’s assume your using MySQL on RDS. There are multiple ways to avoid this type of issue, most of which can be used directly with Amazon Web Services.
What’s the root problem we are trying to solve?
AWS had a single point of failure in one of it’s regional “zones” or data centers. If you’re hosting only one MySQL server in one regional zone and that zone fails, then your site/app/network will be unavailable and you’ll be forced to solely rely on the experts at Amazon to get your server back up and running as soon as possible.
Solution 1 (The best option)
By far the best option is a setting used when you create your new RDS instance called Multi-AZ Deployments . This automatically creates replication between two MySQL instances across multiple availability zones. This means that if zone A is down, then zone B will take over. This creates redundancy and high availability for your site. This method uses standard MySQL replication.
The down side is you’re forced to pay for two RDS instances. If you’re a small app/website, this cost can add up over time. However, if you purchase reserved instances, your costs can be as low as $7/m per server per zone. For that type of redundancy, it’s more than worth it.
Solution 2
This solution is a viable option, but not highly suggested because it requires a little bit more labor instead of “automatic fail over”. That said, you can create a MySQL server instance on any running web server you might have or create a MySQL instance on another hosting provider such as Rack Space. Your option here is to manually configure replication from the RDS server to your remote manually set up and configured MySQL server. Depending on how you set up your load balancing and fail over in case of an outage, it will either auto read from the backup or you’ll be forced to manually change the DNS or MySQL address that your code calls on your web server.
The down side is that it takes more hands-on labor to make this happen. In the event of an outage, it will also take more time to restore service to your customers. Depending on the type of service you offer, this could mean loss of sales or advertising dollars on your site.
Even though Amazon Web Services is supposed to be redundant and failure proof, what we’ve seen over the past year is multiple outages taking down many high profile sites and countless smaller sites. We’ve learned that there is no such thing as 100% redundancy out of the box and it is up to you to do your due diligence to determine if and how your servers need high availability. The cost is minimal and the benefit is a good night’s sleep in the event of an issue.


Leave a Reply

Your email address will not be published. Required fields are marked *