Inside Our Fastly Recovery Plan

Anne Burns
Tue, 26 Nov 2019 01:09:20 +0000
Insights, Cms, Site security, Infrastructure, Website structure, Metrics that matter, Best cms 2019, Data protection, Social cms, Data, Best cms, Analytics

The purpose of our disaster recovery plan is to prevent downtimes with Amazon Web Service (AWS) that directly affect site reliability. We pride ourselves on our ability to reliably keep traffic flowing to our client's sites, even in the event of a disruption.

We partner with Fastly — our CDN provider — for our failover plan, which is based on rerouting traffic directly to our servers in case of an outage.

This Is the DNS Configuration for Clients During Normal Operation:

    www.client.com ~> *.cdn.rebelmouse.com ~> rebelmouse.map.fastly.net ~> origin (us-east-1)

When Fastly Is Unavailable, We Update the Configuration to the Following:

    www.client.com ~> *.cdn.rebelmouse.com ~> origin (us-east-1)

SSL Certificates Are Covered By:

SAN certificates provided by Fastly

Our in-house certificates

Origin Configuration

    ELB ~> EC2 instance (nginx ~> varnish ~> nginx ~> uwsgi ~> application)

1. Nginx: SSL termination

2. Varnish: Caching layers that duplicate the varnish configurations implemented for Fastly; it uses the same surrogate keys and the same TTLs

3. Nginx: Connections pool

4. Uwsgi: Application server

Every EC2 instance also has its own caching layer that allows it to effectively handle incoming requests. Fastly is a stable vendor, and we have never needed to move all of the traffic from the CDN to our side. However, last year, we did successfully handle a 25% switchover.

At the moment, we don't have any automatic failover systems. Amazon Route 53 is under the manual control of our operations team.

Data Centers and Points of Presence (PoPs)

Fastly operates an autonomous system with extensive connectivity relationships. This ensures that a single data center does not become reliant on a single network uplink. Because of this, our CDN can degrade gracefully when outages do occur.

While an outage of the central Fastly app may temporarily prevent us from making configuration updates, all of the Points of Presence (PoP) serving our clients' content operate independently. When an outage occurs in a single PoP, other PoPs continue to serve traffic normally. This ensures our users' requests are routed accordingly, and secures our clients' site reliability.

If you have more questions about our Fastly recovery plan, email support@rebelmouse.com or contact your account manager today.