Plan for downtime and recovery

Every app goes down eventually. A server dies, a bad deploy ships, someone deletes the wrong database, a whole cloud region has a bad day. The difference between a team that shrugs this off and a team that loses data, users, or the business itself is not whether it happens. It is whether you planned for it before it did. This guide covers the five things that separate a real recovery plan from a hope: backups that are actually restorable, the RPO and RTO numbers that tell you what to build, redundancy so one failure does not take everything down, a runbook and an on-call chain so the right person knows what to do at 3am, and a blameless post-mortem that turns the incident into a fix instead of a repeat performance.

Backups done right

A backup earns its name only after four things are true: it happens automatically, on a schedule nobody has to remember; it is encrypted, both while it is stored and while it moves; it lives somewhere other than the system it is backing up, ideally a different provider or region entirely; and somebody has actually restored it. That fourth one is the part almost everyone skips, and it is the part that matters most. A backup you have never restored is a claim about whether it works, not a fact.

• Runs on a schedule, not a person remembering to click a button.
• Encrypted at rest and in transit, so a stolen backup is not a stolen database.
• Stored off-site, in a different provider or region than the system it protects, so one outage cannot take out both the original and the copy.
• Follows the 3-2-1 pattern: three copies of your data, on two different devices, with one of them off-site.⁴
• Retained across multiple points in time, so a backup taken right after data got corrupted is not your only copy.
• Restored, completely, onto a fresh environment, on a schedule, not just taken and left untouched.

GitLab's January 2017 outage is the canonical case study for why that last item matters. An engineer accidentally deleted the primary database directory while trying to fix a lagging replica, and about 300 GB of production data disappeared in seconds. GitLab had four separate backup and replication mechanisms in place: scheduled dumps to cloud storage, disk snapshots, a cloud provider's snapshot feature, and database replication itself. Three of the four had silently stopped working, in one case for weeks, and nobody had noticed because nobody had tried to restore from them. What saved the company was a manual snapshot an engineer happened to take six hours before the incident, on their own initiative. GitLab still lost about six hours of user data, including thousands of projects and comments.⁵ The company's own postmortem is blunt about the cause: the backup procedure was not tested on a regular basis, because nobody owned that job.⁵

RPO and RTO: how much you can afford to lose, how fast you must be back

Two numbers should drive almost every decision in this guide. Recovery Point Objective, RPO, is how much data you can afford to lose, measured in time: if your last good backup is four hours old when everything goes down, your RPO is four hours. Recovery Time Objective, RTO, is how long you can afford to be down, from the moment things break to the moment users can use the product again. You do not pick these numbers because they sound good. You pick them based on what the business can actually survive, then you build backup frequency, redundancy, and failover to hit them.

A payments API and an internal reporting dashboard do not need the same numbers, and building both to the same standard wastes money on one and risks the business on the other. AWS's own guidance on disaster recovery makes this concrete: how often you back up determines the recovery point you can actually hit, and whether your infrastructure can be redeployed from code, not rebuilt by hand, determines how fast you can hit your recovery time.³ A tighter RPO costs more, since it means more frequent backups or continuous replication. A tighter RTO costs more too, since it means standby infrastructure sitting ready to take over. Set the numbers first, then build the cheapest system that meets them.

Redundancy and failover: no single point of failure

Redundancy means no single component can take the whole system down by failing. A single database server, a single app server, a single region: each one is a single point of failure until you add a second one that can take over. The most common way to catch a failure automatically is a health check, a small, frequent request that asks "are you actually working," not "are you turned on." A server can be running and still fail a health check because it cannot reach the database, or is out of memory, or is returning errors on every request. A health check that only confirms the process is alive misses most of the failures that actually happen.

• Run more than one instance of anything that matters, in more than one location if the budget allows it.
• Health-check on real behavior, a working query, a real response, not just whether the process is running.
• Route traffic through a load balancer or DNS layer that can pull a failing instance out of rotation automatically.³
• Decide in advance whether failover is automatic or manual. Automatic is faster; manual is safer when a false alarm would cost you a healthy, working system.³
• Test the failover itself, not just the health check. A load balancer that correctly detects a dead server but has nowhere healthy left to send traffic has not actually solved anything.

The incident runbook: what to have written down before 3am

At 3am, nobody is at their best. That is exactly when you need a written, boring, step-by-step document more than at any other time. A runbook is not a design doc. It is a checklist: how to tell what is actually broken, where the dashboards and logs live, who has access to what, and the specific commands or buttons for the fixes you already know you might need, a rollback, a failover, a restart of a specific service.

• How to confirm what is actually down, and how badly, in under five minutes.
• Links to the dashboards, logs, and alerting that show real-time state, not tribal knowledge of where to look.
• The exact steps for your most likely failures: roll back a deploy, fail over a database, restart a stuck service.
• Who has access and permission to do each of those things, so the on-call engineer is not stuck waiting on someone else.
• A clear point at which to declare an incident and start the process below, instead of quietly firefighting alone.

On-call and escalation: who gets paged, and when

A runbook only helps if someone reads it. On-call means a specific person is reachable, knows they are on point, and knows what happens if they cannot fix it alone. Google's SRE book, describing what actually goes wrong in unmanaged incidents, is direct about the failure mode: "Nobody knew what actions their coworkers were taking," and that gap compounds every other mistake in the room.¹ The fix is not more people on the call. It is clear roles: one person makes the calls, often called an incident commander, one person actually does the fixing, and one person, only one, talks to anyone outside the incident.¹

• One on-call person per rotation, reachable through a channel that does not depend on the system that is down.
• A defined escalation path: if the first responder cannot fix it in a set window, who gets paged next, automatically.
• One person owns communication during the incident, so updates do not come from five different people saying five different things.¹
• A schedule people actually keep, with backup coverage, so on-call is not one person's permanent, silent burden.

Communicating status to users, honestly

The instinct during an outage is to go quiet until you have a fix. That is close to the worst option. Users who cannot reach your product go looking for information anyway, in your support inbox, on social media, in reviews, and silence reads as either incompetence or dishonesty even when the truth is simply "we are working on it." A status page or a short, plain update, posted early and updated as things change, costs you nothing you were not already going to lose, and buys back a surprising amount of trust.

• Post something as soon as you have confirmed a real, user-facing problem, even before you know the cause.
• Say what you know, what you do not know yet, and roughly when you will update again. Do not guess at a fix time you cannot back up.
• Update on a cadence, even a boring "still working on it," rather than going silent between updates.
• Close the loop publicly when it is fixed, and again, briefly, once the root cause is known.

The blameless post-mortem: turn the incident into a fix

The point of a post-mortem is not to find out who to blame. It is a written record of what happened, its impact, what you did to fix it, the root cause, and the specific follow-up work that will stop it from happening the same way again.² Google's SRE book states the working assumption plainly: everyone involved "had good intentions and did the right thing with the information they had."² That framing matters for a practical reason, not just a nice one. A culture where mistakes get punished is a culture where people hide them, and a hidden mistake cannot be fixed. You cannot fix people. You can fix a system that let a person make an expensive mistake too easily.²

Not every hiccup needs a formal write-up, but decide the criteria in advance so it is not a judgment call made under stress. Reasonable triggers: any user-visible downtime past an agreed threshold, any data loss at all, any incident that required an on-call engineer to intervene by hand, or any incident that took longer to resolve than your stated RTO.² Write it while the details are fresh, assign each follow-up action an owner and a date, and actually track those actions to done. A post-mortem that ends at "root cause identified," with no owned next step, is just a longer way of doing nothing.

Takeaways

None of this is exotic. Automated, encrypted, off-site, restore-tested backups. Real RPO and RTO numbers that match what the business can survive, not just a nice-sounding claim. Redundancy so one failure does not become an outage. A runbook and an on-call chain that work at 3am, not just at 2pm on a Tuesday. Honest, early communication instead of silence. And a blameless post-mortem that turns every incident into a fix that makes the next one less likely. Build this before you need it. The whole point of a downtime plan is that you never want to be improvising it live.

Read next: if you have not run through the rest of the pre-launch list yet, The pre-launch checklist covers the other twelve areas this guide assumes are already in place. And none of this works if you cannot see a failure the moment it happens: Set up error tracking and logs is the practical next step, walking through the alerting and logging that catches a problem before your users do.