The multi region disaster recovery plan is the document that survives audits and fails incidents, and it fails for reasons this series has catalogued everywhere else, the untested restore, the undocumented dependency, the runbook that no one has executed. The plan's document says the standby region is ready, the data replicates, the traffic fails over, and the first regional outage tests every sentence the document ever contained, usually finding that the DNS record's TTL was theoretical, the database's replica lag was an estimate, and the one service nobody remembered held a regional dependency that the whole failover waited on. This article builds the recovery capability the way the backup and restore drill article built the restore capability, as a rehearsed, measured, evidenced practice, the recovery tiers and their honest costs, the data replication reality across regions, the failover mechanics, the dependency audit that finds the forgotten single region services, and the game day that converts the plan from a document into a capability.
The recovery tiers, what the numbers actually promise
The recovery objectives are the same two numbers the backup article defined, the recovery time objective, how long until the service serves, and the recovery point objective, how much data may be lost, and the multi region tiers are the positions on the price curve those numbers define. The backup and restore tier is the cheapest, the second region holds the backups and the infrastructure as code, the recovery is the rebuild from the artifacts, and the honest numbers are the slowest, the RTO of hours because the rebuild is the whole infrastructure, and the RPO of the backup schedule. The warm standby tier is the middle, the second region runs the infrastructure at reduced capacity, the data replicates continuously, the recovery is the scale up and the traffic switch, the RTO of tens of minutes and the RPO of the replication's lag. The active active tier is the premium, both regions serving traffic always, the recovery is the surviving region absorbing, the RTO of the routing change and the RPO of the replication window, and the premium is paid in the double infrastructure and the cross region consistency engineering that the active active data tier demands.
The tier selection is per service rather than per organization, the same portfolio judgment the architecture and monitoring articles kept applying, the critical path earns the active tier, the important services earn the warm standby, the long tail earns the backup tier, and the tier is written down per service because the alternative is the plan that promises the premium tier to everything and delivers it to nothing.
The data replication reality across the distance
The data layer is the capability's constraint, and the cross region distance is the constraint's physics. The synchronous replication across the regions is the latency line, the cross continent round trip in the tens of milliseconds added to every committed transaction, and the application's own latency budget usually refuses it, which is why the active active data tier is the rarest and the hardest, the regional partitions with the conflict resolution, the CRDTs and the quorum protocols that the distributed systems literature built careers on. The asynchronous replication is the practical default, the standby's data trailing the primary by the distance's lag, the seconds level RPO that the honest plan states as its number, and the failover that loses the trailing window's writes, the acknowledged transactions that the standby never received, which is the RPO the business signed rather than the zero the brochure implied.
# the standby's honest number, the lag the failover will lose
WAL receiver's latest end LSN: 3A4F/2B1C0060
replication lag: 4.2sThe replication's verification is the drill's data half, the checksums across the regions, the counts that the quarterly game day compares, and the replication that the monitor watches is the same lag vocabulary the single region replication article taught, the lag dashboard that says the standby's age, the alert on the growth that says the standby falling behind, and the failover's RPO that the lag dashboard predicts rather than the incident discovering.
The failover mechanics, the minutes the RTO is made of
The failover is a sequence with named steps and honest timings, and the sequence is the RTO's anatomy. The detection comes first, the health signals from the failing region, the monitor that distinguishes the regional outage from the service's own incident, and the detection's minutes are the RTO's first block. The decision comes second, the declaration that the failover proceeds, the human's judgment call or the automated policy that the organization has rehearsed, and the decision's structure is the runbook's first page, who declares, on what evidence, with what authority. The promotion comes third, the standby's data promoted to the primary, the sequence the replication article taught, the fencing that the split brain prevention demands, and the promotion's minutes are the RTO's data block. The traffic switch comes fourth, the DNS or the global load balancer's update, the TTL's honest ceiling, the anycast's instant path, and the switch's minutes are the routing propagation. The warm up comes fifth, the caches, the connection pools, the cold paths that the standby's reduced traffic never exercised, and the warm up's minutes are the performance dip that the users' first requests after the failover feel.
# the RTO's anatomy, the runbook's first page
detection 2m the monitors, the call
decision 3m the declared, the authority
promotion 4m the data promoted, fenced
traffic switch 2m the routing, the TTL
warm up 5m the caches, the first requests
total 16m the promised 15m, rehearsed to the minuteThe sequence's rehearsal is where the minutes become honest, the game day that runs the sequence against the clock, and the rehearsal's findings are the runbook's corrections, the step that the DNS provider's API rate limit slowed, the promotion that the fencing's verification added minutes to, the warm up that the cache's size made longer than the plan's estimate.
The cost curve, what each tier prices at and where the honesty lives
The tiers' price tags deserve their own arithmetic because the tier selection is a budget decision as much as an engineering one. The backup and restore tier prices at the artifacts' storage, the backup retention in the second region's object store and the infrastructure as code repository, the ten percent of the primary's bill that the compliance sign off usually accepts. The warm standby prices at the infrastructure that runs, the reduced capacity fleet, the database replica that holds the whole data, the network that replicates across the distance, and the honest figure sits at the half to the three quarters of the primary's bill, the capacity that serves the reduced traffic after the failover and the failover's first hour at the reduced performance the business accepted.
The active active tier prices at the full second fleet, the double infrastructure that serves real traffic, and the cross region data architecture's engineering, the replication's egress bill, the consistency protocols' latency, and the honest comparison that the tier selection must run, the cost of the premium against the actual RTO difference, the active active's routing minutes against the warm standby's tens of minutes, priced per the business's outage tolerance. The pricing's honest conclusion in most organizations is the portfolio, the critical path's active posture, the important services' warm standby, and the long tail's backups, the same distribution that every tiered capability in this series reached, and the organization that buys the premium tier for everything has usually priced the brochure's zero rather than the business's actual tolerance.
The dependency audit, the single region services that block the failover
The dependency audit is the plan's archaeology, and it is the step that the document's optimism skips, the inventory of what the failover actually depends on and where each dependency lives. The audit's method is the traffic's and the configuration's examination, the DNS records and their zones, the certificate authorities and their revocation services, the third party integrations and their regional endpoints, the queues and the streams and their regional brokers, the serverless functions and their regional runtimes, and each item is classified, regional, global, or failover-ready, and the regional dependency is the failover's blocker that the audit exists to find:
the audit's questions, per dependency
where does it live, which region
what happens when that region goes
what is the failover path, tested whenThe classic findings are the audit's folklore, the S3 bucket in the primary region that the standby's images reference, the OAuth provider's regional endpoint that the authentication needs, the email service that the first failover's own notifications require, and each finding is the single point of failure that the multi region architecture carried invisibly. The audit's fix is the dependency's own failover or its own removal, the global services where the provider offers them, the replicated where the cost justifies, and the audit's repetition is the game day's first hour, because the dependencies drift in with every new integration, the same drift the state and configuration articles kept catching.
The game day, the drill that converts the plan into the capability
The game day is the article's whole argument compressed into a calendar entry, and its shape is the restore drill's shape scaled to the region. The quarterly exercise declares the regional outage, the simulated failure that the team's notification treats as real, and the failover runs against the clock, the sequence's steps executed, the timings recorded, the dependencies found, and the findings ticketed with the same severity the incident's own postmortem would assign. The first game day in every organization's history finds blockers, the audit's misses, the runbook's stale commands, the DNS provider's propagation surprise, and the second game day finds fewer, and the capability is the trend, the game day's timings improving until the failover is the boring sequence the team has run a dozen times.
The game day's honest variations complete the practice, the partial failover, the service's failover without the region's full loss, the degraded mode, the failover with the reduced capacity that the budget's warm standby actually runs, and the failback, the return to the primary region, the direction that the runbook's optimism forgets and the second game day's findings always name. The working rules compress the whole capability, and the list is short:
- Write the tiers per service, the RTO and RPO numbers the business signs, not the brochure's zero;
- Replicate asynchronously and state the lag as the RPO, monitor it with the single region's vocabulary;
- Rehearse the failover's anatomy against the clock, and the runbook's minutes are the rehearsal's findings;
- Run the dependency audit quarterly, the regional dependency is the failover's silent blocker;
- Drill the failback with the failover's seriousness, the return is the second half of the capability;
- Treat the game day's findings with the incident's severity, the finding fixed before the next quarter's drill.
The region will fail, the cloud providers' histories have taught the industry that lesson without any single provider's name needing repeating, and the difference between the organization that recovers and the organization that becomes the outage's case study is the capability that this article built, the tiers priced honestly, the data lag stated openly, the failover rehearsed to the minute, the dependencies audited until they are gone, and the game day that keeps all of it true. The plan is the document, the capability is the drill, and only one of them survives the day it was written for.