Multi-Region Architecture skill for Codex

Done well means the design starts from written RTO and RPO targets and a cost ceiling, every data store has a stated replication mode and conflict rule, failover does not depend on anything in the failed region, and the failover has actually been exercised recently. Multi-region roughly doubles infrastructure and operational cost; the plan must say what that buys.

Before designing

1. Read AGENTS.md and the infrastructure code (Terraform, CDK, Pulumi, Helm values): which region-scoped resources exist, what is hard-coded to one region (bucket names, KMS keys, endpoints, IAM, DNS), and where state lives.
2. Get the requirements from the user, in numbers:
   • RTO: how long the service may be down after a region loss.
   • RPO: how much recent data may be lost (seconds of writes).
   • Latency goals by user geography.
   • Data residency obligations (for example EU personal data stays in the EU).
   • Budget tolerance.
3. Inventory every stateful component: databases, caches, queues, object storage, search indexes, secrets, and third-party dependencies that are themselves single-region.

Pick the topology from RTO and RPO

Topology	Typical RTO	Typical RPO	Cost	Use when
Backup and restore to another region	Hours	Hours (last backup)	Low	Internal tools, tolerant workloads
Pilot light (data replicated, compute off)	Tens of minutes to hours	Seconds to minutes	Low to medium	DR compliance on a budget
Warm standby (scaled-down full stack running)	Minutes	Seconds	Medium	Most customer-facing services needing DR
Active-passive hot (full stack, traffic only in primary)	Minutes	Seconds (async)	High	Fast failover without write conflicts
Active-active (both regions serve traffic)	Near zero for reads; writes depend on data model	Zero only with synchronous or consensus writes	Highest	Global latency needs or near-zero RTO, with a data model that supports it

Default recommendation: warm standby or active-passive with asynchronous replication. Go active-active only when the latency or availability requirement cannot be met otherwise and the team accepts the data model work.

Data replication

The speed of light sets the trade-off: synchronous cross-region writes add at least one inter-region round trip (roughly 60 to 150 ms between continents) to every commit.

Approach	Writes	RPO	Conflicts	Examples
Single primary, async replicas	One region	Seconds of replica lag	None (one writer)	Postgres/MySQL cross-region replicas, Aurora Global Database (secondaries read-only; write forwarding available)
Multi-writer, async, last-writer-wins	Any region	Seconds	Resolved by timestamp; concurrent updates lost	DynamoDB global tables (default mode), Cosmos DB multi-region writes
Consensus / synchronous	Any region, quorum commit	Zero	None	Spanner, CockroachDB, Aurora DSQL, DynamoDB global tables multi-Region strong consistency mode
Home region per entity	Each user or tenant written in one region	Seconds for that entity	Avoided by design	Any store; routing layer sends writes to the home region

Check the current documentation for each managed service’s limits (region counts, supported failover modes, consistency options); these change between releases.

Conflict handling rules for multi-writer stores:

• Last-writer-wins silently drops one of two concurrent updates. Acceptable for profiles and preferences; not for balances, inventory, or anything counted.
• For counters and sets, use commutative operations (increments, CRDT-style merges) or route those writes to one home region.
• Uniqueness (usernames, email addresses) cannot be enforced across async regions; assign a global owner region for uniqueness checks or use a consensus store.

Traffic routing and failover

• Global entry: DNS with health checks (Route 53 failover or latency routing, Cloud DNS routing policies), anycast load balancers (Google Cloud global load balancing, AWS Global Accelerator, Cloudflare, Azure Front Door).
• DNS failover is bounded by TTL and by clients that ignore it; use TTLs of 60 s or less for failover records and expect a tail of stale clients.
• Static stability: failover must not require the failed region’s control plane (its APIs, its IAM, its CI runners, its secrets store). Pre-provision capacity or keep scaling in the surviving region’s control plane.
• Avoid split brain: promoting a replica while the old primary still accepts writes creates divergent data. Fence the old primary (revoke its writers, block it at the network or routing layer) before promoting.
• Decide who triggers failover. Automatic failover on flapping health checks causes more outages than it prevents; a common choice is automated detection with a human-approved promotion runbook for the database.

Executing a failover, promoting a replica or shifting production traffic is a production-impacting action: present the runbook and wait for the user’s explicit go ahead; never run it on your own initiative.

Cross-cutting checks

• Region-scoped resources: KMS keys, secrets, container images, buckets and certificates must exist in every serving region (replicate images and secrets).
• Caches and queues are usually regional; decide whether to rebuild them on failover (cold cache load) or replicate them.
• Background jobs and cron: exactly one region should run singleton jobs; failover must move them without running them twice.
• Third-party dependencies hosted in one region are a hidden single point of failure.
• Cost: inter-region data transfer is billed per GB; replication of chatty write workloads and cross-region service calls add up fast. Keep request paths in-region.
• Residency: route EU users to EU regions and keep their data there, including backups, logs and analytics copies. Have legal or compliance review residency claims.

Workflow checklist

- [ ] Written RTO, RPO, latency and residency requirements
- [ ] Inventory stateful components and region-pinned resources
- [ ] Choose topology; justify against cost
- [ ] Per data store: replication mode, conflict rule, failover procedure
- [ ] Routing design: entry point, health checks, TTLs, fencing of old primary
- [ ] Remove dependencies on the failed region from the failover path
- [ ] Runbook with decision owner and steps; observability per region
- [ ] Failover exercise in staging, then a scheduled game day with approval

Verification

• Replication lag metric per store, alerting above the RPO.
• Failover exercise in a non-production environment: measured time to recover against RTO, measured data loss against RPO, and confirmation that nothing called the “failed” region (block it with network rules for the test).
• Failback exercise too; returning to the primary region is often the riskier half.
• Re-run at a fixed cadence (quarterly is common); untested failover usually fails.

Gotchas

• Hard-coded regional endpoints in application config or SDK defaults bypass all the routing design.
• Sessions and auth tokens signed with region-local keys stop working after failover.
• Read-after-write across regions: a user writes in region A, the next request lands in region B and reads stale data. Pin a session to a region or read from the primary for a short window after writes.
• Capacity: the surviving region must absorb all traffic; warm standby at 20% size needs a tested scale-up path and quota headroom.

Output template

## Multi-region design: <service>
**Requirements:** RTO 15 min, RPO 30 s, EU data in EU, budget +60%
**Topology:** warm standby, primary eu-west-1, standby eu-central-1
| Store | Replication | Conflict rule | Failover action | Expected RPO |
|---|---|---|---|---|
| Postgres (Aurora) | Global Database, async | single writer | managed failover after fencing | under 1 s typical lag |

**Routing:** <entry point, health checks, TTL>
**Failover trigger and owner:** <who decides, runbook link>
**Not covered:** <single-region dependencies, accepted risks>
**Test plan:** <exercise, date, success criteria>