Reliability
Backup & Disaster Recovery — RPO, RTO, and a restore you have tested
Set recovery targets (RPO/RTO), keep retained and immutable backups, pick a DR strategy by cost vs RTO, and run a database restore to a NEW instance that you actually verify before cutting over.
Quick reference — reach for these first
RPO = max acceptable DATA LOSSHow far back your last good backup may be. Drives backup/replication frequency, not speed.RTO = max acceptable DOWNTIMEHow long recovery may take. Drives the DR strategy you pay for, not how often you back up.Object Lock / vault = can't deleteIsolated, write-once copy in another account/region. Survives ransomware and a compromised prod account.restore -> verify -> discardAn untested backup is not a backup. Restore to a NEW target on a schedule and check the data.backup-restore < pilot-light < warm-standby < multi-siteCost rises and RTO falls left to right. Pick the cheapest tier that still meets the RTO.RPO vs RTO around an incident, and the four DR strategies by cost
Text equivalent: A timeline centered on an INCIDENT. To the left, the gap back to the last good backup is the RPO — the data you lose. To the right, the gap forward to service restored is the RTO — the time to recover. Below, four DR strategies run cheapest-and-slowest to costliest-and-fastest: Backup & Restore (RTO hours, cost $), Pilot Light (RTO tens of minutes, $$), Warm Standby (RTO minutes, $$$), and Multi-site Active/Active (RTO seconds, $$$$). Pick the cheapest tier that still meets the RTO.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| Set RPO/RTO per service tier in the runbook | printf 'tier-1 (payments): RPO=5m RTO=30m\ntier-3 (reports): RPO=24h RTO=24h\n' >> dr-runbook.md | Every service maps to a tier with explicit numeric RPO and RTO targets. |
| Take an on-demand snapshot before a risky change | aws rds create-db-snapshot --db-instance-identifier PROD_DB --db-snapshot-identifier PROD_DB-pre-migration | describe-db-snapshots shows Status=available before you proceed. |
| Enable automated backups / point-in-time recovery | aws rds modify-db-instance --db-instance-identifier PROD_DB --backup-retention-period 14 --apply-immediately | BackupRetentionPeriod=14 and a LatestRestorableTime within ~5 min of now. |
| Tier and expire old backups with a lifecycle policy | aws s3api put-bucket-lifecycle-configuration --bucket BACKUP_BUCKET --lifecycle-configuration file://lifecycle.json | Transition + Expiration days match your retention policy; nothing expires inside the required window. |
| Make backups immutable (Object Lock / WORM) | aws s3api put-object-retention --bucket BACKUP_BUCKET --key KEY --retention 'Mode=COMPLIANCE,RetainUntilDate=2026-12-31T00:00:00Z' | A delete attempt in the window is denied — ransomware and a bad admin can't erase it. |
| Copy a backup to a DR region | aws rds copy-db-snapshot --source-db-snapshot-identifier SNAPSHOT_ARN --target-db-snapshot-identifier DR-COPY --source-region us-east-1 --region us-west-2 --kms-key-id DR_KMS_KEY | Copy exists in the DR region AND the DR-region KMS key is usable there — a region outage can't take both copies. |
| Share a copy to an isolated vault account | aws rds modify-db-snapshot-attribute --db-snapshot-identifier SNAPSHOT_ID --attribute-name restore --values-to-add VAULT_ACCOUNT_ID | A full compromise of the prod account can't delete the vault copy. |
| List restore points newest-first | aws rds describe-db-snapshots --db-instance-identifier PROD_DB --query 'reverse(sort_by(DBSnapshots,&SnapshotCreateTime))[].[DBSnapshotIdentifier,SnapshotCreateTime,Status]' --output table | Pick the newest snapshot taken BEFORE the corruption time — not simply the latest. |
| Restore a snapshot to a NEW instance (drill) | aws rds restore-db-instance-from-db-snapshot --db-instance-identifier RESTORE-TEST --db-snapshot-identifier SNAPSHOT_ID | Target is a NEW id, never prod; Status reaches available. |
| Point-in-time restore to a chosen second | aws rds restore-db-instance-to-point-in-time --source-db-instance-identifier PROD_DB --target-db-instance-identifier PITR-TEST --restore-time 2026-11-14T02:10:00Z | restore-time is <= LatestRestorableTime and inside the RPO — before the damage, not after. |
| Verify restored-data integrity | psql -h RESTORE_HOST -c 'SELECT count(*) FROM orders;' -c 'SELECT md5(string_agg(id::text,\',\' ORDER BY id)) FROM orders;' | Row counts and checksums match a known-good baseline, not just that the DB boots. |
| LABEL (CHANGES PROD) — cut over to the restored DB | aws route53 change-resource-record-sets --hosted-zone-id ZONE --change-batch file://failover.json | App resolves to the restored endpoint; a test write lands there, not on the corrupt instance. |
| LABEL (DELETES DATA) — tear down the drill instance | aws rds delete-db-instance --db-instance-identifier RESTORE-TEST --skip-final-snapshot | Only the TEST id is targeted; prod and all retained snapshots are untouched. |
Worked example — restore a corrupted production database
RESTORE IT
aws rds describe-events --source-identifier PROD_DB --source-type db-instance --duration 120
# migration v42 applied 02:15 UTC -> target a point just BEFORE itaws rds describe-db-instances --db-instance-identifier PROD_DB --query 'DBInstances[0].LatestRestorableTime'
# "2026-11-14T02:16:30Z" -> PITR to 02:10:00Z is valid and pre-damageaws rds restore-db-instance-to-point-in-time --source-db-instance-identifier PROD_DB --target-db-instance-identifier PROD_DB-restore --restore-time 2026-11-14T02:10:00Z
# Status: creating -> available (~15 min)psql -h PROD_DB-restore... -c 'SELECT count(*) FROM orders;' && curl -fsS https://app-staged/healthz
# orders=184,502 matches last night's report; /healthz 200; checksums match baselineaws rds modify-db-instance --db-instance-identifier PROD_DB --new-db-instance-identifier PROD_DB-corrupt --apply-immediately
# then rename PROD_DB-restore -> PROD_DB (or flip DNS); app now reads/writes the restored dataTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Restore runs for hours and blows the RTO | Time a full restore drill end-to-end and note the data size | Actual restore throughput vs the RTO target | Move that tier to warm-standby/pilot-light, snapshot more granularly, or pre-provision faster storage |
| Backup job succeeds but the restore fails or data is incomplete | Actually restore it and run row-count + checksum checks | Backup is genuinely restorable vs silently corrupt | Fix the backup job, add an integrity check to every run, and keep the last known-good copy |
| Restored data is stale — older than the RPO allows | Compare the snapshot/restore timestamp to the incident time | Backup frequency vs the RPO target | Increase snapshot/PITR frequency or add continuous replication for that tier |
| Can't restore in the DR region or isolated account | Confirm the snapshot copy AND a usable KMS key exist in the target | Cross-region/account copy + key access are in place | Copy snapshots with a DR-region KMS key and grant the vault account restore access before you need it |
| After failback, both sites accept writes (split-brain) | Confirm exactly ONE primary accepts writes and check replication direction | A single source of truth during failback | Fence the old primary, reverse replication one way, then cut over in a single direction only |
References
- AWS — Disaster recovery options in the cloud (DR whitepaper)docs.aws.amazon.com
- AWS RDS — Restoring a DB instance to a specified timedocs.aws.amazon.com
- Google Cloud — Disaster recovery planning guidecloud.google.com
- Azure — About Site Recovery (disaster recovery)learn.microsoft.com
- NIST SP 800-34 Rev. 1 — Contingency Planning Guidecsrc.nist.gov
Examples use the AWS RDS / S3 CLI (aws-cli v2) with PostgreSQL as a stand-in stack, but the RPO/RTO, retention, immutability, and DR-tier ideas are cloud-agnostic. PROD_DB, BACKUP_BUCKET, SNAPSHOT_ID, VAULT_ACCOUNT_ID, DR_KMS_KEY, ZONE, and ARNs are CAPS placeholders — replace with your own. docs-checked against vendor documentation, not run against a live account.