← All cheatsheets

Reliability

Backup & Disaster Recovery — RPO, RTO, and a restore you have tested

Set recovery targets (RPO/RTO), keep retained and immutable backups, pick a DR strategy by cost vs RTO, and run a database restore to a NEW instance that you actually verify before cutting over.

Docs-checkedbackupdisaster-recoveryrportorestore

Quick reference — reach for these first

RPORPO = max acceptable DATA LOSSHow far back your last good backup may be. Drives backup/replication frequency, not speed.
RTORTO = max acceptable DOWNTIMEHow long recovery may take. Drives the DR strategy you pay for, not how often you back up.
immutableObject Lock / vault = can't deleteIsolated, write-once copy in another account/region. Survives ransomware and a compromised prod account.
drillrestore -> verify -> discardAn untested backup is not a backup. Restore to a NEW target on a schedule and check the data.
strategybackup-restore < pilot-light < warm-standby < multi-siteCost rises and RTO falls left to right. Pick the cheapest tier that still meets the RTO.

RPO vs RTO around an incident, and the four DR strategies by cost

RPO and RTO around an incident, and the four DR strategies by cost and recovery time A timeline shows RPO as the data-loss window from the last good backup to the incident, and RTO as the downtime from the incident until service is restored. Below, four strategies run from cheap and slow (backup and restore) to costly and near-instant (multi-site active/active). last good backup INCIDENT service restored RPO · data you lose RTO · time to recover Backup & Restore RTO: hours cost $ Pilot Light RTO: 10s of min cost $$ Warm Standby RTO: minutes cost $$$ Multi-site Active/Active RTO: ~seconds cost $$$$ cost up, RTO down — choose the cheapest tier that still meets the RTO

Text equivalent: A timeline centered on an INCIDENT. To the left, the gap back to the last good backup is the RPO — the data you lose. To the right, the gap forward to service restored is the RTO — the time to recover. Below, four DR strategies run cheapest-and-slowest to costliest-and-fastest: Backup & Restore (RTO hours, cost $), Pilot Light (RTO tens of minutes, $$), Warm Standby (RTO minutes, $$$), and Multi-site Active/Active (RTO seconds, $$$$). Pick the cheapest tier that still meets the RTO.

Common tasks

Task · Command · What to check

TaskCommandWhat to check
Set RPO/RTO per service tier in the runbookprintf 'tier-1 (payments): RPO=5m RTO=30m\ntier-3 (reports): RPO=24h RTO=24h\n' >> dr-runbook.mdEvery service maps to a tier with explicit numeric RPO and RTO targets.
Take an on-demand snapshot before a risky changeaws rds create-db-snapshot --db-instance-identifier PROD_DB --db-snapshot-identifier PROD_DB-pre-migrationdescribe-db-snapshots shows Status=available before you proceed.
Enable automated backups / point-in-time recoveryaws rds modify-db-instance --db-instance-identifier PROD_DB --backup-retention-period 14 --apply-immediatelyBackupRetentionPeriod=14 and a LatestRestorableTime within ~5 min of now.
Tier and expire old backups with a lifecycle policyaws s3api put-bucket-lifecycle-configuration --bucket BACKUP_BUCKET --lifecycle-configuration file://lifecycle.jsonTransition + Expiration days match your retention policy; nothing expires inside the required window.
Make backups immutable (Object Lock / WORM)aws s3api put-object-retention --bucket BACKUP_BUCKET --key KEY --retention 'Mode=COMPLIANCE,RetainUntilDate=2026-12-31T00:00:00Z'A delete attempt in the window is denied — ransomware and a bad admin can't erase it.
Copy a backup to a DR regionaws rds copy-db-snapshot --source-db-snapshot-identifier SNAPSHOT_ARN --target-db-snapshot-identifier DR-COPY --source-region us-east-1 --region us-west-2 --kms-key-id DR_KMS_KEYCopy exists in the DR region AND the DR-region KMS key is usable there — a region outage can't take both copies.
Share a copy to an isolated vault accountaws rds modify-db-snapshot-attribute --db-snapshot-identifier SNAPSHOT_ID --attribute-name restore --values-to-add VAULT_ACCOUNT_IDA full compromise of the prod account can't delete the vault copy.
List restore points newest-firstaws rds describe-db-snapshots --db-instance-identifier PROD_DB --query 'reverse(sort_by(DBSnapshots,&SnapshotCreateTime))[].[DBSnapshotIdentifier,SnapshotCreateTime,Status]' --output tablePick the newest snapshot taken BEFORE the corruption time — not simply the latest.
Restore a snapshot to a NEW instance (drill)aws rds restore-db-instance-from-db-snapshot --db-instance-identifier RESTORE-TEST --db-snapshot-identifier SNAPSHOT_IDTarget is a NEW id, never prod; Status reaches available.
Point-in-time restore to a chosen secondaws rds restore-db-instance-to-point-in-time --source-db-instance-identifier PROD_DB --target-db-instance-identifier PITR-TEST --restore-time 2026-11-14T02:10:00Zrestore-time is <= LatestRestorableTime and inside the RPO — before the damage, not after.
Verify restored-data integritypsql -h RESTORE_HOST -c 'SELECT count(*) FROM orders;' -c 'SELECT md5(string_agg(id::text,\',\' ORDER BY id)) FROM orders;'Row counts and checksums match a known-good baseline, not just that the DB boots.
LABEL (CHANGES PROD) — cut over to the restored DBaws route53 change-resource-record-sets --hosted-zone-id ZONE --change-batch file://failover.jsonApp resolves to the restored endpoint; a test write lands there, not on the corrupt instance.
LABEL (DELETES DATA) — tear down the drill instanceaws rds delete-db-instance --db-instance-identifier RESTORE-TEST --skip-final-snapshotOnly the TEST id is targeted; prod and all retained snapshots are untouched.

Worked example — restore a corrupted production database

RESTORE IT

1
A bad migration corrupted orders at ~02:15 UTC. Freeze writes and find exactly when the damage started.
aws rds describe-events --source-identifier PROD_DB --source-type db-instance --duration 120
# migration v42 applied 02:15 UTC -> target a point just BEFORE it
2
Confirm you can restore to a point inside your RPO and before the damage.
aws rds describe-db-instances --db-instance-identifier PROD_DB --query 'DBInstances[0].LatestRestorableTime'
# "2026-11-14T02:16:30Z" -> PITR to 02:10:00Z is valid and pre-damage
3
LABEL (safe: NEW target) — restore to a brand-new instance. Never overwrite prod blindly.
aws rds restore-db-instance-to-point-in-time --source-db-instance-identifier PROD_DB --target-db-instance-identifier PROD_DB-restore --restore-time 2026-11-14T02:10:00Z
# Status: creating -> available (~15 min)
4
Verify BOTH application recovery and data integrity before trusting it.
psql -h PROD_DB-restore... -c 'SELECT count(*) FROM orders;' && curl -fsS https://app-staged/healthz
# orders=184,502 matches last night's report; /healthz 200; checksums match baseline
5
LABEL (CHANGES PROD) — cut over. Rename the corrupt instance aside, promote the restore, repoint the app.
aws rds modify-db-instance --db-instance-identifier PROD_DB --new-db-instance-identifier PROD_DB-corrupt --apply-immediately
# then rename PROD_DB-restore -> PROD_DB (or flip DNS); app now reads/writes the restored data
✓
Smoke-test a real user flow (place one order, read it back) and watch the error rate return to baseline. A green flow plus matching row counts confirms recovery met both RTO and RPO. Keep PROD_DB-corrupt until the postmortem is done, then delete it.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Restore runs for hours and blows the RTOTime a full restore drill end-to-end and note the data sizeActual restore throughput vs the RTO targetMove that tier to warm-standby/pilot-light, snapshot more granularly, or pre-provision faster storage
Backup job succeeds but the restore fails or data is incompleteActually restore it and run row-count + checksum checksBackup is genuinely restorable vs silently corruptFix the backup job, add an integrity check to every run, and keep the last known-good copy
Restored data is stale — older than the RPO allowsCompare the snapshot/restore timestamp to the incident timeBackup frequency vs the RPO targetIncrease snapshot/PITR frequency or add continuous replication for that tier
Can't restore in the DR region or isolated accountConfirm the snapshot copy AND a usable KMS key exist in the targetCross-region/account copy + key access are in placeCopy snapshots with a DR-region KMS key and grant the vault account restore access before you need it
After failback, both sites accept writes (split-brain)Confirm exactly ONE primary accepts writes and check replication directionA single source of truth during failbackFence the old primary, reverse replication one way, then cut over in a single direction only

References

Examples use the AWS RDS / S3 CLI (aws-cli v2) with PostgreSQL as a stand-in stack, but the RPO/RTO, retention, immutability, and DR-tier ideas are cloud-agnostic. PROD_DB, BACKUP_BUCKET, SNAPSHOT_ID, VAULT_ACCOUNT_ID, DR_KMS_KEY, ZONE, and ARNs are CAPS placeholders — replace with your own. docs-checked against vendor documentation, not run against a live account.