← All cheatsheets

Reliability

AWS SRE — the first 15 minutes of an incident

A triage runbook for the first 15 minutes of an AWS incident: confirm which account and Region you are in, size the blast radius, read the four golden signals in CloudWatch, check ALB/NLB target health, correlate recent changes in CloudTrail, then decide mitigate vs roll back — measured against the affected SLI. Inspect-first; prod-changing steps are labeled.

Docs-checkedawssreincidentcloudwatchcloudtrail

Quick reference — reach for these first

Who/where am Iaws sts get-caller-identity --profile PROFILE; aws configure get regionConfirm the account and Region before touching anything. Wrong account is the most common wasted first move.
Blast radiusaws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGIONIs it one service, one AZ, or everything? Edge 5xx over the last 30 min sizes the impact.
Target healthaws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION --query 'TargetHealthDescriptions[].{t:Target.Id,s:TargetHealthDescription:'' ,state:TargetHealth.State,why:TargetHealth.Reason}'Are backends actually in service? Draining/unhealthy targets explain most 5xx and timeouts.
Recent changesaws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService --max-results 15 --region REGIONCorrelate the spike to a deploy, scaling, or IAM change. 'What changed?' answers most incidents.
Backend errorsaws logs start-query --log-group-name /ecs/SERVICE --start-time $(date -u -v-15M +%s) --end-time $(date -u +%s) --query-string 'fields @timestamp,@message | filter @message like /ERROR|Exception|5\\d\\d/ | sort @timestamp desc | limit 50' --region REGIONLogs Insights returns a queryId; poll get-query-results for the actual failing requests.

Incident triage flow — the first 15 minutes

AWS incident triage: the first 15 minutesSix ordered steps flow left to right: confirm account and Region, establish blast radius, read the four golden signals, check load-balancer and target health, correlate recent changes, then decide mitigate or roll back measured against the affected SLI.1 Account+ Region2 Blastradius3 Goldensignals4 LB / targethealth5 Recentchanges6 Mitigate /roll backVerify vs SLI: errors to baseline, SLO burn stoppedEvery step is measured against the affected SLI (error rate / latency / availability)

Text equivalent: Six ordered steps flow left to right: 1 confirm account and Region, 2 establish blast radius, 3 read the four golden signals (latency, traffic, errors, saturation), 4 check load-balancer and target health, 5 correlate recent changes in CloudTrail, 6 decide mitigate or roll back. A loop from step 6 points to a verify box: every step and the final decision are measured against the affected SLI.

Common tasks

Task · Command · What to check

TaskCommandWhat to check
1. Confirm account + Region (read-only)aws sts get-caller-identity --profile PROFILE && aws configure listAccount matches the impacted prod account and Region is the one paging you — not a default/sandbox.
2. Pin down the affected service/stackaws elbv2 describe-load-balancers --names LB_NAME --region REGION --query 'LoadBalancers[0].{dns:DNSName,state:State.Code}'You know the exact load balancer, target group, and ECS/EKS service that fronts the paging alarm.
3. Establish blast radius (error rate + scope)aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGIONError volume and start time are known; compare per-AZ to tell 'whole service' from 'one zone'.
4. Golden signal — latencyaws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics p99 Average --region REGIONp99 TargetResponseTime trend seen; a latency cliff at the start time points at the backend.
5. Golden signal — trafficaws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name RequestCount --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGIONDistinguish a demand spike from a drop-to-zero (dependency/edge outage) vs steady traffic failing.
6. Golden signal — errors (edge vs backend)for m in HTTPCode_ELB_5XX_Count HTTPCode_Target_5XX_Count; do aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name $m --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-15M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION; doneELB_5XX (LB itself: no healthy target) vs Target_5XX (app returned 5xx) splits LB problem from app problem.
7. Golden signal — saturationaws cloudwatch get-metric-statistics --namespace AWS/ECS --metric-name CPUUtilization --dimensions Name=ClusterName,Value=CLUSTER Name=ServiceName,Value=SERVICE --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Average Maximum --region REGIONCPU/memory near 100%, or TargetConnectionErrorCount rising, indicates capacity — scale, don't roll back.
8. Load balancer / target healthaws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGIONCount of healthy vs unhealthy targets and the Reason (e.g. Target.FailedHealthChecks) is recorded.
9. Backend application logsaws logs get-query-results --query-id QUERY_ID --region REGIONActual error messages/stack traces from the window are in hand (start-query from quickRef gives QUERY_ID).
10. Network path — timeouts (read-only)aws ec2 describe-route-tables --filters Name=vpc-id,Values=VPC_ID --region REGION --query 'RouteTables[].Routes'; aws ec2 describe-flow-logs --filter Name=resource-id,Values=VPC_ID --region REGIONRoutes to NAT/IGW intact; flow logs show ACCEPT not REJECT to the dependency port.
11. Correlate recent changes (CloudTrail)aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService --max-results 20 --region REGION --query 'Events[].{t:EventTime,u:Username,e:EventName}'A deploy, scaling, IAM, or security-group change lines up with the incident start time (or rules change out).
12. Deployment status — ECS/EKSaws ecs describe-services --cluster CLUSTER --services SERVICE --region REGION --query 'services[0].deployments' # EKS: kubectl rollout status deployment/NAME -n NSDeployment rolloutState (IN_PROGRESS/FAILED) and running vs desired count show whether a bad release is live.
13. Decide: mitigate vs roll back — against the SLI# compare current error-rate/latency to the SLO target before actingIf a deploy caused it and SLO is burning: roll back. If capacity: scale. If dependency: shed/failover.
14. LABEL: roll back the deploy (prod change)aws ecs update-service --cluster CLUSTER --service SERVICE --task-definition FAMILY:PREV_REVISION --region REGION # EKS: kubectl rollout undo deployment/NAME -n NSPROD CHANGE — confirm PREV_REVISION is the last-known-good; watch Target_5XX and p99 return to baseline.

HTTP 5xx errors spike right after a deployment

5xx after deploy

1
Confirm you are in the impacted account and Region before anything else.
aws sts get-caller-identity --profile PROFILE; aws configure get region
# Account 1234567890 / us-east-1 — correct prod account
2
Split edge vs backend 5xx to localize the fault.
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-20M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION
# Target_5XX jumps from ~0 to 4k/min; ELB_5XX stays low -> the app is returning 5xx
3
Check target health — are backends even passing checks?
aws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION
# 6/10 targets unhealthy, Reason Target.ResponseCodeMismatch
4
Pull the actual errors from the new tasks via Logs Insights.
aws logs start-query --log-group-name /ecs/SERVICE --start-time $(date -u -v-15M +%s) --end-time $(date -u +%s) --query-string 'fields @timestamp,@message | filter @message like /Exception|5\\d\\d/ | sort @timestamp desc | limit 20' --region REGION
# QueryId f1a2... -> get-query-results shows NullPointer on a new config key
5
Correlate to the change — find the deploy that started it.
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService --max-results 5 --region REGION
# UpdateService 3 min before the spike, task def FAMILY:48 -> prior good is :47
6
Measure against the SLI: error budget burning fast, cause is the deploy -> roll back, don't debug live.
!
LABEL — PROD CHANGE: roll the ECS service back to the last-known-good revision.
aws ecs update-service --cluster CLUSTER --service SERVICE --task-definition FAMILY:47 --region REGION
# deployment IN_PROGRESS, rolloutState rolling back to :47
✓
Verify recovery against the SLI before standing down.
aws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION; aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-5M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION
# 10/10 targets healthy, Target_5XX back to ~0 — error budget burn stopped

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
504 Gateway Timeout from the ALBaws elbv2 describe-target-health + TargetResponseTime p99 and TargetConnectionErrorCountWhether backends are slow/overloaded (target responding late) vs the LB can't connect at all (SG/route/port).Slow backend: scale or roll back the slow release. No connection: fix security group / target port / health-check path.
Intermittent DNS resolution failures to a dependencyCheck Route 53 health checks and resolver; from a task, test name resolution and VPC .2 resolver reachabilityWhether it is Route 53 record/health-check, a VPC resolver/ENI issue, or the dependency itself being down.Record/health issue: fix/failover the record. Resolver/ENI: check route tables + NACLs for the resolver path.
AccessDenied / 403 spiking in logs after a changeaws cloudtrail lookup-events for PutRolePolicy/AttachRolePolicy; aws iam simulate-principal-policy for the actionWhether a recent IAM/policy or trust-change removed a permission the workload depends on.LABEL prod change: restore the prior policy/role version; verify with simulate-principal-policy before and after.
Unhealthy backends / targets draining repeatedlydescribe-target-health Reason field + the health-check path, port, and timeout on the target groupWhether the health check is too strict/misconfigured or the app genuinely fails the check (crash, slow start).App failing: roll back or fix startup. Check too strict: widen timeout/threshold (LABEL: config change).
ThrottlingException / quota or capacity errorsService Quotas console/CLI for the limit; CloudWatch for Insufficient-capacity or RejectedConnectionCountWhether you hit a soft service quota, an API rate limit, or AZ capacity for the instance type.Request a quota increase; add backoff/retry; spread across AZs/instance types. Scale within existing headroom first.
5xx only in one Availability ZoneBreak CloudWatch metrics down per-AZ; describe-target-health grouped by AZWhether it is an AZ-localized issue (impaired AZ, zonal deploy, one bad subnet) vs service-wide.Remove the bad AZ from the target group / disable cross-zone temporarily (LABEL) while the zone recovers.
Everything green in metrics but users still report errorsCheck CloudFront/WAF and Route 53 in front of the ALB; query ALB access logs in S3 for the user's requestsWhether the failure is above the ALB (CDN, WAF rule, DNS) rather than at the backend you're watching.Inspect WAF blocked-request count and CloudFront 5xx; roll back the offending WAF/CDN change if it correlates.

References

AWS CLI v2 on macOS/Linux; replace ACCOUNT_ID, REGION, PROFILE, LB_NAME, TARGET_GROUP_ARN, CLUSTER, SERVICE placeholders with your own.