Reliability
AWS SRE — the first 15 minutes of an incident
A triage runbook for the first 15 minutes of an AWS incident: confirm which account and Region you are in, size the blast radius, read the four golden signals in CloudWatch, check ALB/NLB target health, correlate recent changes in CloudTrail, then decide mitigate vs roll back — measured against the affected SLI. Inspect-first; prod-changing steps are labeled.
Quick reference — reach for these first
aws sts get-caller-identity --profile PROFILE; aws configure get regionConfirm the account and Region before touching anything. Wrong account is the most common wasted first move.aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGIONIs it one service, one AZ, or everything? Edge 5xx over the last 30 min sizes the impact.aws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION --query 'TargetHealthDescriptions[].{t:Target.Id,s:TargetHealthDescription:'' ,state:TargetHealth.State,why:TargetHealth.Reason}'Are backends actually in service? Draining/unhealthy targets explain most 5xx and timeouts.aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService --max-results 15 --region REGIONCorrelate the spike to a deploy, scaling, or IAM change. 'What changed?' answers most incidents.aws logs start-query --log-group-name /ecs/SERVICE --start-time $(date -u -v-15M +%s) --end-time $(date -u +%s) --query-string 'fields @timestamp,@message | filter @message like /ERROR|Exception|5\\d\\d/ | sort @timestamp desc | limit 50' --region REGIONLogs Insights returns a queryId; poll get-query-results for the actual failing requests.Incident triage flow — the first 15 minutes
Text equivalent: Six ordered steps flow left to right: 1 confirm account and Region, 2 establish blast radius, 3 read the four golden signals (latency, traffic, errors, saturation), 4 check load-balancer and target health, 5 correlate recent changes in CloudTrail, 6 decide mitigate or roll back. A loop from step 6 points to a verify box: every step and the final decision are measured against the affected SLI.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| 1. Confirm account + Region (read-only) | aws sts get-caller-identity --profile PROFILE && aws configure list | Account matches the impacted prod account and Region is the one paging you — not a default/sandbox. |
| 2. Pin down the affected service/stack | aws elbv2 describe-load-balancers --names LB_NAME --region REGION --query 'LoadBalancers[0].{dns:DNSName,state:State.Code}' | You know the exact load balancer, target group, and ECS/EKS service that fronts the paging alarm. |
| 3. Establish blast radius (error rate + scope) | aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION | Error volume and start time are known; compare per-AZ to tell 'whole service' from 'one zone'. |
| 4. Golden signal — latency | aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics p99 Average --region REGION | p99 TargetResponseTime trend seen; a latency cliff at the start time points at the backend. |
| 5. Golden signal — traffic | aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name RequestCount --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION | Distinguish a demand spike from a drop-to-zero (dependency/edge outage) vs steady traffic failing. |
| 6. Golden signal — errors (edge vs backend) | for m in HTTPCode_ELB_5XX_Count HTTPCode_Target_5XX_Count; do aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name $m --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-15M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION; done | ELB_5XX (LB itself: no healthy target) vs Target_5XX (app returned 5xx) splits LB problem from app problem. |
| 7. Golden signal — saturation | aws cloudwatch get-metric-statistics --namespace AWS/ECS --metric-name CPUUtilization --dimensions Name=ClusterName,Value=CLUSTER Name=ServiceName,Value=SERVICE --start-time $(date -u -v-30M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Average Maximum --region REGION | CPU/memory near 100%, or TargetConnectionErrorCount rising, indicates capacity — scale, don't roll back. |
| 8. Load balancer / target health | aws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION | Count of healthy vs unhealthy targets and the Reason (e.g. Target.FailedHealthChecks) is recorded. |
| 9. Backend application logs | aws logs get-query-results --query-id QUERY_ID --region REGION | Actual error messages/stack traces from the window are in hand (start-query from quickRef gives QUERY_ID). |
| 10. Network path — timeouts (read-only) | aws ec2 describe-route-tables --filters Name=vpc-id,Values=VPC_ID --region REGION --query 'RouteTables[].Routes'; aws ec2 describe-flow-logs --filter Name=resource-id,Values=VPC_ID --region REGION | Routes to NAT/IGW intact; flow logs show ACCEPT not REJECT to the dependency port. |
| 11. Correlate recent changes (CloudTrail) | aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService --max-results 20 --region REGION --query 'Events[].{t:EventTime,u:Username,e:EventName}' | A deploy, scaling, IAM, or security-group change lines up with the incident start time (or rules change out). |
| 12. Deployment status — ECS/EKS | aws ecs describe-services --cluster CLUSTER --services SERVICE --region REGION --query 'services[0].deployments' # EKS: kubectl rollout status deployment/NAME -n NS | Deployment rolloutState (IN_PROGRESS/FAILED) and running vs desired count show whether a bad release is live. |
| 13. Decide: mitigate vs roll back — against the SLI | # compare current error-rate/latency to the SLO target before acting | If a deploy caused it and SLO is burning: roll back. If capacity: scale. If dependency: shed/failover. |
| 14. LABEL: roll back the deploy (prod change) | aws ecs update-service --cluster CLUSTER --service SERVICE --task-definition FAMILY:PREV_REVISION --region REGION # EKS: kubectl rollout undo deployment/NAME -n NS | PROD CHANGE — confirm PREV_REVISION is the last-known-good; watch Target_5XX and p99 return to baseline. |
HTTP 5xx errors spike right after a deployment
5xx after deploy
aws sts get-caller-identity --profile PROFILE; aws configure get region
# Account 1234567890 / us-east-1 — correct prod accountaws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-20M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION
# Target_5XX jumps from ~0 to 4k/min; ELB_5XX stays low -> the app is returning 5xxaws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION
# 6/10 targets unhealthy, Reason Target.ResponseCodeMismatchaws logs start-query --log-group-name /ecs/SERVICE --start-time $(date -u -v-15M +%s) --end-time $(date -u +%s) --query-string 'fields @timestamp,@message | filter @message like /Exception|5\\d\\d/ | sort @timestamp desc | limit 20' --region REGION
# QueryId f1a2... -> get-query-results shows NullPointer on a new config keyaws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateService --max-results 5 --region REGION
# UpdateService 3 min before the spike, task def FAMILY:48 -> prior good is :47aws ecs update-service --cluster CLUSTER --service SERVICE --task-definition FAMILY:47 --region REGION
# deployment IN_PROGRESS, rolloutState rolling back to :47aws elbv2 describe-target-health --target-group-arn TARGET_GROUP_ARN --region REGION; aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID --start-time $(date -u -v-5M +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum --region REGION
# 10/10 targets healthy, Target_5XX back to ~0 — error budget burn stoppedTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| 504 Gateway Timeout from the ALB | aws elbv2 describe-target-health + TargetResponseTime p99 and TargetConnectionErrorCount | Whether backends are slow/overloaded (target responding late) vs the LB can't connect at all (SG/route/port). | Slow backend: scale or roll back the slow release. No connection: fix security group / target port / health-check path. |
| Intermittent DNS resolution failures to a dependency | Check Route 53 health checks and resolver; from a task, test name resolution and VPC .2 resolver reachability | Whether it is Route 53 record/health-check, a VPC resolver/ENI issue, or the dependency itself being down. | Record/health issue: fix/failover the record. Resolver/ENI: check route tables + NACLs for the resolver path. |
| AccessDenied / 403 spiking in logs after a change | aws cloudtrail lookup-events for PutRolePolicy/AttachRolePolicy; aws iam simulate-principal-policy for the action | Whether a recent IAM/policy or trust-change removed a permission the workload depends on. | LABEL prod change: restore the prior policy/role version; verify with simulate-principal-policy before and after. |
| Unhealthy backends / targets draining repeatedly | describe-target-health Reason field + the health-check path, port, and timeout on the target group | Whether the health check is too strict/misconfigured or the app genuinely fails the check (crash, slow start). | App failing: roll back or fix startup. Check too strict: widen timeout/threshold (LABEL: config change). |
| ThrottlingException / quota or capacity errors | Service Quotas console/CLI for the limit; CloudWatch for Insufficient-capacity or RejectedConnectionCount | Whether you hit a soft service quota, an API rate limit, or AZ capacity for the instance type. | Request a quota increase; add backoff/retry; spread across AZs/instance types. Scale within existing headroom first. |
| 5xx only in one Availability Zone | Break CloudWatch metrics down per-AZ; describe-target-health grouped by AZ | Whether it is an AZ-localized issue (impaired AZ, zonal deploy, one bad subnet) vs service-wide. | Remove the bad AZ from the target group / disable cross-zone temporarily (LABEL) while the zone recovers. |
| Everything green in metrics but users still report errors | Check CloudFront/WAF and Route 53 in front of the ALB; query ALB access logs in S3 for the user's requests | Whether the failure is above the ALB (CDN, WAF rule, DNS) rather than at the backend you're watching. | Inspect WAF blocked-request count and CloudFront 5xx; roll back the offending WAF/CDN change if it correlates. |
References
- AWS CLI Command Referencedocs.aws.amazon.com
- CloudWatch Logs Insights query syntaxdocs.aws.amazon.com
- ELB target health states & reasonsdocs.aws.amazon.com
- View CloudTrail events with lookup-eventsdocs.aws.amazon.com
- ECS deployment circuit breaker & rollbackdocs.aws.amazon.com
AWS CLI v2 on macOS/Linux; replace ACCOUNT_ID, REGION, PROFILE, LB_NAME, TARGET_GROUP_ARN, CLUSTER, SERVICE placeholders with your own.