Reliability
GCP SRE — the first 15 minutes of an incident
A first-15-minutes triage runbook for Google Cloud incidents: confirm which account, project and region you are in, establish blast radius, read the four golden signals, check load-balancer backend health, correlate recent deploys/config/IAM changes from Audit Logs, then decide mitigate vs roll back against the affected SLI. Inspect/read steps first; rollbacks and prod-changing actions are labeled.
Quick reference — reach for these first
gcloud config listConfirm active account, PROJECT_ID and REGION before touching anythinggcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_IDCount 5xx and see which services/regions are hit — read-onlygcloud monitoring dashboards list --project=PROJECT_IDLatency, traffic, errors, saturation in Metrics Explorergcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_IDAre LB backends HEALTHY, or has capacity gone?gcloud logging read 'logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"' --freshness=30m --project=PROJECT_IDRecent deploys, config and IAM edits from Audit LogsFirst 15 minutes — GCP incident triage
Text equivalent: A six-step left-to-right flow for the first fifteen minutes of a Google Cloud incident. Step 1: confirm account, project and region. Step 2: establish blast radius — who, what and where is affected. Step 3: inspect the four golden signals, shown below as latency, traffic, errors and saturation. Step 4: check load-balancer and backend health. Step 5: correlate recent changes across deploy, config and IAM from Audit Logs. Step 6, highlighted as the decision: mitigate or roll back, measured against the affected SLI.
Common tasks
Triage step · Command · What it tells you
| Triage step | Command | What it tells you |
|---|---|---|
| Confirm active account, project and region | gcloud config list && gcloud auth list | account / project / region match the affected environment, not a stale default |
| Establish blast radius: count 5xx in last 15m | gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(timestamp)' | wc -l | error count vs normal baseline tells you severity and whether it is growing |
| Scope which service/region is affected | gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(resource.labels.backend_target_name)' | sort | uniq -c | one backend/region concentrated = localized; everything = platform-wide |
| Traffic signal: is request volume normal? | gcloud monitoring dashboards list --project=PROJECT_ID # then Metrics Explorer: loadbalancing.googleapis.com/https/request_count | a sudden drop = upstream outage; a spike = load/DDoS driving saturation |
| Errors signal: split edge vs backend 5xx | gcloud logging read 'resource.type="http_load_balancer"' --freshness=15m --project=PROJECT_ID --format='value(jsonPayload.statusDetails)' | sort | uniq -c | statusDetails names the layer (LB vs backend_connection_* vs app) |
| Latency signal: p95/p99 from LB logs | gcloud logging read 'resource.type="http_load_balancer"' --freshness=15m --project=PROJECT_ID --format='value(httpRequest.latency)' | sort -rn | head | tail latency spike points at a slow backend or dependency |
| Saturation signal: CPU/mem/instances | kubectl top pods -n NAMESPACE # GKE — or Cloud Run: gcloud run services describe SERVICE --region=REGION --project=PROJECT_ID --format='value(status.traffic)' | pegged CPU/mem or maxed instance count = capacity-bound, needs scale or shed |
| Load-balancer backend health | gcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_ID | all HEALTHY rules out the LB/health-check layer; UNHEALTHY points there |
| Target/app health (GKE or Cloud Run) | kubectl get pods -A --field-selector=status.phase!=Running # GKE — or: gcloud run revisions list --service=SERVICE --region=REGION --project=PROJECT_ID | crashlooping pods / failing revision correlates the app layer to the errors |
| VPC / connectivity sanity check | gcloud compute firewall-rules list --project=PROJECT_ID --filter='disabled=false' --format='table(name,direction,sourceRanges.list(),allowed[].map().firewall_rule().list())' | no recently added deny / removed allow rule blocking backend or health-check traffic |
| Correlate recent changes (Admin Activity audit log) | gcloud logging read 'logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"' --freshness=30m --project=PROJECT_ID --limit=20 --format='table(timestamp,protoPayload.methodName,protoPayload.authenticationInfo.principalEmail)' | a deploy/config change whose timestamp lines up with error onset is your prime suspect |
| Check recent IAM changes | gcloud logging read 'protoPayload.methodName="SetIamPolicy"' --freshness=6h --project=PROJECT_ID --limit=20 --format='table(timestamp,protoPayload.authenticationInfo.principalEmail,resource.type)' | a removed binding explains sudden 403/PERMISSION_DENIED between services |
| Inventory recent deploys / last-good version | gcloud run revisions list --service=SERVICE --region=REGION --project=PROJECT_ID --format='table(metadata.name,status.conditions[0].status,metadata.creationTimestamp)' # GKE: kubectl rollout history deployment/DEPLOYMENT | you can name the previous known-good revision to roll back to |
| Decide + act against the SLI (LABEL: prod change / rollback) | # Cloud Run rollback — PROD CHANGE: gcloud run services update-traffic SERVICE --to-revisions=REVISION=100 --region=REGION --project=PROJECT_ID # GKE: kubectl rollout undo deployment/DEPLOYMENT | error-budget burn exceeds threshold -> roll back; marginal -> mitigate (scale/flag) and keep investigating |
HTTP 5xx errors spike after a deployment
5xx after deploy
gcloud config list
# core.project = PROJECT_ID, compute/region = REGION, account = oncall@gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(timestamp)' | wc -l
# 4123 — roughly 18x the normal baselinegcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=15m --project=PROJECT_ID --format='value(jsonPayload.statusDetails)' | sort | uniq -c
# 4012 backend_connection_closed_before_data_sent_to_client -> backend, not edgegcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_ID
# healthState: UNHEALTHY on 6/8 backendsgcloud run revisions list --service=SERVICE --region=REGION --project=PROJECT_ID
# newest REVISION rev-00042 (01:14Z) is serving 100% traffic and throwing 500sgcloud logging read 'logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"' --freshness=30m --project=PROJECT_ID --format='table(timestamp,protoPayload.methodName,protoPayload.authenticationInfo.principalEmail)'
# 01:14Z google.cloud.run.v1.Services.ReplaceService deployer@ -> rev-00042# 18x baseline, fast burn on the availability SLO -> mitigate by reverting to last-good
# last-good = rev-00041 (healthy for 3 days)gcloud run services update-traffic SERVICE --to-revisions=rev-00041=100 --region=REGION --project=PROJECT_ID
# Traffic: rev-00041 = 100%gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status>=500' --freshness=2m --project=PROJECT_ID --format='value(timestamp)' | wc -l
# 3 — back to baseline; get-health now reports 8/8 HEALTHYTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Clients see 504 / gateway timeout | gcloud compute backend-services get-health BACKEND_SERVICE --global --project=PROJECT_ID | whether backends are UNHEALTHY/at capacity vs the app itself being slow | If HEALTHY, read app latency logs and dependency calls; if UNHEALTHY, inspect health-check config and the last deploy, then roll back |
| DNS resolution fails / NXDOMAIN to a service | gcloud dns record-sets list --zone=ZONE --project=PROJECT_ID (and dig NAME) | whether the record exists and has propagated vs a wrong/edited Cloud DNS zone | Verify record + TTL, then check Audit Logs for a recent DNS/config change and revert it |
| Callers suddenly get 403 / PERMISSION_DENIED | gcloud logging read 'protoPayload.methodName="SetIamPolicy"' --freshness=6h --project=PROJECT_ID | whether a recent IAM edit removed a required binding | Diff against gcloud projects get-iam-policy PROJECT_ID and restore the binding (LABEL: prod IAM change) |
| LB reports 0 healthy backends / 502s | gcloud compute backend-services get-health BACKEND_SERVICE --global && gcloud compute health-checks describe HEALTH_CHECK --project=PROJECT_ID | whether the health check is misconfigured vs the instances/pods are actually down | Match HC port/path to the app; if a new revision broke it, roll back to the last-good version |
| 429 / RESOURCE_EXHAUSTED, scale-up blocked | gcloud logging read 'protoPayload.status.message:"Quota" OR severity>=ERROR' --freshness=15m --project=PROJECT_ID && gcloud compute regions describe REGION --project=PROJECT_ID | which quota/capacity limit is being hit in this region | Shed or rate-limit load, scale into another region, and request a quota increase |
| Error rate jumps right after a release | gcloud run revisions list --service=SERVICE --region=REGION (or kubectl rollout history deployment/DEPLOYMENT) | whether the new revision's timestamp correlates with error onset | LABEL rollback: shift 100% traffic to the last-good revision (or kubectl rollout undo) and verify the SLI recovers |
References
- Cloud Monitoring documentationcloud.google.com
- Cloud Logging query languagecloud.google.com
- Cloud Audit Logs overviewcloud.google.com
- Load balancer health check conceptscloud.google.com
- Cloud Run rollouts, rollbacks & trafficcloud.google.com
Assumes gcloud CLI 480+ (2025) on macOS/Linux bash/zsh, Google Cloud, kubectl for GKE; replace CAPS placeholders (PROJECT_ID, REGION, SERVICE, CLUSTER, BACKEND_SERVICE, DEPLOYMENT, REVISION) with your own.