← All cheatsheets

Reliability

Azure SRE — the first 15 minutes of an incident

A first-responder triage runbook for an Azure incident: confirm which subscription and region you are in, size the blast radius, read the four golden signals, check backend health, correlate recent changes, then decide mitigate-or-roll-back against the affected SLI.

Docs-checkedazuresreincidentazure-monitoractivity-log

Quick reference — reach for these first

Where am I?az account show -o tableConfirm the subscription, tenant and default region BEFORE touching anything.
Blast radiusaz monitor activity-log list --offset 1h -o tableWhich resources changed/failed, how many, since when — scope the incident.
Errors signalaz monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation TotalThe errors golden signal on the affected resource, minute by minute.
Backend healthaz network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o tableAre the backend targets Healthy or is the LB pulling them out?
Who changed it?az monitor activity-log list --offset 2h --query "[?category.value=='Administrative']" -o tableRecent deploy / config / RBAC writes — the prime suspect list.

Incident triage flow — the first 15 minutes

Azure incident triage — first 15 minutesSix ordered triage steps flow left to right: confirm which subscription and region you are in, establish the blast radius, read the four golden signals, check load-balancer backend health, correlate recent changes in the activity log, then decide to mitigate or roll back measured against the affected SLI.Azure SRE — first 15 minutes1 Confirmsub · region2 Blast radiuswhat / how wide3 Golden signalslat·traf·err·sat4 Backend healthLB probes5 Recent changesactivity log6 Decidemitigate / roll backmeasured vs the affected SLI

Text equivalent: A left-to-right flow of six triage steps. 1 Confirm context (subscription and region). 2 Establish blast radius (what is affected and how wide). 3 Read the four golden signals (latency, traffic, errors, saturation). 4 Check load-balancer backend health (probes). 5 Correlate recent changes in the activity log. 6 Decide: mitigate or roll back, measured against the affected SLI.

Common tasks

Triage step · Command · Signal / check

Triage stepCommandSignal / check
Pin the context — subscription, tenant, regionaz account show -o tableName/subscription match the incident's environment; if not, do NOT run changes yet.
Switch subscription if you are in the wrong one (read-only)az account set --subscription SUBSCRIPTION_ID`az account show` now reports the correct subscription; no resource is modified.
Establish blast radius from recent platform eventsaz monitor activity-log list --offset 1h --query "[].{t:eventTimestamp,op:operationName.value,res:resourceId,status:status.value}" -o tableWhich resources and how wide; failures clustered on one service vs fleet-wide.
Golden signal — latencyaz monitor metrics list --resource RESOURCE_ID --metric HttpResponseTime --interval PT1M --aggregation AverageResponse time vs baseline; a sharp climb points at a downstream/backed-up tier.
Golden signal — trafficaz monitor metrics list --resource RESOURCE_ID --metric Requests --interval PT1M --aggregation TotalTraffic collapse (edge/DNS outage) vs surge (overload) frames the whole diagnosis.
Golden signal — errorsaz monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total5xx rate and the exact onset minute — anchor for change correlation.
Golden signal — saturation (CPU/memory)az monitor metrics list --resource RESOURCE_ID --metric CpuPercentage --interval PT1M --aggregation AverageResource pinned near its limit indicates capacity, not a code regression.
Load-balancer / gateway backend healthaz network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o tableBackend pool Healthy count vs Unhealthy; a probe failure masquerades as 5xx.
Drill into app/platform logs with KQLaz monitor log-analytics query -w WORKSPACE_ID --analytics-query "AppRequests | where TimeGenerated > ago(15m) and Success == false | summarize count() by ResultCode"The dominant error code / exception and whether it is new this window.
Correlate recent deploys, config and IAM writesaz monitor activity-log list --offset 2h --query "[?category.value=='Administrative'].{t:eventTimestamp,op:operationName.value,who:caller}" -o tableA write immediately before the onset minute is the likely trigger.
Network path — is NSG allowing the traffic?az network watcher test-ip-flow --direction Inbound --protocol Tcp --local LOCAL_IP:443 --remote REMOTE_IP:0 --vm VM_NAME -g RESOURCE_GROUPResult Allow vs Deny and which rule matched; verify the route with `az network watcher show-next-hop`.
If AKS: node, pod and event healthkubectl get nodes; kubectl get pods -A --field-selector=status.phase!=Running; kubectl get events --sort-by=.lastTimestamp | tail -20NotReady nodes, CrashLoopBackOff pods, or recent Warning events pinpoint the failing workload.
Decide — roll back the deploy if it broke the SLI (PROD CHANGE — confirm with IC first)az webapp deployment slot swap -g RESOURCE_GROUP -n APP_NAME --slot staging --target-slot production5xx returns to baseline within minutes; if not, escalate and keep mitigating.

HTTP 5xx errors spike after a deployment

mitigate-or-rollback

1
Pin context — confirm the subscription and region match the alerting environment.
az account show -o table
# Name           SubscriptionId   ...
# learntrainai-prod  00000000-...   (region: eastus) ✔
2
Confirm the spike on the errors golden signal and note the onset minute.
az monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total
# 14:02 Total 0  →  14:09 ramps to 180 / min
3
Check load-balancer backend health — is the gateway pulling targets out?
az network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table
# 4/4 backends Unhealthy — probe returning 502
4
Pull the dominant failure from application logs with KQL.
az monitor log-analytics query -w WORKSPACE_ID --analytics-query "AppRequests | where TimeGenerated > ago(15m) and Success == false | summarize count() by ResultCode"
# ResultCode 502   count 1,640
5
Correlate recent changes — what landed just before 14:09?
az monitor activity-log list --offset 30m --query "[?category.value=='Administrative'].{t:eventTimestamp,op:operationName.value,who:caller}" -o table
# 14:08  Microsoft.Web/sites/write  deploy@...  (release v412)
!
Hypothesis: release v412 changed the health-probe path, so the gateway marks every backend Unhealthy and returns 502. Decision measured against the 5xx SLI: roll back. This is a PROD CHANGE — confirm with the incident commander first.
6
Roll back by swapping production back to the slot holding the last-good build (PROD CHANGE).
az webapp deployment slot swap -g RESOURCE_GROUP -n APP_NAME --slot staging --target-slot production
# Swap complete — production now serves the pre-v412 build
✓
Verify backend health recovered.
az network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table
# 4/4 backends Healthy
✓
Confirm the SLI — 5xx back to baseline — and close the loop.
az monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total
# 14:21 Total 0 / min — SLI recovered; keep v412 fix in staging

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Requests timing out / 504 at the edgeaz network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o tableWhether backends are reachable and Healthy, or the gateway cannot reach them at all.Unhealthy → check NSG/route to backend subnet and backend CPU; Healthy → app thread-pool/downstream latency.
DNS resolution failing for the serviceaz network dns record-set list -g RESOURCE_GROUP -z ZONE_NAME -o tableWhether the record exists and points to the right target, and the Private DNS zone link is intact.Missing/wrong record → find the DNS write in Activity Log; resolver issue → verify Private DNS zone VNet link.
Access denied / 403 / AuthorizationFailedaz role assignment list --assignee PRINCIPAL_ID --all -o tableWhether an RBAC role assignment was removed or changed for the failing identity.Missing role → correlate the role write in Activity Log; secret-based → check Key Vault access policy and secret expiry.
Backend pool marked Unhealthy after a releaseaz network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o tableWhether the health probe path/port is failing vs the application actually being down.Probe path changed by deploy → fix probe; all instances down → roll back the deploy (PROD CHANGE).
429s / quota or capacity limit reachedaz vm list-usage -l REGION -o tableWhether a regional vCPU/SKU quota is exhausted so scale-out is being refused.At the ceiling → spread to another AZ/region or request a quota increase; otherwise throttling sits at the app/APIM tier.
5xx coming from only one instance or zoneaz monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total --filter "Instance eq '*'"Whether this is one poisoned instance vs a fleet-wide regression.Single instance → restart/cordon just that instance; fleet-wide → deploy or config rollback.

References

Azure CLI 2.60+ on macOS/Linux, signed in via `az login`; App Service or AKS behind Application Gateway/Load Balancer; every command is read-only unless tagged PROD CHANGE.