Reliability
Azure SRE — the first 15 minutes of an incident
A first-responder triage runbook for an Azure incident: confirm which subscription and region you are in, size the blast radius, read the four golden signals, check backend health, correlate recent changes, then decide mitigate-or-roll-back against the affected SLI.
Quick reference — reach for these first
az account show -o tableConfirm the subscription, tenant and default region BEFORE touching anything.az monitor activity-log list --offset 1h -o tableWhich resources changed/failed, how many, since when — scope the incident.az monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation TotalThe errors golden signal on the affected resource, minute by minute.az network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o tableAre the backend targets Healthy or is the LB pulling them out?az monitor activity-log list --offset 2h --query "[?category.value=='Administrative']" -o tableRecent deploy / config / RBAC writes — the prime suspect list.Incident triage flow — the first 15 minutes
Text equivalent: A left-to-right flow of six triage steps. 1 Confirm context (subscription and region). 2 Establish blast radius (what is affected and how wide). 3 Read the four golden signals (latency, traffic, errors, saturation). 4 Check load-balancer backend health (probes). 5 Correlate recent changes in the activity log. 6 Decide: mitigate or roll back, measured against the affected SLI.
Common tasks
Triage step · Command · Signal / check
| Triage step | Command | Signal / check |
|---|---|---|
| Pin the context — subscription, tenant, region | az account show -o table | Name/subscription match the incident's environment; if not, do NOT run changes yet. |
| Switch subscription if you are in the wrong one (read-only) | az account set --subscription SUBSCRIPTION_ID | `az account show` now reports the correct subscription; no resource is modified. |
| Establish blast radius from recent platform events | az monitor activity-log list --offset 1h --query "[].{t:eventTimestamp,op:operationName.value,res:resourceId,status:status.value}" -o table | Which resources and how wide; failures clustered on one service vs fleet-wide. |
| Golden signal — latency | az monitor metrics list --resource RESOURCE_ID --metric HttpResponseTime --interval PT1M --aggregation Average | Response time vs baseline; a sharp climb points at a downstream/backed-up tier. |
| Golden signal — traffic | az monitor metrics list --resource RESOURCE_ID --metric Requests --interval PT1M --aggregation Total | Traffic collapse (edge/DNS outage) vs surge (overload) frames the whole diagnosis. |
| Golden signal — errors | az monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total | 5xx rate and the exact onset minute — anchor for change correlation. |
| Golden signal — saturation (CPU/memory) | az monitor metrics list --resource RESOURCE_ID --metric CpuPercentage --interval PT1M --aggregation Average | Resource pinned near its limit indicates capacity, not a code regression. |
| Load-balancer / gateway backend health | az network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table | Backend pool Healthy count vs Unhealthy; a probe failure masquerades as 5xx. |
| Drill into app/platform logs with KQL | az monitor log-analytics query -w WORKSPACE_ID --analytics-query "AppRequests | where TimeGenerated > ago(15m) and Success == false | summarize count() by ResultCode" | The dominant error code / exception and whether it is new this window. |
| Correlate recent deploys, config and IAM writes | az monitor activity-log list --offset 2h --query "[?category.value=='Administrative'].{t:eventTimestamp,op:operationName.value,who:caller}" -o table | A write immediately before the onset minute is the likely trigger. |
| Network path — is NSG allowing the traffic? | az network watcher test-ip-flow --direction Inbound --protocol Tcp --local LOCAL_IP:443 --remote REMOTE_IP:0 --vm VM_NAME -g RESOURCE_GROUP | Result Allow vs Deny and which rule matched; verify the route with `az network watcher show-next-hop`. |
| If AKS: node, pod and event health | kubectl get nodes; kubectl get pods -A --field-selector=status.phase!=Running; kubectl get events --sort-by=.lastTimestamp | tail -20 | NotReady nodes, CrashLoopBackOff pods, or recent Warning events pinpoint the failing workload. |
| Decide — roll back the deploy if it broke the SLI (PROD CHANGE — confirm with IC first) | az webapp deployment slot swap -g RESOURCE_GROUP -n APP_NAME --slot staging --target-slot production | 5xx returns to baseline within minutes; if not, escalate and keep mitigating. |
HTTP 5xx errors spike after a deployment
mitigate-or-rollback
az account show -o table
# Name SubscriptionId ...
# learntrainai-prod 00000000-... (region: eastus) ✔az monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total
# 14:02 Total 0 → 14:09 ramps to 180 / minaz network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table
# 4/4 backends Unhealthy — probe returning 502az monitor log-analytics query -w WORKSPACE_ID --analytics-query "AppRequests | where TimeGenerated > ago(15m) and Success == false | summarize count() by ResultCode"
# ResultCode 502 count 1,640az monitor activity-log list --offset 30m --query "[?category.value=='Administrative'].{t:eventTimestamp,op:operationName.value,who:caller}" -o table
# 14:08 Microsoft.Web/sites/write deploy@... (release v412)az webapp deployment slot swap -g RESOURCE_GROUP -n APP_NAME --slot staging --target-slot production
# Swap complete — production now serves the pre-v412 buildaz network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table
# 4/4 backends Healthyaz monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total
# 14:21 Total 0 / min — SLI recovered; keep v412 fix in stagingTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Requests timing out / 504 at the edge | az network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table | Whether backends are reachable and Healthy, or the gateway cannot reach them at all. | Unhealthy → check NSG/route to backend subnet and backend CPU; Healthy → app thread-pool/downstream latency. |
| DNS resolution failing for the service | az network dns record-set list -g RESOURCE_GROUP -z ZONE_NAME -o table | Whether the record exists and points to the right target, and the Private DNS zone link is intact. | Missing/wrong record → find the DNS write in Activity Log; resolver issue → verify Private DNS zone VNet link. |
| Access denied / 403 / AuthorizationFailed | az role assignment list --assignee PRINCIPAL_ID --all -o table | Whether an RBAC role assignment was removed or changed for the failing identity. | Missing role → correlate the role write in Activity Log; secret-based → check Key Vault access policy and secret expiry. |
| Backend pool marked Unhealthy after a release | az network application-gateway show-backend-health -g RESOURCE_GROUP -n APP_GATEWAY -o table | Whether the health probe path/port is failing vs the application actually being down. | Probe path changed by deploy → fix probe; all instances down → roll back the deploy (PROD CHANGE). |
| 429s / quota or capacity limit reached | az vm list-usage -l REGION -o table | Whether a regional vCPU/SKU quota is exhausted so scale-out is being refused. | At the ceiling → spread to another AZ/region or request a quota increase; otherwise throttling sits at the app/APIM tier. |
| 5xx coming from only one instance or zone | az monitor metrics list --resource RESOURCE_ID --metric Http5xx --interval PT1M --aggregation Total --filter "Instance eq '*'" | Whether this is one poisoned instance vs a fleet-wide regression. | Single instance → restart/cordon just that instance; fleet-wide → deploy or config rollback. |
References
- Azure Monitor metrics overviewlearn.microsoft.com
- az monitor activity-log (CLI reference)learn.microsoft.com
- Query Log Analytics from the CLIlearn.microsoft.com
- Network Watcher IP flow verifylearn.microsoft.com
- Application Gateway backend health troubleshootinglearn.microsoft.com
Azure CLI 2.60+ on macOS/Linux, signed in via `az login`; App Service or AKS behind Application Gateway/Load Balancer; every command is read-only unless tagged PROD CHANGE.