← All cheatsheets

Observability

Observability & Incident Response — signals, triage, and resolution

The three pillars (metrics, logs, traces), the four golden signals, RED and USE, symptom-based alerting, and the detect-to-postmortem incident lifecycle.

Docs-checkedobservabilityincidentgolden-signalsmetricslogstraces

Quick reference — reach for these first

1 · Acknowledgeack the pageAck the alert so escalation pauses — you now own the incident.
2 · Declare & assign/incident declare --sev SEVOpen an incident channel and name an Incident Commander (IC).
3 · Assess signalsopen TRIAGE_DASHBOARDRead the four golden signals: how bad is it, and where.
4 · Communicatepost impact + next-update ETAOne status update: user impact, scope, and when you'll update next.
5 · Correlate changekubectl rollout history deploy/SVCCheck recent deploys, flags, and config — the most likely cause.

Signals feed an alert; the alert starts the incident lifecycle

Signals feeding an alert, then the incident lifecycleMetrics, logs and traces feed a symptom-based alert. The alert starts the incident flow: detect, triage, mitigate (which changes production), resolve, then a blameless postmortem that loops back into better signals and alerts. Incident roles are listed at the bottom.postmortem actions → better signals & tighter alertsMetricsrates & trends · cheapLogswhat happened · detailTraceswhich hop · latencyAlert firessymptom-based · actionablepage starts the lifecycleDetectpage / SLO burnTriagescope · severity · ICMitigaterollback · scale · flag⚠ changes productionResolvefix · confirm greenPostmortemblameless · actionsRoles — Incident Commander (decisions) · Comms Lead (updates) · Ops/SME (hands-on) · Scribe (timeline)

Text equivalent: Three boxes at the top — Metrics (rates and trends), Logs (what happened, in detail), and Traces (which hop was slow) — all feed down into a single accent-colored Alert box labeled symptom-based and actionable. An arrow runs from the alert into a left-to-right incident flow of five boxes: Detect, Triage, Mitigate (accent-colored and flagged as changing production), Resolve, and Postmortem. A dashed arrow loops from Postmortem back up to Metrics, showing that postmortem actions feed better signals and tighter alerts. A line at the bottom lists the roles: Incident Commander, Comms Lead, Ops/SME, and Scribe.

Common tasks

Task · Command · What to check

TaskCommandWhat to check
Define an SLO + error budget for each tier-1 user journeySLO: 99.9% of POST /checkout < 300ms over 28dEvery critical journey has a target and a budget you can burn
Instrument request services with RED (rate, errors, duration)sum(rate(http_requests_total{job="API"}[5m]))Rate, error ratio, and p50/p95/p99 duration are all graphed
Add USE dashboards for hosts/queues (utilization, saturation, errors)rate(node_cpu_seconds_total{mode!="idle"}[5m])Each constrained resource shows U, S, and E
Emit structured logs carrying a trace/request id and levellog.info(msg, trace_id=TRACE_ID, level="error")Logs are JSON and filterable by trace_id and severity
Enable distributed tracing with tail-based sampling for errors/slowotelcol: tail_sampling { error, latency(p99) }Every errored or high-latency request keeps a trace
Alert on SLO burn rate (symptom), not on causesburnrate_1h > 14.4 and burnrate_5m > 14.4Pages fire on user-visible symptoms and link to a runbook
Route pages to on-call with severity and a runbook linkalertmanager: route { severity=~"page" }Only user-impacting alerts page; the rest are tickets
Build a triage dashboard: golden signals + a deploy/change overlayannotate panel with DEPLOY_EVENTSOne screen answers how bad, where, and what changed
Write one runbook per page with concrete mitigation stepslink RUNBOOK_URL in the alert annotationEach alert's annotation points to an actionable runbook
Define incident roles and severity levels before you need themroles: IC, Comms, Ops/SME, Scribe; sev: SEV1..SEV3On-call can declare a severity and assign roles in minutes
[CHANGES PROD] Rehearse a mitigation: roll back the last deploykubectl rollout undo deployment/SVCTraffic shifts to the prior version and error rate drops
Run a blameless postmortem within 48h and file owned action itemstemplate: timeline, impact, factors, actionsEach contributing factor has an owner and a due date

Error-rate alert on /checkout: from page to postmortem

alert → narrow → confirm → mitigate → verify

1
Alert pages: API 5xx ratio is over the fast-burn SLO threshold.
burnrate_5m > 14.4 and burnrate_1h > 14.4
# Fast burn on the checkout SLO — SEV2 declared, IC assigned
2
Check the four golden signals on the triage dashboard to see how bad and where.
3
Narrow with logs: which endpoint and which error?
{app="API"} | json | level="error"
# All 500s are POST /checkout: 'payment client: connection refused'
4
Confirm with a trace: which dependency is failing?
open trace by TRACE_ID
# /checkout span → payments-svc span errored, upstream timeout
5
Correlate the failure with recent changes.
kubectl rollout history deployment/payments-svc
# payments-svc v317 rolled out 6 min before the alert fired
6
[CHANGES PROD] Mitigate: roll payments-svc back to the last good revision.
kubectl rollout undo deployment/payments-svc --to-revision=316
# Rollout reverted to v316
7
VERIFY recovery: watch the error ratio for 2–3x the alert window before resolving.
sum(rate(http_requests_total{job="API",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="API"}[5m]))
# 5xx ratio < 0.1% for 15m, burn rate < 1 → incident resolved
8
Schedule a blameless postmortem and file owned action items.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Alert storm — dozens of pages fire at onceCount distinct root incidents behind the pages; are they symptoms or causes?Whether it's one incident fanning out or many independent failuresGroup and inhibit in Alertmanager, alert on SLO symptoms not every cause, and encode service dependencies so downstream pages are suppressed
No signal — dashboard is flat or the failing endpoint is absentIs the service actually exporting? curl SERVICE:PORT/metricsAn instrumentation blind spot versus genuinely zero trafficAdd RED instrumentation plus a healthcheck; meanwhile reconstruct the picture from logs and traces
High cardinality blows up metrics (Prometheus OOM / slow queries)topk(10, count by (__name__)({__name__=~".+"})) and find the exploding labelWhich label (user_id, full path, request_id) multiplies series countDrop or aggregate the label via metric_relabel_configs; move per-id detail to logs/traces, not labels
Logs too noisy to triage — thousands of lines, error buriedFilter to the failing request window by trace_id and level=errorWhether structured fields exist to filter on at allQuery by request id + severity; adopt structured JSON logging and sample info-level lines
Trace sampling missed the bad request — no trace for the failed callIs sampling head-based or tail-based, and at what rate?That head sampling dropped the span before the error was knownSwitch to tail-based sampling that always keeps errors and high latency; raise the rate for the hot route
Error rate is back to baseline after the mitigationDo the golden signals hold for 2–3x the alert window?A real recovery versus a transient dip before it climbs againKeep the incident open until signals are stable, then resolve and schedule the postmortem

References

A request-serving service with Prometheus-style metrics, structured logs, and OpenTelemetry tracing; Kubernetes used for deploy/rollback examples. Golden signals per Google SRE; RED per Grafana/Weaveworks; USE per Brendan Gregg. Swap CAPS placeholders and tool-specific commands for your stack.