Observability
Observability & Incident Response — signals, triage, and resolution
The three pillars (metrics, logs, traces), the four golden signals, RED and USE, symptom-based alerting, and the detect-to-postmortem incident lifecycle.
Quick reference — reach for these first
ack the pageAck the alert so escalation pauses — you now own the incident./incident declare --sev SEVOpen an incident channel and name an Incident Commander (IC).open TRIAGE_DASHBOARDRead the four golden signals: how bad is it, and where.post impact + next-update ETAOne status update: user impact, scope, and when you'll update next.kubectl rollout history deploy/SVCCheck recent deploys, flags, and config — the most likely cause.Signals feed an alert; the alert starts the incident lifecycle
Text equivalent: Three boxes at the top — Metrics (rates and trends), Logs (what happened, in detail), and Traces (which hop was slow) — all feed down into a single accent-colored Alert box labeled symptom-based and actionable. An arrow runs from the alert into a left-to-right incident flow of five boxes: Detect, Triage, Mitigate (accent-colored and flagged as changing production), Resolve, and Postmortem. A dashed arrow loops from Postmortem back up to Metrics, showing that postmortem actions feed better signals and tighter alerts. A line at the bottom lists the roles: Incident Commander, Comms Lead, Ops/SME, and Scribe.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| Define an SLO + error budget for each tier-1 user journey | SLO: 99.9% of POST /checkout < 300ms over 28d | Every critical journey has a target and a budget you can burn |
| Instrument request services with RED (rate, errors, duration) | sum(rate(http_requests_total{job="API"}[5m])) | Rate, error ratio, and p50/p95/p99 duration are all graphed |
| Add USE dashboards for hosts/queues (utilization, saturation, errors) | rate(node_cpu_seconds_total{mode!="idle"}[5m]) | Each constrained resource shows U, S, and E |
| Emit structured logs carrying a trace/request id and level | log.info(msg, trace_id=TRACE_ID, level="error") | Logs are JSON and filterable by trace_id and severity |
| Enable distributed tracing with tail-based sampling for errors/slow | otelcol: tail_sampling { error, latency(p99) } | Every errored or high-latency request keeps a trace |
| Alert on SLO burn rate (symptom), not on causes | burnrate_1h > 14.4 and burnrate_5m > 14.4 | Pages fire on user-visible symptoms and link to a runbook |
| Route pages to on-call with severity and a runbook link | alertmanager: route { severity=~"page" } | Only user-impacting alerts page; the rest are tickets |
| Build a triage dashboard: golden signals + a deploy/change overlay | annotate panel with DEPLOY_EVENTS | One screen answers how bad, where, and what changed |
| Write one runbook per page with concrete mitigation steps | link RUNBOOK_URL in the alert annotation | Each alert's annotation points to an actionable runbook |
| Define incident roles and severity levels before you need them | roles: IC, Comms, Ops/SME, Scribe; sev: SEV1..SEV3 | On-call can declare a severity and assign roles in minutes |
| [CHANGES PROD] Rehearse a mitigation: roll back the last deploy | kubectl rollout undo deployment/SVC | Traffic shifts to the prior version and error rate drops |
| Run a blameless postmortem within 48h and file owned action items | template: timeline, impact, factors, actions | Each contributing factor has an owner and a due date |
Error-rate alert on /checkout: from page to postmortem
alert → narrow → confirm → mitigate → verify
burnrate_5m > 14.4 and burnrate_1h > 14.4
# Fast burn on the checkout SLO — SEV2 declared, IC assigned{app="API"} | json | level="error"
# All 500s are POST /checkout: 'payment client: connection refused'open trace by TRACE_ID
# /checkout span → payments-svc span errored, upstream timeoutkubectl rollout history deployment/payments-svc
# payments-svc v317 rolled out 6 min before the alert firedkubectl rollout undo deployment/payments-svc --to-revision=316
# Rollout reverted to v316sum(rate(http_requests_total{job="API",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="API"}[5m]))
# 5xx ratio < 0.1% for 15m, burn rate < 1 → incident resolvedTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Alert storm — dozens of pages fire at once | Count distinct root incidents behind the pages; are they symptoms or causes? | Whether it's one incident fanning out or many independent failures | Group and inhibit in Alertmanager, alert on SLO symptoms not every cause, and encode service dependencies so downstream pages are suppressed |
| No signal — dashboard is flat or the failing endpoint is absent | Is the service actually exporting? curl SERVICE:PORT/metrics | An instrumentation blind spot versus genuinely zero traffic | Add RED instrumentation plus a healthcheck; meanwhile reconstruct the picture from logs and traces |
| High cardinality blows up metrics (Prometheus OOM / slow queries) | topk(10, count by (__name__)({__name__=~".+"})) and find the exploding label | Which label (user_id, full path, request_id) multiplies series count | Drop or aggregate the label via metric_relabel_configs; move per-id detail to logs/traces, not labels |
| Logs too noisy to triage — thousands of lines, error buried | Filter to the failing request window by trace_id and level=error | Whether structured fields exist to filter on at all | Query by request id + severity; adopt structured JSON logging and sample info-level lines |
| Trace sampling missed the bad request — no trace for the failed call | Is sampling head-based or tail-based, and at what rate? | That head sampling dropped the span before the error was known | Switch to tail-based sampling that always keeps errors and high latency; raise the rate for the hot route |
| Error rate is back to baseline after the mitigation | Do the golden signals hold for 2–3x the alert window? | A real recovery versus a transient dip before it climbs again | Keep the incident open until signals are stable, then resolve and schedule the postmortem |
References
- Google SRE Book — Monitoring Distributed Systems (four golden signals, symptom alerting)sre.google
- OpenTelemetry — Signals (traces, metrics, logs)opentelemetry.io
- Brendan Gregg — The USE Methodbrendangregg.com
- Grafana — The RED Method: how to instrument your servicesgrafana.com
- PagerDuty — Incident Response Documentationresponse.pagerduty.com
A request-serving service with Prometheus-style metrics, structured logs, and OpenTelemetry tracing; Kubernetes used for deploy/rollback examples. Golden signals per Google SRE; RED per Grafana/Weaveworks; USE per Brendan Gregg. Swap CAPS placeholders and tool-specific commands for your stack.