← All cheatsheets

Reliability

SRE Fundamentals — SLIs, SLOs, error budgets, and burn-rate alerts

Define SLIs and SLOs, turn them into an error budget, and alert with multi-window multi-burn-rate rules that page on fast burn and ticket on slow burn — with the 99.9%/30-day math worked end to end.

Docs-checkedsresloslierror-budgetburn-rate

Quick reference — reach for these first

SLISLI = good events / valid eventsWhat you measure — a ratio users actually feel (success, fast-enough).
SLOSLO: SLI >= 99.9% over 30dInternal target for the SLI. Set it achievable, then ratchet up.
Error budgetbudget = 1 - SLO = 0.1%Allowed unreliability = 43.2 min / 30 days. Spend it deliberately.
Burn rateburn = (1 - SLI) / (1 - SLO)Multiples of budget spend; 1x = on pace to use it all exactly at period end.
SLASLA = SLO + consequencesContract with penalties. Keep the SLO STRICTER so you act before it breaches.

Multi-window multi-burn-rate: when to page vs ticket

Burn-rate alerting: page versus ticketError budget is one minus the SLO: for a 99.9 percent SLO that is 0.1 percent, about 43.2 minutes over 30 days. Burn rate is the observed error ratio divided by the budget rate. A 14.4x burn over a 1-hour window confirmed by 5 minutes spends 2 percent of the monthly budget and pages; a 6x burn over 6 hours confirmed by 30 minutes spends 5 percent and pages; a 1x burn over 3 days confirmed by 6 hours spends 10 percent and files a ticket.Multi-window, multi-burn-rate alertingerror budget = 1 - SLO = 0.1% = 43.2 min / 30 daysburn = (1 - SLI) / (1 - SLO) spent = burn x window / periodBURNLONG WINDOWSHORT WINDOWBUDGETACTION14.4x1 hour5 min2%PAGE6x6 hours30 min5%PAGE1x3 days6 hours10%TICKETPAGEburn fast enough to drain the budget in hours - wake a human now.TICKETreal but slow - fix it in business hours. Short window gates both.

Text equivalent: Error budget equals one minus the SLO. For a 99.9 percent SLO that is 0.1 percent, about 43.2 minutes over 30 days. Burn rate is the observed error ratio divided by the budget rate, and budget spent equals burn rate times window over period. A 14.4x burn over a 1-hour window confirmed by a 5-minute window spends 2 percent of the monthly budget and pages. A 6x burn over 6 hours confirmed by 30 minutes spends 5 percent and pages. A 1x burn over 3 days confirmed by 6 hours spends 10 percent and files a ticket. Page when the budget would be gone in hours; ticket when the burn is real but slow.

Common tasks

Step · PromQL / formula · Done when

StepPromQL / formulaDone when
Pick one user-journey SLI (availability or latency)SLI = good events / valid eventsThe SLI tracks something users feel, not an internal counter
Define 'good' events as a queryGOOD = REQUESTS_TOTAL{job="SERVICE",code!~"5.."}Query returns the count of successful requests
Record the SLI ratio over the windowjob:sli:ratio30d = sum(rate(GOOD[30d])) / sum(rate(VALID[30d]))Recorded value sits between 0 and 1
Set the SLO target and windowSLO = 99.9% over 30dTarget is at or below what the last 28 days actually achieved
Compute the error budget(1 - 0.999) * 30*24*60= 43.2 minutes of allowed badness per month
Add fast-burn PAGE alert [prod]ratio1h > 14.4*0.001 AND ratio5m > 14.4*0.001Fires only when ~2% of the budget burns in 1h
Add mid-burn PAGE alert [prod]ratio6h > 6*0.001 AND ratio30m > 6*0.001Fires when ~5% of the budget burns in 6h
Add slow-burn TICKET alert [prod]ratio3d > 1*0.001 AND ratio6h > 1*0.001Fires when ~10% of the budget burns over 3 days
Dashboard the budget remaining1 - (1 - ratio30d)/0.001Panel shows percent of monthly budget still left
Route alerts: page vs ticket [prod]severity=page -> PAGER ; severity=ticket -> QUEUEPage wakes on-call; ticket lands in the work queue, not the pager
Assign incident roles when a page firesIC + COMMS + OPSExactly one Incident Commander owns coordination
Write the postmortem trigger thresholduser-visible outage > THRESHOLD OR data lossThe trigger is documented, not decided case-by-case
Enforce the budget policy on exhaustion [CHANGES PROD]budget <= 0 -> freeze feature launchesFreeze is pre-agreed by dev and SRE, not negotiated mid-incident
Review the SLO every quartercompare SLO vs achieved SLI vs complaintsSLO still tracks user happiness; adjust if it drifted

99.9% / 30-day SLO: budget, burn rate, and the page-vs-ticket call

page-vs-ticket

1
State the SLO: 99.9% availability measured over a rolling 30 days
2
Convert the window to minutes
30 * 24 * 60
# 43200 minutes in the 30-day window
3
Compute the error budget (1 - SLO of the window)
(1 - 0.999) * 43200
# 43.2 minutes of allowed downtime per month
4
A spike hits 1.44% errors over the last hour — compute burn rate
0.0144 / 0.001
# burn rate = 14.4x
5
How much budget does 14.4x spend in that one hour?
14.4 * (60 / 43200)
# 0.02 = 2% of the WHOLE monthly budget in one hour
6
Time to fully exhaust the budget at this rate
43200 / 14.4
# 3000 min = 50h -> fast, severe -> PAGE (gated by a 5m window)
7
Contrast a slow burn: 0.1% errors over 3 days
0.001 / 0.001
# burn rate = 1x -> 10% of budget over 3 days -> TICKET, not a page
8
Decide: PAGE on fast burn (14.4x/1h confirmed by 5m, or 6x/6h confirmed by 30m); TICKET on slow burn (1x/3d confirmed by 6h). Page = budget gone in hours; ticket = real but fixable in business hours.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
SLO dashboard is green but users are complainingIs the SLI measured at the user's edge (LB/client), not server-internal?The SLI is a proxy, not the real experienceMove measurement to the edge and tighten the SLO so green means happy
Budget is always exhausted; on-call paged constantlyIs the SLO target above what the service hit in the last 28 days?The SLO is stricter than reality (too strict)Loosen the SLO to an achievable number, then ratchet it up as you improve
Pages fire but almost no budget was actually spentDoes the alert require BOTH a long and a short window to be burning?Missing short-window gate makes the alert flapAdd the short-window condition (5m with 1h) so brief blips self-resolve
Budget was blown before anyone got pagedIs there a fast-burn 14.4x/1h alert, not only 24h/3d rules?Only slow-burn alerting is configured (too late)Add the 1h/5m 14.4x page so severe burns alert within minutes
Error budget has hit zeroIs the feature-freeze policy written down and pre-agreed?No budget policy to invokeFreeze risky launches and spend the cycle on reliability until budget recovers
Team can't agree whether an incident breached the targetIs there one documented SLI definition (good vs valid events)?No agreed SLI existsWrite one SLI spec per journey before setting or debating any SLO
Alert thresholds are set to the SLA number (e.g. 99.5%)Are you alerting on the SLA or on a stricter internal SLO?SLA/SLO confusion — alerting at the contract lineSet the internal SLO tighter than the SLA so you act before the contract breaches

References

Math uses a 30-day rolling window (43,200 min) and event-based SLIs. Burn-rate windows and thresholds follow the Google SRE Workbook (14.4× / 6× / 1×). Swap CAPS PLACEHOLDERS (REQUESTS_TOTAL, SERVICE, THRESHOLD) for your own metric and label names.