Observability
Uptime & Health Monitoring — health checks, synthetics, and status pages
External uptime monitoring from the user's side: liveness vs readiness vs deep dependency health endpoints, multi-region synthetic checks, cron heartbeats (dead-man switch), status pages, SLA/error-budget math, and routing alerts to on-call (or an AI SRE assistant like Claude Opus 4.x for first-pass triage).
Quick reference — reach for these first
app.get('/livez',(_,r)=>r.sendStatus(200))Liveness: process is alive. No dependency calls — keep it trivially fast.curl -fsS -m5 -o /dev/null -w '%{http_code}\n' https://API_HOST/healthz-f fails the exit code on 4xx/5xx; -m5 caps the wait at 5s.Checkly ApiCheck → url:/healthz, frequency:60s, regions:[us-east,eu-west], retries:3Interval HTTP check from 2+ regions; retries kill single-blip flapping.backup.sh && curl -fsS -m10 https://hc-ping.com/UUIDDead-man switch: no ping inside the grace window → alert fires.99.9% = 43m/mo (8h45m/yr) · 99.95% = 22m/mo · 99.99% = 4m/moPick an SLA by the monthly downtime budget you can actually afford.External uptime monitoring pipeline
Text equivalent: A left-to-right pipeline: a multi-region synthetic monitor and a cron heartbeat probe a service exposing /livez (liveness), /readyz (readiness) and /healthz (deep). The deep check queries Postgres; when Postgres fails, /healthz returns 503, which pages on-call and flips the public status page from Operational to Degraded. The cron heartbeat is a separate dead-man-switch path into the same pager.
Common tasks
Health endpoints · Synthetic & heartbeat checks · Alerting, status & SLA
| Health endpoints | Synthetic & heartbeat checks | Alerting, status & SLA |
|---|---|---|
| Add a liveness endpoint (process is up) | app.get('/livez',(_,res)=>res.sendStatus(200)) | curl -fsS -m2 http://localhost:PORT/livez returns 200 |
| Add a readiness endpoint (ready for traffic) | app.get('/readyz',(_,res)=> ready ? res.sendStatus(200) : res.sendStatus(503)) | Returns 503 until warmup/migrations/pool are ready, then 200 |
| Add a deep dependency check | app.get('/healthz',async(_,res)=>{try{await db.query('SELECT 1');res.sendStatus(200)}catch{res.sendStatus(503)}}) | Stop the DB → /healthz returns 503 instead of 200 |
| Wire Kubernetes probes to the right endpoints | livenessProbe.httpGet.path=/livez ; readinessProbe.httpGet.path=/readyz | kubectl describe pod POD shows Liveness/Readiness probes passing |
| Curl-test any check with a hard timeout | curl -fsS -m5 -o /dev/null -w '%{http_code} %{time_total}s\n' https://HOST/healthz | Prints 200 + latency; non-zero shell exit on any failure |
| Create a synthetic uptime check | Checkly/UptimeRobot/Pingdom → GET https://HOST/healthz, interval 60s | Dashboard shows the check green across all configured regions |
| Require multi-region + retries before paging | degraded when >=2 regions fail; retries: 3 | A single-region network blip no longer sends a page |
| Set up a heartbeat (dead-man switch) for a cron job | 0 * * * * mycron.sh && curl -fsS -m10 https://hc-ping.com/UUID | Disable the cron → healthchecks.io flips the check to 'down' after the grace window |
| Instrument start / fail / success pings | curl .../UUID/start ; mycron.sh && curl .../UUID || curl .../UUID/fail | Dashboard shows run duration and distinguishes failures from misses |
| Route alerts to on-call paging | check → PagerDuty/Opsgenie integration key → escalation policy | Send a test alert; the on-call phone actually pages |
| [PUBLIC-FACING] Publish a status page tied to the checks | Statuspage/Better Stack component ← uptime check result (auto-incident on) | Force a check down → an incident auto-posts to the public page |
| Define the SLA and error budget | target 99.9% → 43m/mo budget; alert at 50% budget burn | Monthly uptime report stays under the downtime budget |
Green ping, red reality: adding a dependency-aware health check
incident
curl -s -o /dev/null -w '%{http_code}\n' https://HOST/
# 200 (the check hits / only — it never touches the DB)curl -s -o /dev/null -w '%{http_code}\n' https://HOST/healthz
# 503 (Postgres connection pool exhausted)app.get('/healthz',async(_,res)=>{try{await db.query('SELECT 1');res.sendStatus(200)}catch{res.sendStatus(503)}})
# /healthz now returns 503 whenever Postgres is downcheckly: url=/healthz, frequency=60s, regions=[us-east,eu-west], retries=3
# Check goes red within ~1 min of a real DB outage, not on single blips# in staging: stop Postgres and watch the alert + status component
# PagerDuty alert fired; public status page flipped to 'Degraded'curl -fsS -m5 https://HOST/healthz && echo OK
# 200 ... OK (alert auto-resolves; status returns to Operational)Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Check says DOWN but the site loads fine for you | curl -fsS -A 'MonitorBot' the exact monitored URL from a different network/region | Whether it's a wrong path, a blocked bot user-agent/IP, or a region-specific outage vs a false alarm | Fix the monitored path, allowlist the monitor's UA and IP ranges, or add a second region to confirm before paging |
| Alerts flap — down, then up, repeatedly | Inspect check history for single-region, single-sample failures | Whether failures are transient network noise vs a real outage | Require N consecutive failures across 2+ regions plus retries before the check pages |
| Health check is green but users are getting errors | curl the health endpoint and confirm it actually touches the DB/deps, not just returns 200 from '/' | Whether the check is shallow (process-only) and blind to dependency failures | Add a dependency-aware deep check (DB SELECT 1, cache ping) that returns 503 when a dependency is down |
| A cron/batch job silently stopped running | Is there a heartbeat (dead-man switch) on that job at all? | Whether 'no news' is being silently treated as 'healthy' | Add a success ping (hc-ping) and set a grace window so a missed run alerts |
| Status page still shows 'Operational' during an outage | Confirm the monitor is wired to the status-page component/auto-incident automation | Whether the detection→publish automation is actually connected | Link the check to the component or enable auto-incidents, then test end-to-end |
| Liveness probe keeps restarting a healthy pod | Check whether liveness calls a slow dependency or has too-tight timeout/failureThreshold | Whether liveness is wrongly doing readiness's job (deep dependency checks) | Make liveness process-only; move dependency checks to readiness; raise initialDelaySeconds/timeout |
| Readiness passes but first requests 5xx right after a deploy | Does /readyz wait for migrations, cache warmup, and the connection pool? | Whether traffic is admitted before the app is truly ready to serve | Gate readiness on warmup complete so the load balancer holds traffic until the pod is ready |
References
- Kubernetes — Liveness, Readiness & Startup Probeskubernetes.io
- healthchecks.io — Monitoring cron jobs / dead-man's switchhealthchecks.io
- Checkly — API & uptime monitoring docschecklyhq.com
- Atlassian Statuspage — support documentationsupport.atlassian.com
- Google SRE Book — Availability table (the nines)sre.google
Examples use an HTTP JSON service (Node/Express shown) behind Kubernetes with Postgres; adapt the endpoint code and probe YAML to your stack. HOST, PORT, UUID, API_HOST, and region names are CAPS placeholders. Heartbeat examples use healthchecks.io (hc-ping.com/UUID); the same dead-man-switch pattern applies to Cronitor or Better Stack.