← All cheatsheets

System

System Design 101 — load balancing, caching, queues, and the trade-offs

The five building blocks of a scalable backend — load balancing, caching, databases, queues, and replication — and the consistency, latency, availability, and cost trade-offs at every hop of a request.

Docs-checkedsystem-designscalingcachingqueuescap

Quick reference — reach for these first

Load balancernginx / HAProxy / ALBSpreads traffic across N app instances and health-checks them, routing around dead nodes. Trades a hop of latency for availability + horizontal scale.
CacheRedis / MemcachedKeeps hot data in memory near the app (cache-aside + TTL). Cuts read latency and DB cost, at the risk of serving stale data.
DatabasePostgres / DynamoDBDurable system of record. SQL gives joins + ACID; NoSQL gives horizontal scale + flexible schema. Index the columns you filter on.
QueueKafka / SQS / RabbitMQDecouples producers from consumers. Absorbs spikes and moves slow work off the request path, trading immediate consistency for throughput.
Replicationleader + read replicasFollowers add read capacity and availability; writes still go to the single leader. Replica lag means reads are eventually consistent.

One request: client to load balancer to cache/app to database, with a queue for async work

Request flow through a scalable web backendA client calls a load balancer that forwards to an app server. The app checks a cache first using cache-aside with a TTL, then reads or writes the database. Slow work is pushed onto a queue and handled asynchronously by a worker.ClientLoadBalancerAppDatabaseleader + replicasWorkerCacheQueueHTTPread / writecache-aside(TTL)async

Text equivalent: Left to right the flow is Client, Load Balancer, App, Database. The App also talks to a Cache drawn above it (cache-aside with a TTL, a two-way arrow) and pushes async work down into a Queue, which a Worker beside the Database consumes.

Common tasks

Task · Command · What to check

TaskCommandWhat to check
Put a load balancer in front of the app tierupstream app { server APP1:PORT; server APP2:PORT; }curl -s http://LB_HOST/whoami returns different backend IDs across calls
Add health checks so the LB drops dead nodescheck: GET /health interval=INTERVAL fails=NStop APP1; traffic shifts entirely to APP2 within INTERVAL*N
Add a cache-aside read path in front of the DBGET KEY -> miss -> SELECT ... -> SET KEY VALUE EX TTLSecond read of KEY is served from cache: redis-cli GET KEY returns a value
Put a TTL on every cache key so staleness is boundedredis-cli SET KEY VALUE EX SECONDSredis-cli TTL KEY returns a positive, shrinking number
[PROD] Invalidate the cache on write to avoid stale readsUPDATE TABLE ...; redis-cli DEL KEYRead immediately after the write returns the new value, not the old one
Index the columns your hot queries filter onCREATE INDEX idx_col ON TABLE(COLUMN)EXPLAIN on the query shows an Index Scan, not a Seq Scan
Add a read replica and route reads to itwrites -> LEADER ; SELECT -> REPLICAReplica lag stays under THRESHOLD: SHOW REPLICA STATUS / pg_stat_replication
Move slow side-effects off the request pathproducer -> QUEUE -> worker consumes asyncAPI returns 2xx in ms; the job completes later and its effect appears
Add a dead-letter queue for poison messagesmaxReceiveCount=N -> route to DLQA message that always fails lands in the DLQ instead of retrying forever
Add jitter + backoff to prevent a cache stampedettl = BASE + random(0, JITTER) ; retry sleep = 2^n + rand()A mass TTL expiry no longer spikes DB connections to the ceiling
[PROD] Eliminate single points of failure across AZsrun 2+ of every tier across AZ_A, AZ_B behind the LBDrain AZ_A; the service stays up and serves requests from AZ_B
Load-test to find the real bottleneck before scalingk6 run --vus VUS --duration DURATION script.jsp99 latency stays under SLO at target RPS; the saturating tier is identified

Trace one read, then one write, through LB -> cache -> app -> DB (+ queue)

read + write path

1
Client calls the public endpoint. The load balancer picks a healthy app node (availability: it skips drained/dead nodes; cost: one extra network hop; latency: ~1 ms).
curl -s https://APP_HOST/users/USER_ID
# routed to app-node-2 (app-node-1 draining)
2
App checks the cache first (cache-aside). A HIT returns in ~1 ms and skips the DB entirely — big win on latency and DB cost, but it may be stale if the profile changed since the SET.
redis-cli GET user:USER_ID
# (hit) {"name":"Ada",...}   TTL 42
3
On a MISS the app reads a replica. Read replicas add read capacity and availability, but the row may be behind the leader — eventual consistency.
SELECT * FROM users WHERE id = USER_ID;  -- on REPLICA
# 1 row (replica ~200 ms behind leader)
4
App repopulates the cache with a TTL so the next read is cheap and fast; the TTL caps how long a stale value can live.
redis-cli SET user:USER_ID '{...}' EX 60
# OK
5
[PROD] Now a WRITE. It must go to the LEADER only — replicas are read-only. Strong consistency at the leader, but the single leader is a potential SPOF.
UPDATE users SET name='Ada L.' WHERE id=USER_ID;  -- on LEADER
# UPDATE 1
6
[PROD] Invalidate the cache on write (DEL, not overwrite) so readers don't serve the old value; DEL avoids a lost-update race between concurrent writers.
redis-cli DEL user:USER_ID
# (integer) 1
7
Push slow side-effects (email, search reindex) onto a queue instead of blocking the response. Keeps write latency low; the worker runs async, trading immediate consistency for throughput.
PUBLISH jobs '{"type":"reindex","id":USER_ID}'
# queued; API returns 200 in ~5 ms
8
Verify. Read-after-write from the leader returns the new value right away; the replica and cache converge within (replica lag + TTL).
SELECT name FROM users WHERE id=USER_ID;  -- LEADER
# Ada L.

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
DB CPU spikes every time a popular cache key expiresCorrelate DB load spikes with the TTL expiry of HOT_KEY (cache logs + DB metrics)Thundering herd: many requests miss at the same instant and all stampede the DBAdd request coalescing / single-flight so one miss repopulates, stagger TTLs with jitter, or refresh-ahead before expiry
One cache or DB node runs hot while the others sit idleMeasure per-key / per-partition request rate and find the skewed KEY or shardHot key or hot partition: traffic is concentrated on a single node, not spreadAdd a small in-process cache for the key, replicate it to several nodes, or shard with a better key to spread load
Users see stale data immediately after they save a changeMeasure replica lag (seconds behind leader) and confirm whether reads are routed to a REPLICAReplica lag + eventual consistency: the read hit a follower that had not yet applied the writeRoute read-after-write to the LEADER, or pin a user's session to read-your-writes until lag clears
Queue depth climbs steadily and never drainsCompare producer rate vs. consumer throughput and check worker error / DLQ countsBacklog: consumers cannot keep up — under-provisioned, stuck, or crash-looping on bad messagesScale out consumers, fix or DLQ poison messages, and autoscale workers on queue depth
The whole service goes down when a single node diesMap every tier and find any component running with only one instance or one leaderSingle point of failure: that tier has no redundancy or failoverRun 2+ instances per tier across AZs behind the LB; add automated leader failover for the database

References

Examples use nginx, Redis, PostgreSQL, and a generic message queue. The concepts map to any equivalent stack: HAProxy or an ALB for load balancing, Memcached for caching, MySQL or DynamoDB for storage, and Kafka, SQS, or RabbitMQ for queues. CAP-style trade-offs are framed as consistency vs. latency vs. availability vs. cost, not a single "right" answer.