Data
Kafka — topics, producers, consumers, and lag
Fast-lookup reference for Apache Kafka: the topic/partition/offset mental model, producer keys and acks, consumer groups and rebalancing, offset commits, and consumer lag — the #1 health signal — with the CLI to diagnose and drain a backlog.
Quick reference — reach for these first
kafka-topics.sh --bootstrap-server localhost:9092 --listEnumerate topics on the cluster (add --describe TOPIC for partitions/replicas).kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUPThe health check: shows CURRENT-OFFSET, LOG-END-OFFSET, LAG per partition + consumer owner.kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC --from-beginning --property print.key=trueTail/replay messages to inspect payloads (drop --from-beginning to read only new records).kafka-console-producer.sh --bootstrap-server localhost:9092 --topic TOPIC --property parse.key=true --property key.separator=:Type KEY:VALUE lines; the key determines the target partition.kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group GROUP --topic TOPIC --reset-offsets --to-earliest --executeRewinds/skips a group's committed position. Group must have no active members. --dry-run first.Topic → partitions → offsets, and how lag is measured
Text equivalent: A producer sends keyed records into a topic split into three partitions, each an append-only log of offset-numbered cells. A consumer group with three consumers reads the partitions one-to-one. On partition 0, the consumer's committed offset sits behind the log-end offset, and the gap between them is labeled LAG.
Common tasks
Inspect · Operate · Recover
| Inspect | Operate | Recover |
|---|---|---|
| List topics on the cluster | kafka-topics.sh --bootstrap-server localhost:9092 --list | Expected topic names print; cluster is reachable. |
| Inspect a topic's partitions and replication | kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic TOPIC | PartitionCount, ReplicationFactor, Leader, Isr columns shown; Isr size == ReplicationFactor. |
| Create a topic with N partitions | kafka-topics.sh --bootstrap-server localhost:9092 --create --topic TOPIC --partitions 6 --replication-factor 3 | 'Created topic TOPIC.' — partition count caps consumer-group parallelism. |
| Increase partitions on an existing topic | kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic TOPIC --partitions 12 | PartitionCount rises on --describe. NOTE: changes key→partition mapping; cannot decrease partitions. |
| Produce a keyed test message | printf 'user-42:hello\n' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic TOPIC --property parse.key=true --property key.separator=: | Same key always lands on the same partition (hash of key mod partitions). |
| Consume and print keys/offsets | kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC --from-beginning --property print.key=true --property print.offset=true | Records appear with offset and key; Ctrl-C to stop. |
| List all consumer groups | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list | Your application's group id appears. |
| Read lag for a group (primary signal) | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUP | LAG column is near 0 and not trending up; every partition has a CONSUMER-ID. |
| See group members and partition assignment | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUP --members --verbose | Active member count ≤ partition count; no partition is unassigned. |
| Check the group's rebalance/coordinator state | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUP --state | STATE = Stable (not PreparingRebalance/CompletingRebalance flapping). |
| Dry-run an offset reset before executing | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group GROUP --topic TOPIC --reset-offsets --to-earliest --dry-run | Prints the NEW-OFFSET it would set; no change applied yet. |
| Skip a poison message by shifting offsets (DESTRUCTIVE) | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group GROUP --topic TOPIC:2 --reset-offsets --shift-by 1 --execute | Group has NO active members first; CURRENT-OFFSET advances past the stuck record. |
| Inspect retention on a topic | kafka-configs.sh --bootstrap-server localhost:9092 --describe --topic TOPIC --all | grep retention | retention.ms / retention.bytes shown (default retention is cluster log.retention.hours = 168h). |
| Delete a topic (DESTRUCTIVE, PROD-CHANGING) | kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic TOPIC | All data and offsets for the topic are removed; requires delete.topic.enable=true. Confirm topic is unused first. |
Consumers are falling behind — diagnose and drain growing lag
troubleshooting
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor
# TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
# orders 0 4820133 6060251 1240118 consumer-1-abc
# orders 1 4790021 6058900 1268879 -
# orders 2 4805554 6061002 1255448 consumer-1-abckafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor --members --verbose
# CONSUMER-ID HOST #PARTITIONS ASSIGNMENT
# consumer-1-abc /10.0.0.5 2 orders(0,2)
# (only 1 active member for a 3-partition topic → 1 partition queued with no owner)kubectl scale deploy/orders-processor --replicas=3 # or start 2 more consumer instances
# Rebalance assigns orders(0),(1),(2) to three distinct members.watch -n 15 'kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor'
# orders 0 ... LAG 612004 consumer-1-abc
# orders 1 ... LAG 598110 consumer-2-def ← now owned, draining
# orders 2 ... LAG 604887 consumer-3-ghikafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor --state
# STATE: Stable #MEMBERS: 3 — LAG ~0 and steady: healthy.
# (Tip: an LLM such as Claude Opus 4.x / Sonnet can summarize the --describe output into an on-call alert, but the LAG trend is the ground truth.)Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Consumer lag is large and growing | kafka-consumer-groups.sh --describe --group GROUP — watch the LAG column over 30s | Consumers can't keep up: either a consumer is down (partition unowned) or the handler is slower than the produce rate. | If partitions are unassigned, restart/scale consumers up to the partition count; if all owned but LAG still climbs, speed up the handler (batch, async I/O) or raise max.poll.records / add partitions. |
| Group keeps rebalancing (PreparingRebalance loops, 'storms') | --describe --group GROUP --state and broker logs for repeated JoinGroup; check consumer poll loop timing | Members are being evicted because they miss heartbeats or exceed max.poll.interval.ms (slow processing between polls). | Lower per-poll work (max.poll.records) or raise max.poll.interval.ms; tune session.timeout.ms/heartbeat.interval.ms; enable cooperative-sticky assignor to avoid stop-the-world rebalances. |
| Under-replicated or offline partitions | kafka-topics.sh --describe --under-replicated-partitions (Isr smaller than Replicas) | A broker is down or lagging, so some replicas fell out of the in-sync set — durability and (for min.insync.replicas) acks=all writes are at risk. | Bring the failed broker back or reassign replicas; check disk/network on the lagging broker; verify min.insync.replicas vs replication.factor so acks=all can still be satisfied. |
| One partition's lag is stuck; consumer makes no progress (poison message) | --describe --group GROUP — one partition's CURRENT-OFFSET is frozen while LOG-END-OFFSET rises; check consumer error logs | A record repeatedly throws in the handler, so the offset never commits and the partition blocks. | Fix the handler and/or route failures to a dead-letter topic; as a stopgap stop the group and --reset-offsets --shift-by 1 on that TOPIC:PARTITION to skip the record (DESTRUCTIVE — you drop that message). |
| Throughput is capped no matter how many consumers you add | kafka-topics.sh --describe --topic TOPIC — compare PartitionCount to active group members | Parallelism in a group equals partition count; extra consumers beyond that sit idle with no partitions. | Increase partitions (kafka-topics --alter --partitions N) to raise max parallelism — note it rekeys future messages and never reorders within a key; size partitions ahead of peak. |
| Messages for the same key land out of order or on different partitions | Confirm the producer sets a key and that PartitionCount hasn't changed recently | Ordering is guaranteed only within a partition; a null key uses sticky partitioning (Kafka 2.4+, KIP-480) — batched onto one partition then rotated, not per-record round-robin — and adding partitions changes key→partition mapping. | Always set a stable partition key for ordered streams; avoid altering partition count on order-sensitive topics, or plan a migration. |
References
- Apache Kafka Documentationkafka.apache.org
- Managing Consumer Groups (basic ops)kafka.apache.org
- Design: The Log & Partitioningkafka.apache.org
- Producer Configs (acks)kafka.apache.org
- Topic Configs (retention)kafka.apache.org
Classic Kafka CLI shipped with Apache Kafka (kafka-*.sh wrappers) against a broker at localhost:9092; placeholders TOPIC/GROUP in CAPS. Partitioning assumes the default partitioner (hash of key mod partition count). Offset/lag semantics follow committed offsets stored in __consumer_offsets. Retention default cited as the cluster log.retention.hours = 168h (7 days) unless a per-topic retention.ms/retention.bytes override is set.