Consumer Lag Monitoring Tools in Kafka
Is your application keeping up with the data flowing into Kafka, or quietly falling behind? Learn what consumer lag really measures, where the numbers come from, and how the main monitoring tools compare, so you can pick the right one and trust what it tells you.
Picture a restaurant kitchen. Waiters clip order tickets to a rail above the counter. Cooks take tickets from the left end and prepare the dishes. On a calm evening, there are only one or two tickets on the rail, and every dish leaves the kitchen quickly. On a busy Saturday, tickets arrive faster than the cooks can handle them. The rail fills up. Nobody has dropped an order, and nobody is doing anything wrong, but customers are waiting longer and longer.
The number of tickets waiting on that rail is the kitchen’s lag. A smart manager watches it all evening. A short rail means all is well. A rail that keeps growing means trouble is coming, even if every cook still looks busy and cheerful.
Kafka has exactly the same situation. Producers clip messages onto a topic. Consumers take messages off and process them. The count of messages waiting to be processed is called consumer lag, and it is probably the single most important health number for any application that reads from Kafka. This guide explains what lag is, how it is measured, and how the popular monitoring tools show it to you, from a simple built-in command all the way to full dashboards with alerts.
Consumer lag is the gap between the newest message in a partition and the last message a consumer group has confirmed as processed. Consumer lag monitoring tools are programs that read those two numbers on a schedule, show them as graphs, and raise alerts when the gap grows. Common choices include the built-in command line tool, Burrow, Prometheus exporters with Grafana, open-source web interfaces, commercial platforms, and cloud provider metrics.
Why Consumer Lag Deserves Your Attention
Most Kafka problems show up as lag long before they show up as errors.
The quiet symptom
Many failures in a data pipeline make no noise. A consumer that has become slow does not crash. A consumer stuck on one troublesome message does not print a warning every second. From the outside, the application looks alive. Yet the data it produces gets older by the minute. Lag is the number that turns this quiet trouble into something you can see.
Think of the water level in a bathtub with the tap running and the drain partly blocked. Nothing is broken and no alarm sounds. But if you glance at the water level rising a little more each minute, you know that soon it will overflow. Lag is the water level.
What lag tells you
How old is the data?
Large lag means the results your users see are based on old events.
Can consumers keep up?
Lag that keeps growing means consumers are slower than producers.
Is anything stuck?
Lag that stays fixed while new data arrives suggests a stuck or dead consumer.
Will we lose data?
If lag grows larger than what the topic keeps, old messages are deleted before they are read.
Why this needs tools
A single lag number changes every second. It is different for every partition, for every topic, and for every consumer group. A large company may run thousands of consumer groups. No person can check these by hand. Monitoring tools collect the numbers automatically, keep a history so you can see trends, and send alerts. Choosing well among them saves many nights of lost sleep.
A tool that shows lag is only useful if you understand what the number means. A number without meaning leads to false alarms, or worse, to false comfort. The first half of this guide builds that meaning. The second half compares the tools.
The Vocabulary You Need First
A handful of words unlock everything else in this article.
Topic, partition, and offset
A Kafka topic is a named notebook of messages. A big topic is split into pieces called partitions, so many machines can share the work. Every message inside a partition gets a position number called an offset: 0, 1, 2, 3, and so on, always growing. Messages are only added at the end.
Log end offset
The log end offset is the offset that the next new message will receive. It marks the end of the partition, so it moves up every time a producer writes. You will see this shortened to LEO.
Consumer and consumer group
A consumer is a program that reads messages. Consumers that work together on the same job form a consumer group, identified by a name such as “billing-service”. Kafka shares the partitions of a topic among the members of a group. Each partition is read by exactly one member of the group at a time.
Committed offset
As a consumer works, it tells Kafka how far it has got. This is called committing an offset. The committed offset is the last confirmed position for that group in that partition. If the consumer restarts, it continues from there. Commits are stored in a special internal topic named __consumer_offsets.
Position
A consumer also keeps its own position, which is the offset of the next message it will read. The position can be ahead of the last commit, because commits usually happen every few seconds, not after every message.
High watermark and stable offset
Kafka copies each partition to several brokers. The high watermark is the offset up to which all copies are safely stored. Consumers only see messages below it. When transactions are used, consumers may read only up to the last stable offset, which is the point before any still-open transaction. Lag is measured against these visible limits in such cases.
| Word | Simple meaning |
|---|---|
| Offset | A message’s position number in its partition |
| Log end offset | The position where the next message will be written |
| Committed offset | The last position a consumer group has confirmed |
| Consumer group | A team of consumers sharing the partitions of a topic |
| Lag | How far the group is behind the end of the partition |
| Rebalance | Kafka reshuffling partitions among group members |
The Big Picture: Lag Is a Subtraction
The formula is short enough to remember forever.
For one partition and one consumer group:
The log end offset says where the producers are. The committed offset says where the consumers are. The difference is how many messages sit in between, waiting.
flowchart LR
subgraph P["One partition, shown as a line of messages"]
M0["Offset 0 to 899: already processed"] --> M1["Offset 900 to 999: waiting"] --> M2["Offset 1000: next message to be written"]
end
C["Committed offset = 900"] -.-> M1
L["Log end offset = 1000"] -.-> M2
LAG["Lag = 1000 - 900 = 100 messages"]
M1 --> LAG
The picture shows one partition as a line. The first nine hundred messages are done. The next one hundred are waiting. The end of the line is at offset one thousand. The lag is one hundred messages.
Imagine a library book you are reading. The last page number in the book is the log end offset. The page where your bookmark sits is the committed offset. The pages between are what you still have to read. That is your lag. If someone keeps adding new pages to the end while you read, your lag can stay large even though you keep reading.
Two important consequences
- Every consumer group has its own lag. Two groups reading the same topic can have very different lag, because each keeps its own bookmark.
- Lag is not a property of the topic alone. A topic has an end. A group has a position. Only together do they produce a lag number.
A Worked Example with Three Partitions
Real topics have many partitions, so let us add up a small case.
The “orders” topic has three partitions. The “billing-service” group is reading it. At one moment, the numbers are as follows.
| Partition | Log end offset | Committed offset | Lag |
|---|---|---|---|
| 0 | 5,000 | 4,990 | 10 |
| 1 | 5,200 | 5,190 | 10 |
| 2 | 9,000 | 4,000 | 5,000 |
Total lag and maximum lag
Monitoring tools usually show two summaries. The total lag adds all partitions: 10 + 10 + 5,000 = 5,020 messages. The maximum lag takes the largest single partition: 5,000 messages.
The two summaries tell different stories. The total says “about five thousand messages are waiting”. The maximum, together with the per-partition view, says “almost all of them are stuck in partition 2”. That points to one troubled partition, not a general slowdown.
flowchart TD
G["Group: billing-service"] --> P0["Partition 0: lag 10"]
G --> P1["Partition 1: lag 10"]
G --> P2["Partition 2: lag 5000"]
P2 --> Q{"Why so large?"}
Q --> A1["One consumer stuck"]
Q --> A2["One hot key producing too much"]
Q --> A3["One slow message repeated"]
A supermarket has three checkout lanes. Two lanes have two shoppers each. The third has fifty shoppers. If a manager only looks at the total of fifty-four shoppers, she may open more lanes. If she looks lane by lane, she sees that lane three is jammed, perhaps by one customer with a problem, and opening a new lane will not fix it.
Always keep a per-partition view available. Summaries are good for alerts and for the first glance, but the per-partition view is what solves the mystery.
Lag in Messages versus Lag in Time
One hundred thousand messages of lag can be nothing, or a disaster. Time gives it meaning.
The trouble with counting messages
A lag of 100,000 messages sounds scary. But if the consumer handles 50,000 messages per second, that lag clears in two seconds. On the other hand, a lag of 1,000 messages sounds tiny, but if the consumer handles one message per minute, that lag is more than sixteen hours.
Time lag: how old is the oldest waiting message?
Time lag (also called latency lag) expresses lag as a duration: how long ago was the first unprocessed message written? This is the number that business people understand. “Our dashboard is fifteen minutes behind” is clearer than “we have 900,000 messages of lag”.
| Measure | How to get it | Strength | Weakness |
|---|---|---|---|
| Offset lag (messages) | Subtract two offsets | Exact, cheap, simple | Hard to judge without knowing speed |
| Time lag, direct | Read the timestamp of the first unread message | Precise | Needs an extra read per partition; costly at scale |
| Time lag, estimated | Combine lag with recent produce or consume rates, or interpolate from remembered offset and time pairs | Cheap, good enough for trends | An estimate, not exact |
Telling a friend “there are forty cars ahead of us” does not say how long you will wait. Telling them “the queue is about ten minutes long” does. Offset lag is the number of cars. Time lag is the wait.
Using both
Good monitoring keeps both. Offset lag is precise and easy for machines. Time lag is meaningful for people and works well in alerts such as “the payments consumer is more than two minutes behind”. Several tools you will meet later estimate time lag for you, since measuring it exactly for thousands of partitions would put a heavy load on the brokers.
Why Lag Grows: Common Causes
Knowing the usual suspects makes every lag graph easier to read.
A traffic spike
More messages arrive than usual, for example during a sale. Lag rises, then falls once traffic calms.
Slow processing
Each message takes longer than before, perhaps because a database or service it calls has become slow.
A stuck message
One message keeps failing and being retried, blocking everything behind it in that partition.
Rebalances
Members join and leave, and while partitions are reassigned, reading pauses. Frequent rebalances mean frequent pauses.
Skew and hot keys
One partition receives far more messages than the others because a few keys are extremely busy.
Too few consumers
There are not enough members, or not enough partitions to add more, to share the work.
Pauses and resource limits
Long garbage collection pauses, starved CPU, or full memory make a consumer stall.
Slow connections
Distance or congestion between consumers and brokers reduces how fast messages arrive.
Reading the cause from the shape
| What you see | Likely cause |
|---|---|
| All partitions rise together, then fall | A temporary traffic spike or a short slowdown |
| All partitions rise steadily and never fall | Consumers are permanently slower than producers |
| One partition rises, others are flat | A stuck message, a hot key, or a single unhealthy consumer |
| Sharp cliff upward, then a long slow decline | A pause or outage followed by catching up |
| Saw-tooth pattern with a regular rhythm | Batch-like commits or scheduled bursts, usually harmless |
| Lag stays constant while new data arrives | The consumer has stopped, but commits are stale |
Adding more consumers does not always help. A group can have at most one active consumer per partition. If a topic has six partitions and your group already has six consumers, a seventh will simply sit idle. Skew and stuck messages also do not respond to more consumers.
Is Lag Good or Bad? Reading the Trend
A lag number on its own says little. The direction of travel says a lot.
Beginners often ask, “What is a good lag value?” There is no single answer, because it depends on your traffic and your business. A safer question is, “Is lag steady, shrinking, or growing?”
Steady and low
Consumers keep pace. Small ups and downs come from commit timing and batching. This is healthy.
High but steady
Consumers keep pace with the producers, but a backlog remains. It will not clear on its own. Decide whether the delay is acceptable.
Rising
Producers are faster than consumers. This is the classic warning sign. Act before lag reaches the retention limit.
Falling
Consumers are faster than producers and are working through a backlog. Estimate how long it will take.
Flat at zero with no activity
Either producers are quiet, or something is wrong upstream. Compare with the produce rate before deciding.
A doctor cares less about one temperature reading and more about whether the fever is rising or falling. Lag works the same way: the trend is the diagnosis.
The best alert is often on the rate of change of lag, or on lag that has stayed above a value for a set period, and not on the raw number at a single moment. Brief bumps are normal. Sustained growth is not.
Architecture: Where the Numbers Come From
Every lag monitoring tool relies on the same few data sources. Once you know them, the tools are easy to compare.
flowchart TD
subgraph K["Kafka Cluster"]
BR["Brokers hold partitions and know each log end offset"]
CO["Group coordinator tracks group members and state"]
OT[("__consumer_offsets topic holds committed offsets")]
end
subgraph APP["Consumer applications"]
C1["Consumer 1 with client metrics"]
C2["Consumer 2 with client metrics"]
end
subgraph MON["Monitoring side"]
AD["Admin API reader"]
JM["Metrics scraper for client metrics"]
ST[("Time series store")]
DASH["Dashboards and alerts"]
end
C1 --> OT
C2 --> OT
C1 -->|"reads messages"| BR
C2 -->|"reads messages"| BR
AD -->|"ask for end offsets"| BR
AD -->|"ask for committed offsets"| OT
AD -->|"ask for group state"| CO
C1 -.-> JM
C2 -.-> JM
AD --> ST
JM --> ST
ST --> DASH
The diagram has three groups of boxes. The Kafka cluster knows the end offset of every partition through its brokers, and knows the committed offset of every group through the internal __consumer_offsets topic and the group coordinator. The consumer applications commit offsets and also produce their own client metrics. The monitoring side has two ways of gathering data: an admin reader that asks the cluster directly, and a metrics scraper that collects the numbers each consumer reports about itself. Both feed a time series store, which powers dashboards and alerts.
| Source | What it gives | Seen from |
|---|---|---|
| Broker log end offsets | Where producers are, per partition | Outside, via the admin interface |
| Committed offsets | Where each group says it is | Outside, via the admin interface |
| Group coordinator | Members, assignments, and state such as stable or rebalancing | Outside |
| Consumer client metrics | The consumer’s own view of lag and speed | Inside the consumer, via JMX or a metrics library |
To know how far behind a student is, a teacher can check the class register (committed offsets) against the syllabus (end offsets). Or she can ask the student, “How far do you think you are?” (client metrics). The register is easy to check even when the student is absent. The student’s own answer is more detailed, but only available while the student is in the room.
Internal Working: How a Commit Becomes a Lag Number
Follow the tiny steps that end with a number on your dashboard.
A consumer reads and processes messages
It advances its position as it goes, one batch at a time.
The consumer commits an offset
Automatically on a timer, or when the application asks. The commit is sent to the group coordinator.
The commit is stored
The coordinator writes it as a record in the __consumer_offsets topic. The key is the group, topic, and partition. The value is the offset.
Producers keep writing
Each write moves the log end offset up on the broker that leads the partition.
The monitoring tool polls
On a schedule, it asks for the committed offsets of each group and the end offsets of each partition.
The tool subtracts
For every partition of every group, it computes end offset minus committed offset.
The number is stored and shown
The result is saved with a timestamp, drawn as a graph, and compared with alert rules.
Why the number is always a little old
Two delays sit inside the process. First, consumers commit only now and then. Between commits, the real progress is ahead of the committed offset, so measured lag looks larger than the truth. Second, the monitoring tool polls only every few seconds, so the picture is a snapshot from a moment ago. This is why lag graphs often look like a saw with small teeth: lag rises until a commit arrives, then drops, and repeats.
Some applications do not store their progress in Kafka at all. They keep offsets in their own database, for example to make writes and progress commit together. The tools in this guide will see no committed offsets for such a group, and will report nothing or misleading numbers. For those applications, you must measure lag inside the application itself.
Real-world picture: an invisible group
A team stores consumer progress in its own database. Their lag dashboard shows no group at all, and everyone assumes the application is healthy. In reality, it fell hours behind after a slow database day. The team adds a small metric published from inside the application, comparing the offset it just processed with the end offset it fetches from Kafka, and finally sees the truth.
The Map: Six Families of Lag Monitoring Tools
Dozens of products exist, but they fall into a few families with shared strengths and weaknesses.
Built-in command line
A tool that ships with Kafka. Great for quick checks. It shows one moment, with no history and no alerts.
Client-side metrics
Numbers each consumer reports about itself through JMX or a metrics library. Detailed, but they vanish when the consumer dies.
External lag evaluators
Standalone services like Burrow that watch commits from outside and judge whether a group is healthy, without fixed thresholds.
Metric exporters
Small services that read offsets and publish them to a metrics system such as Prometheus, for graphs and alerts.
Web interfaces and platforms
Open-source consoles and commercial products that show topics, groups, and lag in one screen.
Cloud provider metrics
Managed Kafka services that publish lag numbers into the cloud’s own monitoring system.
flowchart LR
K["Kafka cluster"] --> F1["1. CLI: one-time look"]
K --> F3["3. Burrow: status verdicts"]
K --> F4["4. Exporters: metrics for graphs"]
K --> F5["5. Consoles and platforms: screens"]
K --> F6["6. Cloud provider metrics"]
APP["Consumer apps"] --> F2["2. Client metrics"]
F3 --> OUT["Alerts and dashboards"]
F4 --> OUT
F2 --> OUT
F5 --> OUT
F6 --> OUT
The picture shows that almost every family reads from the Kafka cluster itself. Only client-side metrics come from the consumer applications. The outputs, dashboards and alerts, are where you meet them in daily life.
Checking the health of a fleet of delivery vans can be done in several ways. You can walk up and look at the fuel gauge of one van (CLI). You can ask the driver how it feels (client metrics). You can hire an inspector who judges the whole fleet by its behavior (evaluators). You can install trackers that send data to a control room (exporters and platforms). Or you can rent vans from a company that provides its own tracking (cloud metrics).
Most healthy setups use more than one family. For example: an exporter with Grafana for daily dashboards and alerts, and the built-in command line tool for quick manual checks during an incident.
The Built-In Tool: kafka-consumer-groups
Every Kafka installation already includes a lag viewer. Start here.
What it does
Kafka ships with a command line program named kafka-consumer-groups. Its “describe” mode prints a table for a group, with one row per partition. Each row shows the topic, the partition number, the current committed offset, the log end offset, and the lag. It also shows which consumer instance currently owns the partition, along with its host and client name.
What else it can do
- List groups. See all the consumer groups the cluster knows about.
- Show group state. See whether a group is stable, rebalancing, empty, or dead.
- Show members. See who belongs to the group and which partitions they hold.
- Reset offsets. Move a group’s position, for example back to the start or to a point in time. This is powerful and dangerous, and is covered in the mistakes chapter.
- Delete a group. Remove an unused group, but only when it has no active members.
Strengths
- Already installed, nothing to set up
- Exact numbers straight from the cluster
- Shows members and hosts, useful in an incident
- Works even when the consumer application is dead
Limits
- A snapshot only, with no history or trend
- No graphs and no alerts
- Awkward across hundreds of groups
- No time lag
It is like the dashboard fuel gauge in a car. It tells you the level right now, very reliably. It does not draw a graph of your fuel use, and it will not phone you when the tank runs low.
Real-world picture: the first ten minutes of an incident
An alert says orders are late. The engineer opens a terminal, runs the describe mode for the orders group, and immediately sees that partition 4 has huge lag and belongs to a consumer on a host that just restarted. She has a lead in under a minute, before opening any dashboard.
Client-Side Metrics: The Consumer Reports Itself
Kafka consumers publish their own numbers, and these are worth using with care.
What the client offers
The Kafka consumer library exposes metrics through JMX, a standard way for Java programs to publish numbers. Among them are the largest lag across the partitions the consumer owns, the lag of each individual partition, and the average lag. There is also a metric for the opposite quantity, called lead, which is how far the consumer’s position is from the start of the partition. A shrinking lead warns that the consumer is falling toward the oldest retained messages, which can precede data loss.
Metrics libraries and agents can collect these numbers and send them to a monitoring system. Many application frameworks also include a bridge that turns them into ordinary metrics.
Why these numbers are attractive
- Very fresh. The consumer knows its true position, not only its last commit.
- Tied to the application. They come with the application’s own labels, such as service name and version.
- No extra service to run. They come from what you already deploy.
Why they are not enough alone
When a consumer crashes or hangs, its metrics stop arriving at exactly the moment you need them most. A missing graph is easy to mistake for “no lag”. Also, a group whose members are all gone has no client metrics at all, though its lag may be growing fast. This is why external monitoring, which reads committed offsets from the cluster, is the safety net.
| Question | Client metrics | External monitoring |
|---|---|---|
| Freshness of the number | Very fresh | As fresh as the last commit and poll |
| Works when consumers are dead | No | Yes |
| Sees groups you did not instrument | No | Yes |
| Extra setup per application | Yes | No, one setup for the whole cluster |
| Extra load on brokers | None | Small, from polling |
The healthiest design uses both: client metrics for detail and speed, and an external tool for coverage and safety.
A pilot’s own instruments give the most precise, up-to-the-second readings, but if the pilot cannot report, the control tower’s radar is the only view left. You want both.
Burrow: A Verdict Instead of a Number
Some lag is normal and some is alarming, and Burrow tries to tell the difference for you.
Where it came from
Burrow is an open-source lag monitoring service created at LinkedIn, the company where Kafka itself was born, and released publicly in the mid-2010s. It was built for a large environment with many consumer groups, where setting a hand-picked threshold for each one was impractical.
How it works
Burrow does not just compare lag to a number. It reads the stream of offset commits for each group and the log end offsets of each partition, and keeps a sliding window of recent observations, for example the last ten commits. It then applies simple rules over that window to decide the group’s health.
- Is the consumer committing at all?
- Is the committed offset moving forward?
- Is the lag going up, staying flat, or coming down across the window?
- Is the consumer making progress even though lag is not zero?
The status it reports
| Status | Meaning in plain words |
|---|---|
| OK | The group is keeping up, or lag is falling |
| WARN | Lag has been rising through the window, so the group is falling behind |
| ERR | The group has stopped consuming, or lag is high and not improving |
| STALL | The group commits offsets, but the offsets are not moving forward |
| STOP | The group has stopped committing offsets altogether |
| REWIND | The group’s committed offset has moved backward, for example after a reset |
Think of a coach watching a runner. She does not say “the runner is 200 metres behind, so panic”. She watches for a few minutes and says “he is falling further behind with each lap, so something is wrong”, or “he is behind but closing the gap, so relax”. Burrow acts like that coach, judging the trend.
How you use it
Burrow runs as a service and offers an HTTP interface for looking up the status of any group. It can also send notifications by email or to other services when a group’s status changes. It is often connected to dashboards by turning its status into metrics or by polling its interface.
Strengths
- No thresholds to tune per group
- Catches stuck consumers that plain lag hides
- Understands rising versus falling lag
- Good fit when there are very many groups
Limits
- Gives a status, not a rich history of graphs by itself
- No time lag estimate
- Another service to deploy and keep updated
- Judgement depends on the commit pattern of each application
Before adopting any long-lived open-source tool, look at how recently it has been updated and how active its community is. Check the current project page for its maintenance state, then decide whether it fits your plans.
Prometheus Exporters and Grafana
The most common setup in modern teams: an exporter, a time series database, and a dashboard.
The idea
An exporter is a small program that asks Kafka for offsets, does the subtraction, and publishes the results in a form that a metrics system can collect. Prometheus is a popular metrics system that visits each exporter every few seconds and stores the numbers. Grafana then draws graphs from that stored history and can send alerts.
flowchart LR
K["Kafka cluster"] --> EX["Lag exporter"]
EX -->|"metrics page"| PR[("Prometheus")]
PR --> GR["Grafana dashboards"]
PR --> AM["Alert manager"]
AM --> ON["Chat, email, or paging"]
The exporter reads from Kafka and offers a page of numbers. Prometheus collects that page on a schedule and stores it. Grafana draws graphs from what Prometheus stored. The alert manager watches the same data and notifies people through chat, email, or paging when rules are met.
Popular exporters
| Exporter | What it is known for |
|---|---|
| Kafka Exporter (a widely used community project) | Simple, popular, reports lag per group, topic, and partition, plus topic details. Often bundled in Kubernetes operators for Kafka. |
| KMinion (open source) | Reports lag and also other cluster details, designed to be light on the cluster. |
| Kafka Lag Exporter (open source) | Known for estimating time lag by remembering past offsets and interpolating. Check its current maintenance status before choosing. |
| OpenTelemetry collector Kafka metrics receiver | Lets teams that already use OpenTelemetry collect group lag with the same pipeline as other metrics. |
What a good Grafana lag dashboard shows
- Total lag per consumer group over time.
- Maximum partition lag per group, to catch single stuck partitions.
- Time lag or an estimate, if the exporter provides it.
- Produce rate and consume rate side by side, to see whether consumers are catching up.
- Number of members per group, and how it changes.
- Group state and rebalance frequency, if available.
The number of unique metric series can grow fast: groups times topics times partitions. On a cluster with thousands of partitions and many groups, this can strain Prometheus. Filter out unimportant groups, and keep per-partition detail only where you need it.
Real-world picture: a Kubernetes team
A team runs Kafka on Kubernetes with an operator that deploys an exporter beside the brokers. Prometheus finds it automatically, and a shared Grafana board shows every group. The team adds one alert: any group whose total lag has risen for fifteen minutes straight. It catches most real problems and stays quiet on ordinary spikes.
Open-Source Web Interfaces
When you want to click around and see groups, topics, and messages on one screen.
Several open-source web consoles let people browse Kafka in a browser. They show topics, partitions, consumer groups, members, and lag. Some also let you view messages, manage topics, and see schemas. They are friendly to developers who do not want to use the command line.
AKHQ
A web console for browsing topics and groups, viewing messages, and managing configuration, with access control options.
UI for Apache Kafka
A popular open-source console that shows clusters, topics, groups, and lag, with role-based access options.
Kafdrop
A lightweight viewer for topics, partitions, and consumer groups with lag. Simple to run.
Redpanda Console
A console that works with Kafka-compatible clusters, showing groups, lag, and messages.
Where they shine
- Discovery. New engineers can see what exists.
- Debugging. Jump from a lagging partition straight to the message at the committed offset.
- Team self-service. Developers check their own groups without asking the platform team.
Where they are weaker
- Limited history. Most show the current state and do not store long trends.
- Limited alerting. They are viewers, so pair them with an alerting system.
- Data exposure. Because they can display message content, they need careful access control.
Features and maintenance vary across these projects and change over time. Try two of them against a test cluster and check their current documentation before choosing one for your team.
A web console is like a building’s reception desk with a large screen. Visitors can see who is in which office, and which rooms are busy. But it does not keep a diary of last month’s visitors, nor does it ring the manager when a room floods. For those jobs you need other equipment.
Commercial Platforms
Paid products bundle lag monitoring with much more.
Many vendors offer platforms that include consumer lag as one feature among many. They typically add long-term history, alert rules, team access control, support, and connections to the rest of your monitoring.
| Kind of product | Examples | What lag monitoring looks like |
|---|---|---|
| Kafka vendor platforms | Confluent Control Center and Confluent Cloud tools, Lenses, Conduktor | Lag views per group and topic, alerts, and links to schemas, connectors, and streams in the same product |
| General observability suites | Datadog, New Relic, Dynatrace, Splunk Observability, Grafana Cloud | Kafka integrations that collect lag alongside application and infrastructure metrics, so you can see them together |
| Managed Kafka provider consoles | Consoles from cloud Kafka services | Lag charts and alerts included with the service |
Reasons to choose a platform
Why teams pay
- Fast to start, little to build or maintain
- History, alerts, and team permissions included
- One place for lag, application traces, and infrastructure
- Vendor support during incidents
Things to weigh
- Recurring cost that grows with data volume or host count
- Less control over how lag is computed
- Some features tie you to one vendor
- Data about your cluster leaves your network
Product names, features, and pricing change often. Treat the examples above as starting points for your shortlist, and check each vendor’s current documentation for how they define and collect lag, including whether they offer time lag.
Real-world picture: one screen for the whole path
A company already uses a general observability suite for its web services. Adding the Kafka integration lets an engineer look at a slow checkout page, follow the trace to the queue consumer, and see the lag graph on the same screen. The value is not the lag number itself, but seeing it next to everything else.
Cloud Provider Metrics: MSK and Confluent Cloud
If someone else runs your Kafka, they may also give you the lag numbers.
Amazon MSK
Amazon Managed Streaming for Apache Kafka publishes metrics into Amazon CloudWatch, AWS’s monitoring service. These include consumer lag measures: the offset lag of a group on a partition, the total offset lag across a group, the largest lag, and estimates of time lag. Which metrics you receive depends on the monitoring level you turn on for the cluster, and more detailed levels cost more, so check the current MSK documentation for what each level includes. You can draw these in CloudWatch dashboards and set CloudWatch alarms that notify through Amazon SNS. Many MSK users also run an open-source exporter or a commercial tool for per-partition detail.
Confluent Cloud
Confluent Cloud exposes a metrics service that includes a consumer lag measure per group and topic, which you can pull into your own monitoring tools or view in its console. It is designed so that a team can watch lag without operating any monitoring software of its own.
flowchart LR
MSK["Managed Kafka service"] --> CW["Provider metrics: lag, throughput"]
CW --> DB["Provider dashboards"]
CW --> AL["Provider alarms"]
AL --> N["Notification service"]
CW -. "export" .-> EXT["Your own monitoring stack"]
The service publishes lag numbers to its own metric system. From there, you can use built-in dashboards and alarms, or export the numbers to the monitoring stack your company already uses.
| Question | Provider metrics | Self-run exporter |
|---|---|---|
| Setup effort | Very low | Moderate |
| Control over calculation | Low | High |
| Per-partition detail | Depends on plan and level | Full |
| Cost | Included or per-metric fees | Compute you run plus operations |
| Portability across clouds | Tied to the provider | Same everywhere |
Provider lag metrics are usually built on the same committed offsets as other tools, so they share the same blind spots: applications that store progress outside Kafka are invisible, and groups with no recent commits may not appear. Also check how often the values refresh.
Tools Side by Side
A single table to compare the families on what matters most.
| Family | History and graphs | Alerts | Time lag | Works if consumer is dead | Setup effort |
|---|---|---|---|---|---|
| Command line tool | No | No | No | Yes | None |
| Client metrics | With a metrics system | With a metrics system | Rarely | No | Per application |
| Burrow | Limited | Yes, via notifiers | No | Yes | Moderate |
| Prometheus exporters and Grafana | Yes | Yes | Some exporters | Yes | Moderate |
| Open-source web consoles | Limited | Usually no | Rarely | Yes | Low |
| Commercial platforms | Yes | Yes | Often | Yes | Low |
| Cloud provider metrics | Yes | Yes | Some | Yes | Very low |
No single row wins on every column. The command line tool is the quickest but the thinnest. Exporters with Grafana give the most control at moderate effort. Commercial platforms and provider metrics reduce work but increase cost or reduce control.
Choosing Your Tools: A Decision Guide
Answer these questions in order and the shortlist writes itself.
Who runs your Kafka?
If a cloud provider does, start with their metrics. If you do, plan on running an exporter or a platform.
Do you already have a monitoring stack?
If you use Prometheus and Grafana, or a general observability suite, choose the tool that plugs into it. Do not build a second island.
How many consumer groups do you have?
A few dozen can be handled with hand-set alert rules. Hundreds or thousands benefit from trend-based judgement, as offered by Burrow-style tools.
Do you need time lag?
If business owners talk in minutes, pick a tool that provides time lag or a good estimate.
Who needs to look?
If many developers need self-service views, add a web console with proper access control.
What is the budget for cost and for people?
Open source costs engineering time. Platforms cost money. Decide which is scarcer for you.
| Situation | Suggested combination |
|---|---|
| Small team, self-run Kafka, already on Prometheus | Exporter, Grafana dashboards and alerts, command line tool for incidents |
| Large company, hundreds of groups, many teams | Exporter or Burrow-style evaluator for coverage, client metrics for detail, web console for self-service |
| Managed Kafka on a cloud provider | Provider metrics and alarms, plus an exporter if you need per-partition detail |
| Already using a general observability suite | Its Kafka integration, combined with client metrics from applications |
| Regulated environment with strict data rules | Self-hosted tools inside your network, with restricted access to consoles that can show message content |
| Application keeps offsets outside Kafka | Custom metric inside the application, since standard tools cannot see it |
Alerting: Turning Numbers into Action
A graph nobody looks at does not protect anyone. Alerts do.
Rules that work
Lag above a level for a time
For example, total lag above a level for fifteen minutes. This ignores brief spikes.
Lag growing steadily
Lag that has increased across the last several samples means consumers cannot keep up.
Time lag above a promise
Tie the rule to a business target, such as “payments are never more than two minutes old”.
Offsets not moving
Committed offset unchanged for a period while the end offset moves. Catches stuck consumers.
Group empty or dead
A group that normally has members but now has none.
Data disappears
No lag metrics at all for a group, or from the exporter itself. Silence is a symptom.
Tiers of urgency
| Level | Example | Who is told |
|---|---|---|
| Information | Lag is above normal for ten minutes on a low-priority group | A team channel |
| Warning | Lag rising for thirty minutes on an important group | The owning team, in working hours |
| Critical | Time lag above the business promise, or a payments group stalled | The on-call person, at any hour |
| Emergency | Lag approaching the retention limit | On-call and a manager |
Alert fatigue is real. If alerts fire daily for harmless spikes, people will start ignoring them, including the one that matters. Prefer fewer, sustained, well-owned alerts over many twitchy ones.
Every alert needs an owner and a next step
An alert should say which group is affected, which topic and partition are worst, how long the condition has lasted, and where to find the runbook. Otherwise the person on call spends the first ten minutes just working out what the alert means.
A smoke alarm that beeps every time you make toast is soon taken off the ceiling. A good alarm is tuned to ring for real fire and to tell you which room it is in.
Designing Dashboards That Answer Questions
A good dashboard is arranged from the general to the specific.
Top row: is anything on fire?
A list of groups with the worst status or highest time lag, sorted so the worst is first.
Second row: the picked group
Total lag, maximum partition lag, and time lag over the last day for one selected group.
Third row: cause hunting
Produce rate and consume rate together, plus member count and rebalance events.
Fourth row: partition detail
A per-partition view to find the one stuck or skewed partition.
Bottom row: context
Consumer application health, such as CPU, memory, and error counts, so lag can be tied to its cause.
Design habits
- Show two lines together. Produce rate and consume rate on one graph make the story obvious: when the consume line is below the produce line, lag grows.
- Use a log-friendly scale when spikes are huge, so normal behavior is still visible.
- Mark deployments on the graph. Many lag changes follow a release.
- Label with owners. Add the team name so the right person can be contacted.
- Keep one overview per environment, so testing noise does not hide production trouble.
Real-world picture: the two-line trick
A team’s lag graph rises for an hour. Someone assumes consumers are slow. Then they add the producer rate to the same panel and see that the producers doubled their output at the same moment. The consumers were fine; they simply needed more capacity for a doubled load. The fix is scaling, not debugging.
Lag and Autoscaling
Lag is one of the best signals for adding or removing consumer instances.
CPU use is a poor guide for consumers that wait on databases or other services. Lag, on the other hand, measures exactly what matters: how much work is waiting. Autoscaling tools can watch lag and change the number of consumer instances to match. For example, KEDA, an event-driven autoscaler for Kubernetes, has a Kafka scaler that uses group lag to decide how many pods to run.
flowchart LR
LAG["Lag per group"] --> SC["Autoscaler"]
SC -->|"lag high: more replicas"| DEP["Consumer deployment"]
SC -->|"lag low: fewer replicas"| DEP
DEP --> K["Kafka partitions"]
K --> LAG
Lag flows to the autoscaler. When lag is high, it adds instances. When lag is low, it removes them. The instances read from the partitions, which changes the lag, and the loop continues.
Rules that keep it safe
- Never exceed the partition count. Extra instances beyond the number of partitions do nothing. Set a maximum equal to the partition count.
- Scale gently. Each change triggers a rebalance that pauses reading. Frequent scaling can make lag worse.
- Use a cooldown. Wait after a change before deciding again.
- Set a sensible minimum. Keep enough instances for availability, even with zero lag.
- Watch downstream limits. More consumers can overload a database that the consumers write to.
Autoscaling on lag cannot fix a stuck message or a hot partition. It would simply add idle instances. Always pair autoscaling with alerts on stalls and on skew.
Capacity Math: How Long Until the Backlog Clears?
Simple arithmetic turns a scary lag number into a plan.
When lag is falling, you can estimate how long the backlog will take to clear. Use three numbers: the current lag, the speed at which consumers process messages, and the speed at which producers add them.
| Lag | Consume rate | Produce rate | Net catch-up | Time to clear |
|---|---|---|---|---|
| 600,000 | 5,000 per second | 3,000 per second | 2,000 per second | 300 seconds, about 5 minutes |
| 600,000 | 3,200 per second | 3,000 per second | 200 per second | 3,000 seconds, about 50 minutes |
| 600,000 | 3,000 per second | 3,000 per second | 0 | Never; lag stays where it is |
| 600,000 | 2,800 per second | 3,000 per second | Negative | Never; lag keeps growing |
The table shows why the same lag can be harmless or serious. When consumers are barely faster than producers, catching up takes a very long time. When they are slower, the backlog grows forever. The lesson: judge lag by the difference in speeds, not by the lag itself.
You are bailing water from a leaking boat. What matters is not how much water is in the boat but whether you bail faster than the leak. If you bail at the same speed as the leak, the level never falls, however hard you work.
Planning headroom
Aim to run consumers at a speed comfortably above the busiest expected production speed, so that backlogs from short outages clear quickly. A common rule of thumb is to have enough capacity to clear a typical outage’s backlog within a small fraction of the time available. Set the target based on how long your business can tolerate delay.
The Retention Cliff: When Lag Becomes Data Loss
This is the most serious consequence of ignoring lag.
Kafka does not keep messages forever by default. A topic has a retention period, such as seven days, after which old messages are deleted. If a consumer falls so far behind that its next message has already been deleted, that message is gone. The consumer’s committed offset now points to nothing.
flowchart LR
A["Oldest message still stored: offset 5000"] --> B["Consumer committed offset: 3000"]
B --> C{"Committed offset below oldest stored?"}
C -->|"yes"| D["Offset out of range"]
D --> E["Reset policy decides: jump to earliest or latest"]
E --> F["Messages 3000 to 4999 are lost to this consumer"]
The diagram shows the moment of loss. The oldest stored message is at offset 5,000, but the group still stands at 3,000. That offset is out of range. The consumer’s reset policy then decides what to do, usually jumping to either the earliest message still available or the newest. Either way, messages 3,000 to 4,999 are never processed by this group.
If the reset policy jumps to the latest offset, the consumer skips the entire backlog silently. If it jumps to the earliest, it may reprocess a huge amount of old data. Neither is a happy ending. Prevent the situation with alerts.
How to guard against it
- Alert on time lag against retention. Warn when the oldest unprocessed message is older than, say, half of the topic’s retention.
- Use client “lead” metrics where available. A shrinking lead warns that the consumer is approaching the deletion edge.
- Choose retention with recovery time in mind. Retention should comfortably exceed the longest outage plus the time to catch up.
- Practice recovery. Know in advance who can extend retention temporarily during an incident.
Real-world picture: the long weekend
A consumer breaks on Friday evening. Retention is three days. Nobody notices until Tuesday morning. By then, the earliest messages have been deleted, and the team must explain missing data to the business. A single alert on time lag above one day would have paged someone on Saturday, with days to spare.
Advantages, Disadvantages, and Trade-offs
Every monitoring approach gives something and costs something.
External monitoring (reading offsets from the cluster)
Pros
- Sees every group, including ones nobody instrumented
- Works when consumers are dead
- One setup for the whole cluster
- No changes to applications
Cons
- Only as fresh as commits and polling
- Blind to offsets stored outside Kafka
- Adds polling load to brokers
- Little insight into what the application is doing
Client-side metrics
Pros
- Fresh, detailed, and tied to the application
- No load on brokers
- Include lead and speed metrics
Cons
- Disappear when the consumer fails
- Need setup in every application
- Cannot see groups without running members
Trade-offs at a glance
| Trade-off | Leaning external | Leaning client-side |
|---|---|---|
| Coverage | Whole cluster | Only instrumented apps |
| Freshness | Seconds to a minute old | Almost instant |
| Failure behavior | Keeps working | Goes silent |
| Effort | Once per cluster | Per application |
| Cluster load | Small polling cost | None |
A broader trade-off is between alert sensitivity and alert noise. Sensitive rules catch problems early but cry wolf. Calm rules stay quiet but may be late. The right balance depends on how costly delay is for each pipeline.
Security Considerations
Monitoring tools can see a lot. Give them only what they need.
Describe, not read
Lag tools need permission to describe topics and groups. They do not need permission to read message contents. Grant the smaller set.
A dedicated account
Give the monitoring tool its own identity, so its actions are visible in audit logs and can be revoked separately.
Encrypted connections
Use TLS between tools and brokers, and between dashboards and their users.
Sign-in for consoles
Put web consoles behind single sign-on and roles, since they may show message data.
Data exposure through labels
Lag metrics carry names of groups, topics, and sometimes client hosts. These names can reveal business details, such as “salary-changes” or “customer-deletions”. Limit who can see dashboards that include every topic name, and remember that metrics stored in third-party services leave your network.
The ability to reset a group’s offsets is powerful. A person or tool with that permission can make a consumer skip or replay data. Keep this permission away from monitoring accounts and dashboards that are only meant to look.
Real-world picture: a read-only viewer
A platform team creates a read-only role for their lag exporter with permission to describe groups and topics, and nothing else. When a security review asks what the tool can do, the answer is short and comfortable: it can see counts and names, but it cannot read messages or change positions.
Performance and Scalability of Monitoring
Watching Kafka should not slow Kafka down.
Cost of polling
Each poll asks the cluster for the committed offsets of every group and the end offsets of every partition. With a handful of groups this is trivial. With thousands of groups and tens of thousands of partitions, polling every few seconds can become noticeable. Choose a polling interval that matches your needs. Fifteen to sixty seconds is plenty for most alerting.
Cost of storing metrics
Each combination of group, topic, and partition becomes a separate series in a metrics database. The count multiplies quickly. Cost and speed problems in the metrics system often start here.
| Problem | Cause | Remedy |
|---|---|---|
| Slow dashboards | Too many series | Aggregate by group and topic; keep per-partition only for key groups |
| High cluster load from polling | Short interval, many groups | Lengthen the interval; skip idle or test groups |
| Large bills at a metrics vendor | Per-series pricing | Drop unused labels; filter groups |
| Stale data | Exporter overloaded | Split exporters by cluster or by topic prefix |
A doctor who takes a patient’s pulse every second will only annoy the patient and fill the chart with noise. Once every minute, or every few minutes, gives the same picture without the fuss.
Tidy up old groups
Groups that are no longer used still appear in listings. Their committed offsets become stale, their lag grows forever, and they can trigger false alarms. Delete abandoned groups after checking with their owners, and filter known temporary groups, such as those created by ad-hoc console sessions, out of alerts.
Reliability: Monitoring the Monitor
If the lag tool is down, you are blind exactly when you need eyes.
Failure modes to plan for
The exporter crashes
Graphs go flat or empty. Add an alert for missing data so silence is treated as a problem.
The exporter loses its connection to Kafka
Some tools keep showing the last value. Check that the tool reports its own last successful poll time.
The metrics store is down
No graphs and no alerts. Run it with redundancy, and send a heartbeat alert from a separate system.
The alert path is broken
Alerts fire, but the notification service fails. Test the path end to end regularly.
Practical protections
- Run two copies of critical monitoring services, or use a platform that provides redundancy.
- Add a “dead man’s switch.” A constant alert that is always firing, whose absence tells you the monitoring pipeline itself has broken.
- Keep a manual fallback. Make sure at least a few engineers know how to use the command line tool.
- Watch the freshness of the lag data. A lag reading that is ten minutes old is dangerous when treated as current.
Test your alerting the way you test a fire drill. Create a small, harmless lag on a test group and confirm that the alert arrives, in the right channel, with useful information.
Deployment and Cloud
Where and how you run the tools matters as much as which ones you pick.
Running exporters and evaluators
These are small services, and they run happily as containers. On Kubernetes, operators such as Strimzi can deploy an exporter alongside the brokers. Otherwise, you can use a simple deployment with a service definition and a scrape rule. Keep configuration in source control, including the list of clusters and the groups to include or exclude.
Placement and network
- Run monitoring close to the cluster to reduce delay and avoid crossing costly network boundaries.
- Use one exporter per cluster, and label metrics with the cluster name, so multi-cluster dashboards stay clear.
- For multi-region setups, remember that offsets and group names may differ between clusters. Monitor each cluster on its own.
Managed and cloud choices
| Environment | Typical choice | Extra note |
|---|---|---|
| Self-managed Kafka on servers | Exporter, Prometheus, Grafana | You operate all parts |
| Kafka on Kubernetes with an operator | Operator-provided exporter and dashboards | Quick to start, easy to standardize |
| Amazon MSK | CloudWatch metrics and alarms, optionally an exporter | Check which monitoring level includes lag |
| Confluent Cloud | Provider metrics, console views | Export to your own tools if desired |
| Hybrid or multi-cloud | A neutral tool that works everywhere | Avoids relearning tools per cloud |
Create a read-only monitoring identity
With permission to describe topics and groups only.
Deploy the collector
Exporter or evaluator, with a sensible polling interval and group filters.
Build shared dashboards
Overview, group, and partition views, with owner labels.
Write alerts with runbooks
Sustained lag, stalls, time lag against promises, and missing data.
Rehearse
Cause a harmless lag on a test group and confirm the whole chain works.
Design Patterns and Anti-Patterns
The same good habits and the same bad habits appear again and again.
Patterns that work
Two layers of monitoring
External offset monitoring for coverage, plus client metrics for detail and speed.
Alert on time, not just counts
Tie the main alert to a business promise, such as data no older than two minutes.
Two-line panels
Show produce and consume rates beside lag, so cause and effect are visible.
Owner labels
Every group has a team and a runbook link attached to its alerts.
Retention guard
An alert when the oldest unprocessed message is a set fraction of retention.
Heartbeat check
A constant test signal that proves the monitoring pipeline itself is alive.
Anti-patterns to avoid
What people do
They alert when lag exceeds one fixed number, for every group.
Why it hurts
A number that suits one group is far too high or too low for another. The result is false alarms on busy groups and silence on quiet ones.
Better approach
Use sustained-trend rules, time lag, or per-group thresholds based on each group’s normal behavior.
What people do
They rely only on client metrics.
Why it hurts
When a consumer dies, its metrics vanish and the graph shows nothing, which is easy to misread as “no lag”.
Better approach
Add external monitoring from committed offsets, and alert on missing data.
What people do
They look only at total lag and never at partitions.
Why it hurts
One stuck partition can hide behind a modest total, and adding consumers will not fix it.
Better approach
Alert on maximum partition lag as well, and keep a per-partition view.
What people do
They reset a group’s offsets to the latest position to make a lag alert go away.
Why it hurts
The alert stops, but every message in the skipped backlog is never processed. The problem is hidden and data is lost.
Better approach
Find the cause, scale or fix the consumer, and skip data only as a deliberate, approved decision with the business informed.
What people do
They autoscale consumers on lag with no upper limit.
Why it hurts
Instances beyond the partition count sit idle, cost money, and cause repeated rebalances.
Better approach
Cap replicas at the partition count, add cooldowns, and check downstream limits.
What people do
They never clean up abandoned consumer groups.
Why it hurts
Old groups show ever-growing lag, add noise, inflate metric costs, and train people to ignore red numbers.
Better approach
Review group lists regularly, confirm owners, and delete groups that are no longer used.
Best Practices and Common Mistakes
Small habits that prevent most lag surprises.
Habits that pay off
Decide the promise first
For every important consumer group, write down how fresh its data must be. Alerts follow from that promise.
Cover the whole cluster externally
Make sure every group is watched, including ones created by teams you have not met.
Track time lag for critical groups
People understand minutes better than message counts.
Watch group state and rebalances
Frequent rebalances often explain a saw-tooth lag pattern.
Commit regularly and sensibly
Very rare commits make lag look worse than it is. Very frequent commits add load. Choose a balance.
Guard retention
Alert well before lag reaches the deletion edge.
Write a runbook
Steps for the first ten minutes of a lag incident: check partitions, check members, check producer rate, check downstream systems.
Common mistakes
Reading zero lag as healthy
Zero can also mean producers are silent or the wrong topic is being watched.
Ignoring offsets stored elsewhere
Applications that keep progress outside Kafka are invisible to standard tools.
Adding consumers past the partition count
Extra members stay idle and add rebalances.
Careless offset resets
Resetting a group’s position can skip data or replay huge volumes. Do it only with a clear plan, and with the group stopped.
Comparing groups with different jobs
A batch loader and a real-time service have very different normal lag.
Forgetting to monitor the monitor
A silent exporter looks like a healthy cluster.
Before you reset offsets, first stop the group’s consumers, note the current positions in your incident record, and decide whether to reset to a time, to the start, or to the end. Having the old positions written down makes a mistake reversible.
A Guided Walkthrough: A Lag Incident from Alert to Fix
Follow an engineer through a realistic evening.
Brightcart, an online store, has a “billing-service” consumer group reading the “orders” topic, which has six partitions. Billing promises that every order will be billed within two minutes. The team uses an exporter with Prometheus and Grafana, plus the command line tool for quick looks.
flowchart TD
A["Alert: billing time lag above 5 minutes"] --> B["Open overview dashboard"]
B --> C{"All partitions or one?"}
C -->|"all rising"| D["Compare produce rate and consume rate"]
C -->|"one partition"| E["Check the member owning that partition"]
D --> F{"Producers doubled?"}
F -->|"yes"| G["Scale consumers up to partition count"]
F -->|"no"| H["Check downstream database latency"]
E --> I{"Consumer stuck on one message?"}
I -->|"yes"| J["Inspect message, fix or route to error topic"]
I -->|"no"| K["Restart or replace the unhealthy instance"]
The alert fires
The estimated time lag for the billing group is over five minutes, and has been for ten minutes. The alert links to the dashboard and runbook.
All partitions or one?
The per-partition panel shows five partitions with tiny lag and partition 3 with huge lag. This is not a general slowdown.
Who owns partition 3?
The command line tool shows that partition 3 belongs to one billing instance whose committed offset has not moved for twelve minutes.
Why is it stuck?
The application log shows the same message failing repeatedly with a data error. The message has an amount in a format the billing code cannot read.
Unblock the partition
The team deploys a fix that sends unreadable messages to an error topic and continues. The consumer moves on, and lag on partition 3 starts to fall.
Confirm recovery
Using the capacity formula, they estimate ten minutes to clear the backlog. They watch time lag drop below the two-minute promise.
Follow-up
They ask the producer team to fix the bad format, review the error topic, and add an alert on stalled offsets so this is caught in five minutes next time, not ten.
| Clue | What it told them |
|---|---|
| Five partitions fine, one huge | A single-partition problem, so more consumers would not help |
| Committed offset unchanged for twelve minutes | A stuck consumer, not a slow one |
| Same message failing in the log | A poison message, not a resource issue |
| Time lag versus promise | Clear business impact, easy to explain to others |
Myths and Misconceptions
Some wrong beliefs travel from team to team. Here are the corrections.
“Zero lag means everything is fine.”
It may also mean producers stopped, or that you are watching the wrong topic. Compare with the produce rate.
“Any lag is bad.”
Small, steady lag is normal. The trend and the business promise decide what is acceptable.
“More consumers always fix lag.”
Only up to the number of partitions, and not for stuck messages or hot partitions.
“Lag is measured in bytes or time.”
The basic measure is a count of messages. Time and bytes are derived or estimated.
“Kafka alerts me when consumers fall behind.”
Kafka stores offsets but does not judge them or send alerts. You need tools for that.
“The lag number is exact and instant.”
It depends on commit timing and polling, so it is always slightly old and slightly rough.
Think of lag as a fuel gauge with a small delay, not as a speedometer. It tells you roughly where you stand and which way you are heading, and that is exactly what you need it for.
Real-World and Industry Patterns
The same needs appear in company after company.
Where Burrow came from
LinkedIn, the birthplace of Kafka, ran so many consumer groups that hand-tuned thresholds were impossible. It built Burrow to judge groups by their behavior over time, then shared it with the community. The lesson many teams took away is that at large scale, trend-based rules beat fixed numbers.
Kafka on Kubernetes with operators
Teams that run Kafka through operators often get a lag exporter as part of the package, along with ready-made dashboards. This makes basic lag graphs a default, not a project, and lets platform teams spend their effort on alert design.
Managed Kafka on the cloud
Teams using managed Kafka services on major clouds start with the provider’s lag metrics and alarms. As their needs grow, they add exporters for per-partition detail and connect lag data to the same dashboards as their applications.
Event-driven autoscaling
Teams running consumers on Kubernetes use lag as the trigger for scaling, because it measures waiting work better than CPU. They cap replicas at the partition count and pair scaling with alerts for stalls.
Regulated services
Financial and healthcare teams tie alerts to service promises and to retention limits, because data that ages out before it is processed can be a compliance problem. They also lock down consoles that can display message contents.
Group State and Rebalances: The Other Half of the Story
Lag tells you how far behind a group is. Group state tells you whether the group is even able to work.
The states of a consumer group
A consumer group is always in some state, and monitoring tools can read it. Knowing the main states helps you interpret a lag graph.
| State | Plain meaning | What it means for lag |
|---|---|---|
| Stable | Members are present and every partition is assigned | Normal. Lag reflects real speed |
| Preparing or completing a rebalance | Members are agreeing on who reads which partition | Reading may pause, so lag rises for a moment |
| Empty | The group exists but has no active members | Nobody is reading; lag will grow if producers are active |
| Dead | The group has no members and its stored data is being removed | Nothing to monitor; usually a retired group |
A restaurant kitchen can be busy, closed, or in the middle of a shift change. Counting tickets on the rail (lag) means little unless you also know which of those three situations you are looking at. A long rail during a shift change is normal. A long rail with an empty kitchen is an emergency.
What triggers a rebalance
- A member joins or leaves, for example during a deployment or when a pod restarts.
- A member is judged dead. If a consumer stops sending its heartbeat signal within the allowed time, the group removes it.
- A consumer takes too long between reads. If processing one batch takes longer than the allowed time between polls, the group assumes the member is stuck and removes it, even if it is merely slow.
- The set of partitions changes, for example when a topic gets more partitions.
flowchart LR
S["Slow processing of one batch"] --> T["Time between reads exceeds the limit"]
T --> R["Group removes the member"]
R --> B["Rebalance: reading pauses"]
B --> L["Lag rises"]
L --> J["Member rejoins and processes again"]
J --> S
The diagram shows a vicious circle. A batch is slow, so the consumer misses its deadline. The group removes it and rebalances, which pauses reading and raises lag. The member returns, meets a batch that is still slow, and the cycle starts again. A lag graph with repeating bumps, combined with a high rebalance count, points to this pattern.
Ways to reduce rebalance pain
- Process smaller batches or lengthen the allowed time between reads.
- Use cooperative rebalancing, which moves only the partitions that need to move, so most members keep reading.
- Use static membership for members that restart quickly, so a short restart does not trigger a full reshuffle.
- Roll deployments gently, one instance at a time.
Put group state and the rebalance count on the same dashboard as lag. Many puzzling lag patterns stop being puzzling once you see the rebalances beside them.
Lag for Kafka Streams and Kafka Connect
Higher-level Kafka tools are consumers underneath, so the same lag rules apply.
Kafka Streams
A Kafka Streams application reads its input topics through ordinary consumers. Its consumer group name is the application ID, so a lag tool sees it like any other group. Two extra ideas matter here. First, stateful steps such as tables can spend a long time restoring their state after a restart or a move, during which processing is paused and lag rises. Second, Streams applications expose their own metrics about processing speed and lag, whose names depend on the version, so check the documentation for the version you run.
Kafka Connect
A sink connector, which copies data out of Kafka, also uses an ordinary consumer group, named after the connector, usually with a prefix such as “connect-“. Its lag tells you how far behind the destination is. This is very useful: if a connector loading data into storage is thirty minutes behind, analysts are looking at data that is thirty minutes old. A source connector works the other way round, reading from outside systems and writing into Kafka, so its health is measured differently, by its own throughput and status.
Group is the application ID
Find it in lag tools by the name you gave the application.
Restores raise lag
After a restart, tables may be rebuilt before new work resumes.
Sink lag is destination delay
It shows how stale the data in the target system is.
Tasks map to partitions
Parallelism is limited by the topic’s partitions, as with any consumer.
A delivery driver and a warehouse worker both take parcels off the same conveyor. Whether they are called drivers or workers, the question for the manager is the same: how many parcels are still waiting on the belt?
Real-world picture: a stale report
Analysts complain that today’s sales report looks too low. The data team checks the lag of the connector that loads orders into the warehouse and finds it is forty minutes behind because a task keeps restarting. The fix is in the connector, not in the report, and the lag graph led them straight to it.
Frequently Asked Questions
Summary and Key Takeaways
Consumer lag is the gap between where producers have written and where a consumer group has confirmed its work. It is the most useful early warning for freshness problems, stuck consumers, and looming data loss. Monitoring tools all read the same few sources: end offsets, committed offsets, group state, and client metrics. They differ in how they store history, raise alerts, estimate time lag, and fit into your environment. The right choice is usually a combination, backed by clear promises and well-owned alerts.
Key Takeaways
- Lag is a subtraction: log end offset minus committed offset, per partition and per group.
- Trend matters more than a single number. Steady small lag is healthy, and steadily rising lag is the warning.
- Watch both messages and time. Time lag ties the number to a business promise.
- Use two layers: external monitoring for coverage and safety, and client metrics for detail and speed.
- Know your tool families: command line, client metrics, Burrow-style evaluators, exporters with Grafana, web consoles, commercial platforms, and cloud provider metrics.
- Guard against the retention cliff: alert long before lag approaches the retention limit, or messages will be lost.
- Alert with care: sustained, well-owned alerts with runbooks, a check for missing data, and a heartbeat that watches the monitor itself.