Kafka runs quietly in the background. That is exactly why you need numbers to tell you how it is doing.

Kafka in one paragraph

Apache Kafka is a system for moving streams of messages between applications. Applications that write messages are called producers. Applications that read messages are called consumers. In between sit the Kafka servers, called brokers, which store messages safely on disk in channels called topics. Each topic is split into smaller pieces called partitions, and each partition is copied to several brokers, so that data survives when a machine fails.

What goes wrong without visibility

The goal of monitoring

Good monitoring answers three questions. First, is the system working right now? Second, is it about to have a problem? Third, when something breaks, where is the cause? The metrics in this guide were chosen because they answer these questions for the three main parts of Kafka.

A metric is simply a number that is measured again and again. The way that number behaves decides how you read it.

Metrics are measurements over time

A single measurement, such as “the broker handled 5,000 messages this second”, tells you little. A series of measurements over minutes, hours and days tells a story: whether traffic is rising, whether a spike is normal for Monday mornings, whether a dip appeared right after a deployment. Always look at metrics as lines on a chart, not as isolated numbers.

The common kinds of metrics

Averages hide pain: why percentiles matter

Suppose ninety-nine requests take one millisecond and one request takes five seconds. The average looks very healthy, yet one user had a terrible experience. That is why timing metrics are often reported as percentiles. The 99th percentile means “ninety-nine out of a hundred requests were faster than this.” Watching it shows the slow tail that averages hide. Whenever you can, look at the maximum and the high percentiles next to the average.

Kafka is written in Java, and it publishes its numbers through a standard Java mechanism. Everything else is built on top of that.

JMX: the built-in window

Java programs can expose internal measurements through a technology called JMX, which stands for Java Management Extensions. Brokers, producers and consumers all register their metrics as named entries, and any tool that can talk JMX can read them. The names follow a pattern that starts with a group, such as the server, the network or the log, followed by the type and the name of the metric. You will see names like this in documentation, and they tell you where in Kafka the number lives.

From JMX to dashboards

Reading JMX by hand is possible but painful. In practice, teams place a small helper next to each Kafka process that reads JMX and republishes the numbers in a friendlier form. A monitoring system then collects those numbers every few seconds, stores them, draws charts and sends alerts. Many products can do this job, including open-source stacks and commercial platforms. The helper and the storage differ from company to company, but the metric names and meanings stay the same.

flowchart LR
    B[Broker JMX] --> X[Metrics exporter]
    P[Producer JMX] --> X
    C[Consumer JMX] --> X
    X --> S[Metrics storage]
    S --> D[Dashboards]
    S --> A[Alerts]
    A --> N[People on call]
        

Two important sources

Other sources worth combining

Kafka’s own metrics do not tell the whole story. You should also collect numbers from the machines and the Java runtime that host Kafka: processor use, memory, disk space, disk speed, network traffic, open file counts and garbage collection pauses. When a Kafka metric looks bad, the cause very often shows up first in one of these host-level numbers.

With hundreds of metrics available, a simple mental map keeps you from getting lost.

Engineers who run large systems often use four questions as a map, sometimes called the four golden signals. They apply nicely to Kafka.

How the map applies to each part of Kafka

Knowing the journey of a message tells you where each group of metrics measures.

flowchart LR
    subgraph PA[Producer application]
        PM[Producer metrics]
    end
    subgraph BR[Kafka cluster]
        BM[Broker metrics]
        L[(Partition logs)]
    end
    subgraph CA[Consumer application]
        CM[Consumer metrics]
    end
    PA -->|Send batches| BR
    BR -->|Replicate to followers| BR
    BR -->|Deliver fetches| CA
        

Because the three views describe the same journey from different positions, a problem in one place often leaves fingerprints in the others. A slow disk on a broker raises broker request times, which raises producer request latency, which may fill the producer’s buffer, and which may leave consumers waiting for fresh data. Keeping this chain in mind is the key to reading metrics well, and we will practise it in later chapters.

Brokers are the heart of Kafka. Their metrics fall into a few families, and each family answers a different question.

Why watch every broker separately

A cluster average can hide a struggling machine. If one broker out of six is overloaded, the average across the cluster still looks fine, yet all the partitions led by that broker suffer. Always chart broker metrics per broker, and compare brokers against each other. A broker that behaves very differently from its neighbours is worth investigating.

Traffic metrics tell you how busy each broker is and how that changes over time.

How to read them

Per-topic traffic

The same traffic metrics are also available per topic. When the total traffic grows, per-topic charts tell you which topic caused it. This is very useful for finding a noisy application, planning capacity and sharing costs between teams.

These are the most important broker metrics of all, because they tell you whether your data is safe and reachable.

A quick reminder about replication

Each partition has one leader broker, which handles reads and writes, and several follower brokers, which copy the data. The set of copies that are fully caught up with the leader is called the in-sync replicas, or ISR. If a follower falls too far behind, it is removed from the ISR. When the ISR gets smaller, your data has fewer safe copies, and you are one failure closer to losing availability or data.

flowchart TD
    A[Follower falls behind] --> B[Removed from in-sync set]
    B --> C[UnderReplicatedPartitions rises]
    C --> D{Fewer copies than minimum needed?}
    D -- Yes --> E[Writes with strict acknowledgements are rejected]
    D -- No --> F[Writes continue with reduced safety]
    C --> G{Leader also fails?}
    G -- Yes --> H[Risk of offline partition or data loss]
    G -- No --> I[Follower catches up and rejoins]
        

What a worrying pattern looks like

Note that cluster designs differ. Newer Kafka versions manage the controller through a built-in quorum instead of an external coordination service, and some controller-related metrics have different names or additional companions in that design. The ideas remain the same, so check the documentation for your version when building alerts.

These metrics tell you how comfortably a broker serves its clients, and whether it is close to its limits.

The life of a request

When a producer sends data or a consumer asks for data, the broker treats it as a request. Each request passes through several stages, and Kafka measures how long is spent in each stage. This breakdown is one of the most powerful tools for finding where slowness comes from.

The sum of these stages is the total time of the request. It is reported separately for produce requests, consumer fetch requests and follower fetch requests. When total time is high, look at the stage breakdown to see which stage grew.

Saturation signals

Kafka stores everything on disk and leans heavily on the operating system. Host metrics explain many Kafka symptoms.

Disk space

Kafka keeps messages for a configured time or up to a configured size. If traffic grows faster than expected, the disks fill. A full disk can make a broker stop or corrupt its work, so disk usage is one of the most important numbers to watch. Chart it per broker, look at how fast it is growing and calculate how many days remain until it is full. Growth rate matters even more than the current percentage.

Disk speed and the page cache

Kafka writes to disk sequentially, which is fast, and relies on the operating system’s page cache, a part of memory used to hold recently used file data. Consumers that read recent data are usually served straight from memory. But consumers that read very old data, for example after a long outage, force the disk to work hard, and can evict useful cached data for everyone else. Watch disk read and write speed, wait times and memory use to see when this is happening.

Log flush metrics

Kafka also reports how often and how long its log flushes to disk take. In most setups Kafka lets the operating system decide when to flush, so these numbers are informational, but unusually high flush times can reveal a struggling disk.

A producer application sees the cluster through its own eyes. Its metrics show how well it is delivering messages.

A quick reminder of how a producer works

A Kafka producer does not send each message on its own. It gathers messages into batches in memory, one waiting area per partition, and sends a batch when it is full or when a small waiting time has passed. A background thread carries the batches to the brokers and waits for acknowledgements. Producer metrics describe each step of this process.

Efficient batching is the secret of a fast producer. These metrics show how well it is doing.

Reading the batching story

Speed is only useful if messages arrive. These metrics show how reliably and how quickly.

Why retries matter

When a send fails temporarily, the producer retries automatically. That is helpful, but retries have costs. They add delay, they load the brokers a second time, and, unless the producer is set to avoid duplicates, they can create duplicate messages. A rising retry rate is often the first visible symptom of a problem that will later become failures. Treat it as an early warning.

flowchart TD
    A[Producer sends batch] --> B{Broker acknowledges in time?}
    B -- Yes --> C[Record delivered]
    B -- No --> D{Error can be retried?}
    D -- Yes --> E[Retry counted, delay added]
    E --> A
    D -- No --> F[Record error counted]
    E --> G{Retry time exhausted?}
    G -- Yes --> F
        

Latency as the application sees it

Your application does not care about broker internals. It cares about how long it waits. If your application blocks while sending, high producer latency directly slows your users. Compare the producer’s request latency with the broker’s own produce total time. If the producer’s number is much larger, the extra delay lives in the network or in the producer’s own queueing, not inside the broker.

A producer has a limited amount of memory. When it runs short, the application starts to feel it.

The buffer

The producer keeps unsent batches in a memory area of fixed size. If the application produces faster than the cluster accepts, the buffer fills. When it is full, the send call must wait, and if the wait is too long it fails with an error. This is a form of automatic back-pressure, and the metrics let you see it building before it hurts.

Throttling: a message from the cluster

Cluster administrators can set quotas that limit how much data a client may send or read per second, so that one noisy application cannot harm the rest. When a producer exceeds its quota, the broker delays its responses, and the producer records the time it was held back. A throttled producer is not broken, it is being told to slow down. The fix is either to reduce its rate or to ask for a larger quota.

For consumers, one number stands above the others. Understanding it well is worth the effort.

What is consumer lag?

Every message in a partition has a position number called an offset. The newest message sits at the end of the log. A consumer keeps track of how far it has read. Consumer lag is the distance between the end of the log and the consumer’s position, counted in messages. It answers a simple question: how far behind is this consumer?

flowchart LR
    A[Start of log] --> B[Consumer position]
    B -->|Lag = messages not yet read| C[End of log]
        

Reading lag correctly

Where to see lag

Each consumer reports its own maximum lag across the partitions it reads, and per-partition lag as well. Lag is also visible from outside, by comparing the group’s committed positions with the end of each log. Many teams collect lag from outside with a dedicated monitor, because it also works when a consumer has died and stopped reporting anything about itself.

Lag in messages versus lag in time

A lag of ten thousand messages may mean two seconds on a fast topic and two hours on a slow one. Business users care about time. Where possible, estimate how long the backlog would take to drain, or how old the oldest unread message is. That number is far easier to explain and to set alerts on.

Lag says that something is slow. Fetch and poll metrics help explain where.

How a consumer works, briefly

A consumer runs a loop. It asks the brokers for a batch of messages, which is a fetch, then processes them, and then asks again by calling poll. If it takes too long to come back to poll, the group decides the consumer is dead and reassigns its work. So the balance between fetching and processing time is central.

flowchart TD
    A[Lag is growing] --> B{Is poll idle ratio very low?}
    B -- Yes --> C[Application processing is the bottleneck]
    B -- No --> D{Is fetch latency high?}
    D -- Yes --> E[Broker or network is slow]
    D -- No --> F{Are all partitions being read?}
    F -- No --> G[Group assignment or a stuck consumer]
    F -- Yes --> H[Add consumers up to partition count or tune fetch sizes]
        

Consumers work in groups. The health of the group is as important as the health of each member.

What a rebalance is

Consumers in a group share the partitions of a topic. When a consumer joins, leaves, crashes or is considered dead, the group performs a rebalance and redistributes the partitions. During a rebalance, some or all consumers may pause reading for a short time. Occasional rebalances are normal. Frequent ones hurt throughput and increase lag.

Common causes of frequent rebalances

The real skill of monitoring is connecting a symptom on one side to its cause on another.

A walk-through

Imagine that a business report is late every morning. You open the consumer dashboard and see that lag grows for hours after midnight. You check the poll idle ratio: it is near zero, meaning consumers are busy all the time. Fetch latency is normal, and broker metrics look calm. The story is now clear: the cluster is fine, and the consumer application cannot process the night-time surge fast enough. The fix lies in the consumer’s processing speed or in the number of partitions and consumers, not in the brokers. Without the three views, the team might have wasted a day tuning the wrong thing.

Too few alerts means missed problems. Too many alerts means people ignore them all. The trick is to alert on what truly matters.

Two kinds of alerts

Exact thresholds depend on your workload and your promises to users, so treat the table as a starting point and tune it against real history. Base alerts on duration and trend, not only on single readings. A metric that touches a bad value for one second is often harmless, while a metric that stays bad for ten minutes is not.

flowchart LR
    M[Metric readings] --> R{Rule: bad value for long enough?}
    R -- No --> OK[No alert]
    R -- Yes --> S{User impact or data risk now?}
    S -- Yes --> PG[Page the on-call person]
    S -- No --> TK[Create a ticket]
        

A dashboard is a story told in charts. A cluttered one hides the plot.

Habits that keep monitoring trustworthy, and traps that make it useless.

Good habits

Anti-patterns to avoid

Let us see how the metrics tell stories in situations that teams meet all the time.

If you must choose one on the broker side, choose under-replicated partitions, because it reflects the safety of your data. On the consumer side, choose consumer lag, and on the producer side, the error and retry rates.

Zero in steady state. Short spikes can happen during restarts or brief network issues, but a value that stays above zero means followers are not keeping up and needs investigation.

Lag is the number of messages a consumer has not yet read. Some lag is normal, because new messages keep arriving. Growing or stuck lag is the warning sign.

Because healthy brokers do not guarantee happy applications. Full producer buffers, slow processing in consumers and rebalance storms are visible only from the client side.

Averages hide slow requests. High percentiles and maximums reveal the slow tail that some of your users or applications actually experience.

Kafka publishes them through JMX. A helper component reads them and passes them to a monitoring system, which stores them, draws charts and sends alerts.

Consumer fetch requests may be held by the broker on purpose until enough data arrives or a waiting limit passes. This waiting is part of normal operation, so judge fetch latency together with throughput and lag.

The broker’s handler threads are almost always busy. Requests begin to queue, latency rises, and any extra traffic can cause a sudden slowdown. Investigate load, disk speed and configuration before it gets worse.

During a rebalance, consumers may pause reading, so frequent rebalances reduce throughput and increase lag. Investigate unstable consumers, long processing between polls and deployment practices.

Mostly they are stable, but some are added, renamed or deprecated over time, and clusters without an external coordination service expose additional controller metrics. Always check the documentation for the version you run.

As few as possible while still catching real user impact and data risk. Put everything else on dashboards. Review alerts after every incident and remove those that create noise.

Yes, if the library exposes the standard client metrics. If it hides them, add an external lag monitor so that you are never blind to how far behind a consumer group is.

Kafka metrics are the dashboard of your streaming platform. The broker side tells you about traffic, replication safety, request handling and storage. The producer side shows how efficiently and reliably applications send data. The consumer side reveals how well applications keep up, above all through lag, and how healthy their groups are. No single view tells the whole story. Problems usually leave clues in several places, and the skill lies in following those clues from the symptom to the cause. With sensible alerts, clear dashboards and the habit of watching trends, you can catch trouble long before your users do.