The three letters hide a simple idea: a standard way for a running Java program to describe itself and let others look inside.

JMX stands for Java Management Extensions. It is a part of the Java platform that lets a running program publish information about itself, such as counters and settings, and sometimes accept commands. Because Kafka’s servers and clients are written in Java or Scala and run on the Java Virtual Machine, they can use JMX without any extra software.

The key words

What JMX is not

JMX is only the window. It does not store history, draw charts or send alerts. It shows what a number is right now, and nothing more. To see how a number changed over an hour, or to be woken up when it crosses a limit, you connect JMX to a monitoring system that reads it regularly and records the results. Later chapters show how.

Kafka could have invented its own reporting system. It chose JMX for practical reasons that still hold today.

Built into the platform

Every Java program already has an MBean server waiting to be used. Kafka only has to place its numbers into it. No extra service has to be installed, no extra port has to be built into Kafka’s own code, and no new protocol has to be invented. Any tool that understands JMX can read Kafka, and almost every monitoring product does.

A common language for all components

Brokers, producers, consumers, Kafka Streams applications and Kafka Connect workers all publish metrics in the same way. A person who learns to read one learns to read them all. The same monitoring tool, the same naming style and the same habits work across the whole ecosystem.

Live, low-cost, always on

Kafka updates its counters as it works, often with a single tiny in-memory step per event. Reading them later costs almost nothing, because the numbers are already sitting in memory. The cost of collecting metrics is therefore small compared with the work Kafka does.

Two families of metric code

Under the surface, Kafka has two metric systems. The brokers mostly use a library called Yammer Metrics, which offers meters, histograms and timers with familiar names such as one-minute rate and 99th percentile. The clients, meaning producers, consumers, Streams and Connect, use Kafka’s own metrics library, which is simpler and reports plain values like averages, maximums and rates. Both publish through JMX, but their naming styles differ a little, which is why you will see two flavours of names in this guide.

The names look scary, but they follow a strict pattern. Learn the pattern once, and every metric becomes readable.

The pattern

An MBean’s ObjectName has two parts separated by a colon. The part before the colon is the domain, which groups metrics by area. The part after the colon is a list of key=value labels separated by commas, which pin down exactly which metric is meant. Here is a real example from a broker:

kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

flowchart LR
    A["Domain: kafka.server (which area)"] --> B["type=BrokerTopicMetrics (which group)"]
    B --> C["name=MessagesInPerSec (which metric)"]
    C --> D["Attribute: OneMinuteRate (which reading)"]
    

The common domains

Extra labels for clients and topics

Client metrics carry an extra label for the client’s identifier, so many applications in the same process or many processes on the same monitoring system do not collide. A producer’s name looks like kafka.producer:type=producer-metrics,client-id=orders-service. Topic-level and partition-level metrics add labels such as topic and partition, which multiplies the number of metrics quickly. A cluster with thousands of partitions publishes tens of thousands of per-partition metrics, which is worth remembering when we discuss overhead in chapter 17.

Not all numbers behave alike. Knowing which kind you are looking at tells you how to read it and how to graph it.

Reading meters

A broker meter such as messages in per second offers several attributes. Count is the total since startup. MeanRate is the average since startup, which is rarely useful on a long-running broker. OneMinuteRate, FiveMinuteRate and FifteenMinuteRate are smoothed averages that give more weight to recent moments. For alerting on current traffic, the one-minute or five-minute rate is usually the best choice.

Reading histograms and percentiles

Averages hide pain. Suppose ninety-nine requests take one millisecond and one request takes a full second. The average looks fine, yet one user waited a long time. Percentiles reveal this. The 99th percentile means “ninety-nine percent of requests were faster than this”. Broker timing metrics offer the median (50th), 75th, 95th, 98th, 99th and 99.9th percentiles. Alerting on a high percentile catches trouble that averages would miss.

Client metrics use plainer names

Kafka’s own client metrics library reports values such as average, maximum and rate as separate metric names, for instance request-latency-avg and request-latency-max. They are calculated over a short sampling window, which by default is thirty seconds with two samples, so client numbers reflect roughly the last minute of activity.

Every Kafka program has its own set of metrics. Knowing whom to ask about what saves a great deal of time.

flowchart TB
    subgraph Cluster["Kafka cluster"]
        BR["Brokers: traffic, replication, requests"]
        CT["Controller: leaders and elections"]
    end
    PR["Producer applications"] --> BR
    BR --> CO["Consumer applications"]
    KS["Kafka Streams applications"] --> BR
    KC["Kafka Connect workers"] --> BR
    BR -. "each publishes JMX metrics" .-> MON["Monitoring system"]
    PR -. "JMX" .-> MON
    CO -. "JMX" .-> MON
    KS -. "JMX" .-> MON
    KC -. "JMX" .-> MON
    

Brokers versus clients

A common mistake is to watch only the brokers. Brokers can look perfectly healthy while an application is failing, because a bad client may simply stop sending or reading. Watching the client side is equally important. Broker metrics answer “is the service up and fast?”, and client metrics answer “is my use of the service working?”.

Reading Kafka’s numbers takes a few small steps: allow access, connect a tool, and browse.

Local access versus remote access

Every Java program has JMX inside, but by default it is only reachable from the same machine, and tools must attach to the running process. To read metrics from another machine, the program has to open a network port for remote JMX. Kafka’s start-up scripts make this easy: setting an environment variable named JMX_PORT before starting a broker or a client tells the script to open remote JMX on that port.

Tools for looking around

A first exploration

A gentle way to begin is to connect JConsole to a test broker and open three MBeans. Look at the messages-in rate under broker topic metrics, the under-replicated partition count under the replica manager, and the heap memory under the Java runtime. Send a few test messages and watch the first number change. In ten minutes you will understand what a JMX metric feels like.

JMX shows numbers for this moment only. To see trends and receive alerts, you need a pipeline that reads them regularly, stores them and draws them.

The four jobs of a monitoring pipeline

flowchart LR
    K["Kafka broker JVM with MBeans"] --> A["Exporter or agent reads JMX"]
    A --> S["Metrics store: time-series database"]
    S --> D["Dashboards"]
    S --> R["Alert rules"]
    R --> N["Notification: pager, chat or email"]
    

Popular ways to collect

Pull and push

In a pull design, the monitoring system visits each Kafka machine and asks for the current numbers. This is how Prometheus works. In a push design, an agent on each machine sends numbers out to a central place. Pull makes it easy to see when a machine has stopped answering, because the visit fails. Push suits environments where the monitoring system cannot reach into the machines.

Making names friendly

Raw MBean names are long and use characters that many databases dislike. Exporters therefore apply rules that translate a name such as the messages-in meter into a short flat name like kafka_server_brokertopicmetrics_messagesin_total, and turn labels such as topic or client identifier into database labels. Well-written rules keep the number of distinct names small and consistent, which makes dashboards reusable across clusters.

If you can watch only a few numbers, watch these. They tell you whether the cluster is safe, available and keeping up.

Reading the checklist together

• Availability: offline partitions and active controller count tell you whether the cluster can serve traffic at all.
• Safety: under-replicated partitions and unclean elections tell you whether data is protected against a broker failure.
• Capacity: the idle percentages tell you whether threads are running out of headroom.
• Speed: total request time tells you whether users feel slowness.

Two families answer the question “how busy is the cluster, and how quickly does it respond?”.

Traffic: how much is flowing

These metrics exist for the whole broker and also for each topic, when you add the topic label. Broker-wide numbers give the big picture, and per-topic numbers show which topic is responsible for a change.

Latency: how long requests take

Every request to a broker passes through five stages, and Kafka measures each one. The stages tell you not only that something is slow, but where the time is spent.

flowchart LR
    A["Request arrives"] --> B["RequestQueueTimeMs: waiting in the request queue"]
    B --> C["LocalTimeMs: processed on the leader"]
    C --> D["RemoteTimeMs: waiting for followers or data"]
    D --> E["ResponseQueueTimeMs: waiting in the response queue"]
    E --> F["ResponseSendTimeMs: sending the reply"]
    F --> G["TotalTimeMs is the sum of all stages"]
    

Fetch requests come in two kinds

Requests are also labelled by type. Produce is a write from a producer. FetchConsumer is a read from a consumer. FetchFollower is a read by another broker copying data. Consumer fetches may legitimately wait for new data to arrive, so their remote time is often high and is usually not a problem. Produce requests with acknowledgement of all show remote time as the time waiting for followers, which is a real signal about replication health.

Kafka keeps several copies of each partition. These metrics tell you whether the copies are keeping up and who is in charge.

The basic idea of replication

Each partition has one leader copy that handles reads and writes, plus follower copies on other brokers that keep themselves up to date by fetching from the leader. The set of followers that are fully caught up, together with the leader, is called the in-sync replica set, or ISR. A follower that falls too far behind, by default for about thirty seconds, is removed from the set.

sequenceDiagram
    participant L as Leader broker
    participant F1 as Follower 1
    participant F2 as Follower 2
    L->>F1: Data fetched, in sync
    L->>F2: Data fetched, in sync
    Note over F2: Slow disk, falls behind
    L->>L: Follower 2 removed from ISR
    Note over L: IsrShrinksPerSec rises and UnderReplicatedPartitions becomes 1
    F2->>L: Catches up again
    L->>L: Follower 2 rejoins ISR
    Note over L: IsrExpandsPerSec rises and UnderReplicatedPartitions returns to 0
    

The key metrics explained

The controller

One broker, or in newer clusters a small group of controller nodes, takes the role of controller. It decides which broker leads each partition and reacts when brokers join or leave. Metrics such as the active controller count, the rate and duration of leader elections, and the preferred replica imbalance count show whether the control plane is stable. Frequent elections point to brokers flapping in and out of the cluster.

Behind the scenes, a broker is a small factory of threads and queues. These metrics show when the factory is running out of room.

Two thread pools

A broker uses network threads to read requests from and write replies to the network, and request handler threads to do the actual work such as writing to disk. Each pool reports an average idle percentage between zero and one. Near one means the threads are mostly waiting for work, which is healthy. Near zero means they are always busy, and requests will start to queue.

Queues

Storage and disk behaviour

Kafka stores data in log files divided into segments. Metrics under the log domain report on flush time, segment counts and cleaning for compacted topics. Kafka leans heavily on the operating system’s page cache for speed, so many of the most useful storage numbers, such as disk utilization, read and write latency and network throughput, come from the operating system rather than JMX. A complete picture uses both.

Kafka is a Java program, so the health of the Java runtime is part of the health of Kafka. JMX exposes it under the java.lang domain.

Memory and garbage collection

A Java program keeps most of its objects on the heap. From time to time, a component called the garbage collector cleans up objects that are no longer needed. During some cleaning steps, the program pauses briefly. Long pauses are dangerous for Kafka, because a paused broker cannot answer followers, and it may be removed from the in-sync set or lose its session with the cluster.

Why file descriptors matter for Kafka

A broker keeps open files for every log segment it manages, plus a network connection for every client and follower. A cluster with thousands of partitions and many clients can hold tens or hundreds of thousands of open handles. If the count reaches the operating system limit, the broker can no longer open files or accept connections, and it fails in confusing ways. Watching the open count against the maximum, both available under the operating system MBean, prevents this.

A producer reports on its own writing. These numbers tell you whether your application is sending quickly, reliably and efficiently.

Producer metrics live under the kafka.producer domain, in groups labelled with the client identifier. The most important group is producer-metrics, which describes the whole producer. Other groups cover each topic and each connection to a broker.

Reading retries and errors

A small, steady retry rate is normal on a busy cluster, because networks have hiccups. A retry rate that suddenly grows is an early warning that brokers are slow or leaders are changing. Retries that finally fail appear as errors, and errors mean data your application believed it was sending did not arrive. If you use idempotent producers, retries are safe from duplicates, yet they still cost time, so a high retry rate is still worth investigating.

Reading batch size and compression

Small batches mean many small requests, which use more processor time and more network overhead per record. If the average batch is only a few hundred bytes, the producer is probably sending as soon as any record arrives. Allowing a short wait time and larger batch size often improves throughput dramatically. Compression rate tells you whether compression is worthwhile: a value of 0.3 means the compressed batch is 30 percent of the original size, a big saving.

Consumers face a different question from producers: are they keeping up? One number, called lag, answers it.

What lag means

Every message in a partition has a position number called an offset. The newest message sits at the log end offset. A consumer saves how far it has read, the committed offset. The gap between the two is the lag: how many messages are waiting to be read. Small, steady lag is normal. Lag that keeps growing means the consumer is slower than the producers.

flowchart LR
    A["Oldest message"] --> B["Committed offset: consumer has read up to here"]
    B --> C["Messages waiting: this gap is the lag"]
    C --> D["Log end offset: newest message"]
    

The important consumer metrics

Two kinds of trouble

Measuring lag from outside

The consumer’s own lag metric only exists while the consumer is running. If it crashes, the metric disappears at exactly the moment you most need it. For this reason, many teams also measure lag from outside, comparing each group’s committed offsets with the log end offsets. Kafka ships a command-line tool for this, and LinkedIn open-sourced a service called Burrow that watches groups over time and classifies them as healthy, warning or in error, rather than relying on a fixed lag threshold.

Two higher-level Kafka tools publish their own sets of metrics, layered on top of the producer and consumer ones.

Kafka Streams

A Streams application is built from threads, and each thread runs tasks. The metrics follow that structure and sit under the kafka.streams domain. Metrics are grouped by level of detail through the metrics.recording.level setting: the default INFO level records the most useful numbers cheaply, while DEBUG and TRACE add detail that costs more processing time and should be turned on only when investigating.

Kafka Connect

Connect metrics are grouped under kafka.connect and follow its structure of workers, connectors and tasks. A worker reports how many connectors and tasks it runs and how often they start or fail. Each task reports its status, how often it commits progress, and how many records it reads or writes.

A metric nobody watches is only decoration. Good alerts turn numbers into timely, actionable messages, without waking people for nothing.

Four questions for every service

Site reliability teams often describe the health of a service using four signals. They map neatly onto Kafka.

Choosing what to alert on

flowchart TB
    M["Metric crosses a limit"] --> D{"Does it last longer than the wait time?"}
    D -- "no" --> I["Ignore: brief spike"]
    D -- "yes" --> S{"Does it hurt users or data now?"}
    S -- "yes" --> P["Page the on-call person"]
    S -- "no" --> T["Open a ticket for working hours"]
    P --> R["Link to a runbook"]
    T --> R
    

Habits of good alerts

• Alert on symptoms first: a user-facing problem such as high latency matters more than a cause such as a busy thread, though causes help diagnosis.
• Add a waiting period: require the condition to hold for several minutes to ignore brief blips.
• Attach a runbook: each alert should link to a page explaining what it means and what to try first.
• Review regularly: delete alerts that never lead to action, and tighten those that fire too late.

Monitoring has a cost. Done carelessly, it can slow the very brokers it watches or flood your metrics database.

Why reading JMX is usually cheap

Kafka updates its counters as part of normal work with tiny in-memory steps. Reading a handful of MBean attributes every thirty seconds costs very little. Problems appear when a collector asks for far more than it needs, or asks too often.

The cardinality problem

Cardinality means the number of distinct time series a system must track. Each combination of metric name and labels is one series. Per-partition metrics on a large cluster multiply quickly. A cluster with two thousand partitions, ten metrics per partition and three copies of each can produce tens of thousands of series from one metric family alone. Storing and drawing that many series slows dashboards, raises storage costs and, in hosted services, raises the bill.

Practical ways to keep it light

A metrics port is a door into a running server. It deserves the same care as any other door.

The danger of an open JMX port

Remote JMX is powerful. Beyond reading numbers, it can let a connected client load code or change settings inside the program. An unprotected JMX port reachable from an untrusted network has a long history of being abused to take over servers. Treat any open JMX port as a potential entrance for attackers.

What metrics can reveal

Metrics are not secrets, but they are informative. Topic names, client identifiers, consumer group names and connector names can reveal business structure. Traffic patterns can hint at customer activity. Give dashboards and metric stores the same access rules you give other operational data, and avoid putting sensitive words into client identifiers.

Metrics and logs work together

Metrics say that something is wrong and roughly when. Logs say why. When under-replicated partitions rise at 02:14, the broker log around 02:14 might show a long garbage collection pause, a disk error or a session timeout. Keep clocks synchronized on all machines, so timestamps line up, and keep logs long enough to look back after an incident. Turn on garbage collection logging, since JMX shows totals while the log shows each pause.

Teams that run Kafka for years tend to converge on the same handful of good habits, and to trip over the same handful of bad ones.

Patterns worth copying

Anti-patterns to avoid

A compact list of habits to adopt and traps to avoid when you monitor Kafka through JMX.

Mistakes beginners meet first

A starter routine for a new cluster

Public talks, open-source projects and vendor documentation show how well-known organizations turn Kafka’s metrics into daily practice.

Short answers to questions that beginners and experienced engineers ask most often about JMX and Kafka metrics.

If you keep only one page of this guide, keep this one.

JMX is the window through which a Kafka cluster tells you how it is doing. Brokers, producers, consumers, Kafka Streams applications and Kafka Connect workers all publish measurements as named MBeans. By reading a small, well-chosen set of them, and turning those readings into dashboards and thoughtful alerts, you can notice trouble before customers do and understand it quickly when it happens.