Before diagnosing anything, we need a shared vocabulary and a clear mental picture of what the alert is measuring.

Partitions and their copies

Kafka stores messages in topics, and each topic is cut into smaller pieces called partitions. A partition is an ordered list of messages that only grows at the end, a bit like a long paper roll where new notes are always added at the bottom. To survive machine failures, Kafka keeps several copies of every partition on different servers. These servers are called brokers, and each copy is called a replica.

The number of copies is called the replication factor. A replication factor of three is very common in production because it survives the loss of one machine comfortably and even two machines with some care. The complete list of brokers that should hold a copy of a partition is called the assigned replica set.

The definition in one sentence

A partition is under-replicated when the number of replicas currently in sync is smaller than the number of replicas assigned to it. If a partition is assigned to three brokers but only two of them are in sync, the partition is under-replicated. The count of such partitions is one of the most watched numbers in any Kafka dashboard.

Who counts the problem?

There is a subtle detail that confuses many people. The under-replicated count is reported by the broker that is the leader of the partition, not by the slow or missing follower. The leader is the one who knows which followers are keeping up, so it is the leader that notices when the in-sync set has shrunk. This means that when one broker becomes sick, the alerts usually appear on the healthy brokers that lead partitions with a copy on the sick broker.

Because of this, the broker that is shouting is very often not the broker that is broken. Beginners waste a lot of time restarting the noisy broker. Experienced engineers instead ask a different question: which broker is missing from the in-sync sets of all these partitions? The answer to that question usually points straight at the real cause.

To diagnose a copy that is falling behind, you must first understand exactly how a copy is kept up to date.

The log and its offsets

Each replica stores its partition as a log on disk. Every message in the log gets a number called an offset, starting at zero and increasing by one for each new message. The offset of the next message that will be written is called the log end offset. Two replicas are perfectly identical when their log end offsets match and their messages are the same.

The leader always has the newest data because every producer writes to it. Followers must copy this data as fast as they can. The distance between the leader’s log end offset and a follower’s log end offset is the replication lag, measured in messages. A lag of zero means the follower is a perfect mirror.

Followers pull, the leader does not push

Kafka replication is pull-based. A follower sends fetch requests to the leader, very similar to how a consumer reads data. The leader answers with the next batch of messages, the follower appends them to its own log, and then immediately asks again. This design means the follower controls its own pace, and the leader never has to track a complicated list of pushes.

Each fetch request also carries a hidden message: the follower tells the leader what offset it wants next. From this the leader learns exactly how far each follower has copied. This tiny piece of information is the foundation of everything in this guide, because the leader uses it to decide who is in sync and who is not.

The high watermark: what counts as safely stored

A message is not considered committed just because the leader wrote it. It becomes committed only after every replica in the in-sync set has copied it. The offset up to which everything is committed is called the high watermark. Consumers are allowed to read only messages below the high watermark, so they never see data that could disappear if the leader crashed a moment later.

Followers learn the high watermark from the leader’s fetch responses, so the high watermark on a follower always trails a little behind the leader. This small delay is normal, and it is one reason why a slow follower can slow down the whole partition: the high watermark cannot move forward faster than the slowest member of the in-sync set.

sequenceDiagram
    participant P as Producer
    participant L as Leader Broker
    participant F1 as Follower A
    participant F2 as Follower B
    P->>L: "Send message batch with acks=all"
    L->>L: "Append to local log"
    F1->>L: "Fetch from my next offset"
    L-->>F1: "Here are the new messages"
    F2->>L: "Fetch from my next offset"
    L-->>F2: "Here are the new messages"
    F1->>L: "Fetch again - I now have them"
    F2->>L: "Fetch again - I now have them"
    L->>L: "Advance high watermark"
    L-->>P: "Acknowledged - safely committed"
        

Leader epochs and clean recovery

Leadership changes over time. Each new leadership period is labeled with a number called the leader epoch. When a follower comes back after being away, it uses the epoch history to work out exactly where its log stopped agreeing with the current leader, cuts off any extra messages that never became committed, and then continues copying. This mechanism makes recovery safe and automatic, but it also means a returning follower has real work to do before it can rejoin the in-sync set.

The in-sync replica set is the heart of the whole topic, so it deserves a careful, patient explanation.

How a replica earns membership

The in-sync replica set always contains the leader. A follower stays in the set as long as it has kept up with the leader closely enough. The rule in modern Kafka is based on time, not on the number of missing messages. A follower is considered out of sync if it has not managed to catch up to the leader’s latest data within a limit controlled by the setting replica.lag.time.max.ms, which defaults to thirty seconds.

More precisely, the leader remembers the last moment each follower was fully caught up. If that moment is older than the allowed limit, the follower is removed from the set. Removal is called an ISR shrink. When the follower later catches up again, it is added back, which is called an ISR expand.

Why time and not message count?

Older Kafka versions also allowed a limit based on how many messages a follower could trail. This turned out to be a poor rule. A short burst of traffic could push healthy followers past the message limit for a moment, and quiet topics could hide real trouble. Time-based measurement solves this, because it asks a fairer question: has this follower been able to fully catch up recently? A busy follower that is always a few messages behind but catches up every second is healthy. A follower that never catches up for thirty seconds is not.

This has an important consequence for diagnosis. A very quiet partition with little traffic almost never becomes under-replicated by itself, because followers reach the leader’s latest offset easily. If a quiet partition is under-replicated, the follower is probably truly unreachable or stuck, not merely slow.

flowchart LR
    A["Follower is in sync"] -->|"Not caught up for longer than lag limit"| B["Leader shrinks ISR"]
    B --> C["Partition is under-replicated"]
    C -->|"Follower fetches and reaches leader high watermark"| D["Leader expands ISR"]
    D --> A
    C -->|"Follower stays broken"| E["Under-replication persists"]
        

The settings that shape ISR behavior

How a change to the ISR is recorded

When a leader decides that a follower must leave or join the in-sync set, it cannot simply announce the decision and move on. The official record lives with the controller, so the leader sends a request asking the controller to update the membership. Only after the controller agrees and stores the new membership does the change become official and get shared with the other brokers. This careful order prevents two leaders from holding different opinions about who is in sync.

This detail explains a puzzling situation seen in real clusters: a partition can look healthy on the leader’s own view of its followers while the official record still shows a smaller set, or the other way around. The gap usually lasts a very short time, but if the controller is slow or unreachable, it can last long enough to trigger alerts. Whenever the numbers on the dashboard and the numbers from the topics tool disagree, remember that they may be looking at two different records of the same truth.

What the follower knows about itself

A follower does not know whether the leader considers it in sync. It only knows how far it has copied and what the leader last told it about the high watermark. This is why followers rarely log complaints when they are dropped. The evidence of being dropped appears on the leader, in its logs and metrics, and in the controller record. When you investigate a suspect follower broker, remember that its own logs will show symptoms such as fetch errors or slow disk warnings, but the moment of removal is written by the leader.

The alert deserves respect because it quietly changes what Kafka can promise you.

Durability shrinks with the ISR

Durability means data survives failures. With three copies in sync, Kafka can lose a broker and still have all committed data safe on the others. When the ISR shrinks to two, you can still lose one more broker. When it shrinks to one, the leader is holding the only trustworthy copy, and a single hardware failure at that moment can lose data that producers were told was written.

This is why an under-replicated partition is called a risk that has not yet become an incident. Nothing has been lost, but the safety margin has thinned, and the thinner it gets, the more a small accident matters.

Availability and the minimum ISR rule

Producers that use the strongest acknowledgment level, called acks=all, wait until every in-sync replica has the message. To prevent the dangerous case of “all” meaning “just the leader”, Kafka offers the topic and broker setting min.insync.replicas. If the ISR shrinks below this number, the leader refuses new writes with an error, and producers receive a message saying there are not enough replicas.

This behavior is deliberate. Kafka chooses to protect data over accepting writes. A common production pattern uses three replicas with a minimum ISR of two, which tolerates one broker failure without any interruption while refusing writes if two brokers are lost.

Latency and throughput

Under-replication also hurts speed. With acks=all, the leader must wait for the slowest in-sync follower. A follower that is slow but still barely in the set can drag write latency up for every producer. Consumers feel it too, because the high watermark, which limits what they can read, moves forward only as fast as the slowest in-sync member.

When a lagging follower is catching up, it reads old data from the leader’s disk instead of from fast memory. This extra disk reading competes with normal traffic and can make things worse for everyone, a pattern that creates a nasty feedback loop later in this guide.

What producers and consumers actually experience

Applications rarely see the words “under-replicated”. They see slower writes, occasional errors, and delayed reads. A producer using the strongest acknowledgment level waits for the in-sync followers, so its send time grows when a follower is slow. If the in-sync set falls below the minimum, the producer receives a retriable error, and well-configured producers retry until their overall delivery time limit expires. When retries are combined with the idempotent producer setting, retries do not create duplicate messages, which is why that combination is the recommended default for important data.

Consumers experience the problem differently. Because they can only read below the high watermark, a slow follower that is still in the in-sync set holds the high watermark back, so new messages become visible to consumers a little later. This shows up as growing end-to-end delay even though consumer processing is perfectly healthy. Kafka also allows consumers to read from a nearby follower instead of the leader, which is helpful for reducing cross-zone traffic, but a follower that is lagging simply returns older data to those consumers, so the same delay is visible there too.

These application-level symptoms are useful clues. If several unrelated applications report slow sends to the same topics at the same time, replication is a strong suspect. If only one application slows down while others are fine, the problem is more likely in that client or its network path than inside the cluster.

A good diagnosis is built from a small group of metrics that each answer a different question.

The core metric family

Kafka brokers expose their internals through JMX metrics. Several of them describe replication health. No single metric is enough. You combine them the way a doctor combines temperature, blood pressure, and pulse to understand a patient.

Reading the shrink and expand pair

ISR shrinks and expands should be read together. A single shrink followed quickly by an expand is a brief stumble, such as a short pause or a network hiccup. Many shrinks with no expands means followers are leaving and not returning, which points to a persistent fault. A steady, repeating pattern of shrink and expand over and over is called flapping, and it usually means a resource is hovering right at its limit.

Metrics that reveal the cause, not only the effect

The replication metrics tell you that something is wrong. To learn why, you need resource and request metrics. Request handler idle percentage and network processor idle percentage show whether brokers have spare thinking capacity. Request time breakdowns, especially for follower fetch requests and for produce requests, show whether time is being lost queueing, in local disk work, or waiting for other brokers. The size of the produce purgatory shows how many writes are waiting for followers to confirm.

When the alert fires, a calm method beats a fast guess every time.

Ask four questions before touching anything

The best engineers do not start by changing settings. They start by asking questions that quickly narrow the problem. The four most valuable questions are about scope, timing, direction, and correlation. Each one removes a large set of possible causes.

flowchart TD
    S["Under-replicated partitions alert fires"] --> Q1{"Is any partition offline or under minimum ISR?"}
    Q1 -->|"Yes"| U["Treat as urgent - customer impact possible"]
    Q1 -->|"No"| Q2{"Is a planned operation running?"}
    U --> Q2
    Q2 -->|"Yes"| E["Expected - watch that the count trends down"]
    Q2 -->|"No"| Q3{"Do all affected partitions share one broker?"}
    Q3 -->|"Yes"| B["Investigate that broker - process, disk, network, memory"]
    Q3 -->|"No"| Q4{"Is the spread across many brokers?"}
    Q4 -->|"Yes"| C["Investigate cluster wide causes - controller, network, load"]
    Q4 -->|"No"| T["Investigate specific topics - hot partitions, large messages"]
        

Find the missing broker

The fastest technique in the entire diagnosis is to list the affected partitions together with their replica lists and in-sync lists, then look for a broker that appears in the replica lists but is absent from the in-sync lists. If the same broker identifier is missing again and again, that broker is your prime suspect. If the missing broker changes from partition to partition, the problem is more general.

The most frequent real-world pattern is one broker that has either vanished or slowed to a crawl.

Dead versus slow

A broker can fail in two very different ways. A dead broker has stopped completely: the process crashed, the machine rebooted, or the network cable is gone. A slow broker is still running but cannot keep up. Both make followers on that broker fall out of the in-sync sets, but the clues differ.

A dead broker disappears from the cluster membership, its metrics stop arriving, and its leaders move elsewhere. A slow broker still reports metrics, still appears in the cluster, and may even continue leading partitions while performing poorly. Slow brokers are often more damaging than dead ones, because the cluster keeps sending them work.

Investigating a dead broker

Start with the operating system. Was the process killed because the machine ran out of memory? Did the host reboot? Was a disk unmounted? Read the last lines of the broker log before it stopped, since Kafka usually writes a reason if it can. Common reasons include a fatal error on a storage directory, running out of file descriptors, running out of memory, or a corrupted log discovered during startup.

If the broker is trying to start but takes a very long time, it may be performing log recovery. After an unclean shutdown, Kafka must check the tail of every log segment to make sure it is valid, and this can take a long time when a broker holds many partitions. During recovery the broker is not serving traffic, so under-replication persists until it finishes and then catches up.

Investigating a slow broker

For a slow broker, compare its resource graphs against its peers. Disk busy time and write latency, network throughput and errors, CPU usage, garbage collection pause time, and request handler idle percentage should all look similar across brokers that carry similar load. The one that stands out is the culprit. When a single broker looks different from all others while carrying the same work, suspect hardware or environment, not Kafka itself.

Alive but cut off from its peers

A third kind of broker failure sits between dead and slow. The broker is running, its disks are fine, and clients may even reach it, yet it cannot talk properly to some other brokers. This happens with partial network splits, firewall rules that block only certain ports, or name resolution problems that send replication traffic to the wrong address. Followers on such a broker fail to fetch from leaders elsewhere, while the broker looks perfectly healthy in its own graphs.

The clue is in the logs. The follower broker records repeated errors when it tries to fetch from particular leaders, such as connection timeouts or refused connections, and only for particular partners. Test the connection between the specific pair of brokers involved, in both directions, because a path can work one way and fail the other.

When many brokers are affected and no single one is guilty, the cause is usually shared: load, network, or coordination.

Overload across the board

A cluster can simply be asked to do more than it can handle. A traffic spike, a burst of new producers, a consumer group replaying many days of history, or a large batch job can push every broker close to its limits at the same time. Replication is one of the first things to suffer, because followers compete for the same disks, network, and threads that serve clients.

The clue is that resource graphs are high on all brokers together, and the alert begins soon after a change in traffic. The fix is to reduce load, add capacity, or protect replication with quotas on clients so that a single heavy user cannot starve the followers.

Network trouble

A shared network problem, such as a congested switch, a failing link between two racks or zones, or a firewall change, can break replication between many broker pairs at once. A useful pattern is that the missing followers are grouped by location. If every follower missing from the in-sync lists lives in one zone or rack while leaders live elsewhere, the network path between those locations is the first thing to examine.

Look at packet loss, retransmissions, connection resets, and round-trip times between the affected brokers. Even a small amount of loss can hurt replication badly, because Kafka relies on long-lived connections and steady throughput, and retransmissions reduce speed sharply.

Control plane trouble

Sometimes replication itself is fine, but the coordination layer is not. Leaders must tell the controller when the in-sync set changes. If the controller is slow, overloaded, or being repeatedly re-elected, these updates can fail or lag, leaving partitions marked as under-replicated even though the data is caught up. Chapter 12 covers this in detail.

flowchart LR
    subgraph ZoneA["Zone A"]
        L1["Leaders"]
    end
    subgraph ZoneB["Zone B"]
        F1["Followers"]
    end
    L1 -->|"Healthy link"| F1
    L1 -.->|"Congested or lossy link"| F1
    F1 --> R["Followers leave ISR and partitions become under-replicated"]
        

When the problem follows specific topics, look at the data and the shape of the traffic rather than the machines.

Hot partitions

A hot partition receives far more traffic than its siblings, usually because many messages share the same key and Kafka sends every message with the same key to the same partition. The leader of a hot partition must handle a huge write rate, and its followers must copy all of it. The followers of that one partition may struggle while the rest of the cluster looks calm.

The clue is that under-replicated partitions belong to one topic or a small group of topics, and a look at per-partition byte rates shows a few partitions far busier than others. The cure is a better key design, more partitions with a more even key spread, or a custom partitioning strategy that scatters heavy keys.

Large messages and large batches

Very large messages take longer to write, read, and copy. If replica fetch limits are small compared to message size, followers must make many round trips to copy each batch. On very old Kafka versions a message larger than the fetch limit could even stop replication completely; modern versions always send at least one batch to guarantee progress, but the copying can still be inefficient. Compare the largest message size allowed on the topic with the follower fetch limits, and confirm that they are consistent.

Partition skew and leader imbalance

If some brokers lead far more partitions than others, they carry more client traffic and more follower fetch load. Under a heavy day, those brokers hit their limits first and their followers fall behind. The metric that counts partitions led by a non-preferred replica shows this imbalance. Rebalancing leadership, or distributing partitions more evenly, spreads the load.

Compaction and retention pressure

Topics with log compaction run background cleaning that reads and rewrites large amounts of data. A busy cleaner competes for disk bandwidth with replication on the same broker. Similarly, aggressive retention deletion of very large segments can cause a burst of disk work. When under-replication appears on compacted topics at regular intervals, check the cleaner activity and its timing.

Every replication delay ultimately traces to one of a few physical or logical resources running out of room.

Disk

Kafka writes sequentially, which disks handle well, but replication multiplies the work. With three copies, every produced byte is written to disk three times across the cluster. Disks that cannot keep up show high busy time, long write latency, and long queues. Cloud volumes add another twist: many have throughput or operation limits and may have burst allowances that run out after a while, which explains why a cluster can run smoothly for days and then suddenly struggle.

Disk problems also come from failed drives, unhealthy storage arrays that are rebuilding, and shared network storage under contention. When a storage directory fails completely, Kafka marks it offline and moves leadership of its partitions to other brokers, which itself creates under-replication until repair.

Network

Replication traffic is easy to forget when planning network capacity. Each broker receives data from producers and also sends data to followers on other brokers, in addition to serving consumers. Outbound traffic is roughly the consumer read traffic plus the replication traffic, which is the produce rate multiplied by the number of followers. A network interface that seems comfortable for clients alone can be overwhelmed once replication is included.

CPU and request threads

Brokers use network threads to move bytes and request handler threads to do the work. If encryption on connections, message conversion between formats, or compression uses much processor time, these threads become busy. When idle percentages fall low, requests wait in queues, including follower fetch requests, and followers appear to lag.

Replica fetcher threads

Followers copy through fetcher threads, one group per source broker. With very many partitions per broker and only one fetcher thread per source, that single thread may not be able to move data quickly enough. Increasing the number of fetcher threads can raise catch-up speed, at the cost of more CPU and more connections, so it should be done with measurement, not by habit.

Storage layout matters

Many brokers use several disks and spread partitions across them. This gives more total capacity and throughput, but it also means that one slow or failing disk affects only the partitions placed on it, which produces a confusing pattern: only some partitions on a broker are under-replicated, not all. If the affected partitions on a broker all live in the same storage directory, look at that disk and not at the broker as a whole.

Uneven placement is a second trap. If one disk holds far more busy partitions than the others, it saturates first while the rest sit idle. Checking per-disk usage and per-directory partition counts reveals this imbalance, and moving a few busy partitions between directories, or using tools that balance directories automatically, brings relief without adding any hardware.

flowchart TD
    A["Follower fetch request arrives at leader"] --> B["Network thread receives it"]
    B --> C["Request queue"]
    C --> D["Request handler thread reads log"]
    D --> E{"Data in page cache?"}
    E -->|"Yes"| F["Fast reply from memory"]
    E -->|"No"| G["Slow read from disk"]
    F --> H["Follower stays in sync"]
    G --> I["Follower may fall behind"]
        

Kafka is unusual because it depends on memory that it does not even own.

The page cache is Kafka’s secret weapon

Kafka does not keep its own big cache of messages. Instead it relies on the operating system’s page cache, a region of free memory the system uses to keep recently used file contents. When followers and consumers are close to the end of the log, their reads are served straight from this cache and never touch the disk. This is a major reason Kafka is fast.

When a follower falls far behind, it needs older data that is no longer in the cache. The leader must read it from disk, which is slower and also pushes fresh data out of the cache. This can harm every other reader on that broker and creates the feedback loop mentioned earlier: a lagging follower makes the leader slower, which makes more followers lag.

Heap size and garbage collection

Kafka runs on the Java virtual machine, which periodically pauses to clean unused objects. This cleaning is called garbage collection. Long pauses freeze the broker: it stops answering fetch requests, stops sending heartbeats, and may even be considered dead by the rest of the cluster. If a pause is longer than the replica lag limit, all its followers leave the in-sync sets, and if it is longer than the session timeout with the controller, the broker can be removed from the cluster.

A common mistake is giving Kafka a very large heap, believing more memory is always better. A larger heap can make collections longer, and it also takes memory away from the page cache. Most production brokers run a modest heap, often in the range of a few gigabytes, and leave the rest of memory for the operating system to use as cache. Garbage collection logs, with pause durations and frequency, are the evidence to read.

Memory pressure outside the heap

Other things use memory too: direct buffers for network operations, memory mapped index files, and in containers, the memory limit set by the platform. If the total goes over the container limit, the platform can kill the process without warning, leaving a dead broker and a long recovery. Check for kill events at the operating system or orchestration level whenever a broker vanishes without a clean shutdown message.

Data replication happens between brokers, but decisions about who is in sync are recorded by a separate coordination layer.

What the controller does

One broker in the cluster acts as the controller. It keeps the official map of the cluster: which brokers are alive, which broker leads each partition, and which replicas are in sync. When a leader decides that a follower should leave or join the in-sync set, it sends a request to the controller, which records the change and shares it with the other brokers. In older clusters this map was kept with help from an external system called ZooKeeper. In modern clusters, Kafka uses its own built-in quorum called KRaft, and newer major releases run only in this mode.

How control plane trouble creates under-replication

There are several ways this happens. A leader may be unable to reach the controller, so its request to shrink or expand the in-sync set fails, which shows up in the failed ISR update metric. The controller itself may be overloaded, for example after a huge number of partitions changed state at once. The controller role may keep moving between brokers because of unstable sessions, which shows in the controller count metric flickering or briefly showing zero or more than one.

Session timeouts are also a factor. A broker keeps its place in the cluster by sending regular heartbeats. If a long pause, network problem, or overloaded machine stops these for longer than the allowed session time, the controller removes the broker, and every partition with a replica on it becomes under-replicated at once. The broker may then return moments later, having caused a noisy and unnecessary episode.

Signs that the control plane is the cause

Not every alert is a fault. Learning to recognize planned causes prevents wasted effort and false alarms.

Rolling restarts and upgrades

When a broker is restarted, its followers leave the in-sync sets while it is down, then catch up after it returns. Kafka asks a stopping broker to move its leadership away first, which is called controlled shutdown, so clients see little disruption. But under-replication is unavoidable until the broker has fully caught up. The safe rule is to wait until the count returns to zero before touching the next broker. Restarting a second broker while the first is still catching up can push some partitions below their minimum ISR or even offline.

Partition reassignment and cluster growth

Moving partitions to new brokers means creating new copies and copying all the historical data. While a new replica is catching up, the partition may be shown as having extra replicas or as being under-replicated, depending on the stage. Copying large volumes competes with normal traffic, so operators use throttles to limit the bandwidth used. If the throttle is set too high, clients suffer; if too low, the move takes very long and the catch-up may never finish before the next problem arrives.

Broker replacement

Replacing a failed machine with a fresh one that has an empty disk forces it to copy everything it should hold. This can involve terabytes and take hours. During that time, every partition it should follow is under-replicated. Reusing the identity of the old broker is simple, but the recovery load falls on the remaining brokers, so plan for it.

Preferred leader elections

After a broker returns, leadership is often moved back to the preferred replica, the first in each assignment list. This shifts traffic and may briefly cause short disturbances but does not itself cause lasting under-replication.

Modern clusters run on virtual machines and containers, which add their own limits and surprises.

Instance and volume limits

Cloud machines advertise network and disk performance, but many have a steady baseline and a higher burst level that lasts only while credits remain. Kafka runs steadily around the clock, so it can drain these credits and then drop suddenly to the baseline. This produces the classic story of a cluster that was fine for weeks and then began to lag at the same time every day or after a traffic increase. Check both instance network limits and storage volume throughput limits against actual usage.

Zones, racks, and cross-zone latency

Spreading replicas across availability zones protects against zone failures, and Kafka supports this by allowing each broker to declare its rack. This is highly recommended, but it means replication crosses zone boundaries and experiences greater latency and sometimes cost. A degraded inter-zone link affects replication for every partition that spans it. Diagnosis should include latency and loss measurements between zones, not only inside them.

Containers and orchestrators

In container platforms, brokers can be rescheduled onto new machines, throttled by processor limits, or killed for exceeding memory limits. A processor limit that is too low can stall threads in brief bursts even when average usage looks low, which causes tiny pauses that add up to lag. A rescheduled broker on a new node may also arrive with a cold page cache or with network attached storage that behaves differently. Stable identities, local fast storage, and reserved resources are the usual protection.

Here is a repeatable sequence you can follow calmly, even at three in the morning.

flowchart TD
    A["Missing broker found"] --> B{"Is the broker process running and registered?"}
    B -->|"No"| C["Find crash reason - memory, disk, host, configuration"]
    B -->|"Yes"| D{"Do its resource graphs differ from peers?"}
    D -->|"Yes"| E["Fix the resource - disk, network, pauses, threads"]
    D -->|"No"| F{"Is the controller or quorum healthy?"}
    F -->|"No"| G["Repair control plane"]
    F -->|"Yes"| H["Check network path and fetch errors between brokers"]
        

Capture evidence before you change anything

Every fix changes the system, and some fixes erase the clues. A restart clears thread dumps and memory state. A configuration change hides the previous behavior. Before acting, save a small bundle of evidence: a screenshot or export of the dashboards for the affected window, the list of affected partitions with their replica and in-sync lists, the relevant log sections from the suspect broker and the controller, and, when a process seems stuck, a thread dump and the latest garbage collection log.

This habit costs a few minutes and pays back many times. It allows a calm review the next day, lets a colleague or vendor look at the same facts, and often reveals a cause that the quick fix would have concealed. It also protects you from a common temptation: declaring victory because the count fell, without ever knowing what caused it.

Once the cause is known, the fix is usually straightforward. The hard part is choosing the right one and avoiding side effects.

Dampening the pain while you work

Sometimes the underlying fix takes hours, such as copying a large disk. In the meantime you can reduce risk. Move heavy producers to a lower rate if possible, add client quotas so noisy users do not crowd out replication, and make sure the remaining brokers are not pushed to their limit. If a partition is at its minimum ISR, avoid any planned work on the brokers holding its remaining replicas.

Confirm you fixed the cause, not just the number

A dropping count feels good, but the goal is understanding. Ask whether the cause is gone or only hiding. If a restart cleared a problem, ask what condition made the broker stuck, because it will return. If load simply fell overnight, ask what happens when the next peak arrives.

Some tempting shortcuts make the problem worse or hide it until it hurts more. Recognize them and refuse them.

Good alerts speak early, speak clearly, and stay quiet when nothing needs attention.

Layer your alerts by urgency

One alert for everything is a recipe for either constant noise or missed emergencies. A better approach uses layers, each with its own meaning and its own response. The lowest layer warns that redundancy is reduced. The middle layer warns that safety margins are almost gone. The top layer says customers are being affected right now.

Use a waiting period, but not too long

A restart or a brief network glitch causes a short rise that heals by itself. A waiting period of several minutes before warning avoids noise from these normal blips. But the waiting period for the critical layer should be very short, because the customer impact is already real.

Suppress expected noise carefully

During planned maintenance, temporarily lower the urgency of the warning layer, but never disable the critical layer. Tie suppression to a specific maintenance window with an automatic end, so that an alert is never left silent by accident.

Build a dashboard that tells a story

The best diagnosis is the one you never need, because the cluster was built and run with replication in mind.

Design choices that protect replication

Use a replication factor of three for important data, together with a minimum ISR of two. Spread replicas across racks or zones using rack awareness so that one failure domain cannot remove several copies at once. Keep the number of partitions per broker reasonable, because every partition adds replication work, metadata, files, and recovery time. Newer coordination modes allow much larger partition counts than older ones, but the per-broker cost of each partition never becomes zero.

Capacity planning that includes replication

When estimating capacity, count replication traffic honestly. Disk writes are the produce rate multiplied by the replication factor. Outbound network traffic is consumer reads plus the produce rate multiplied by the number of followers. Leave generous headroom, because the moments when a broker fails or is replaced are exactly the moments when the survivors must do extra work, and a cluster running near its limit in normal times cannot survive them.

Keep leadership balanced

Enable automatic preferred leader rebalancing or run a periodic rebalance, so no broker ends up leading far more than its share. Automated tools that watch load and propose partition moves, such as the open source Cruise Control project that originated at LinkedIn, can help keep disk, network, and leader counts even, but they should be used with sensible throttles and human review.

Watch for configuration drift and upgrade surprises

Clusters that run for years slowly collect differences. One broker has a slightly different thread setting, another has an older disk type, a third was tuned by hand during a past incident and never brought back in line. These differences make brokers behave unevenly under load, and the weakest one becomes the recurring source of lagging followers. Keep broker configuration in version control, compare brokers regularly, and treat any intentional difference as something that needs a written reason.

Upgrades deserve the same care. New versions can change defaults, replication behavior, and the way metadata is coordinated. Read the release notes for replication and controller changes, upgrade a test cluster first, and follow the documented order for rolling upgrades. During the upgrade window, expect brief under-replication for each restarted broker and keep the one-broker-at-a-time rule strictly.

Operate with discipline

These illustrative stories show how the clues combine. They describe common patterns and are not tied to any specific company.

flowchart LR
    A["Sudden start"] --> B["Look for an event: crash, deployment, network change"]
    C["Repeating pattern"] --> D["Look for a limit: pauses, burst credits, scheduled jobs"]
    E["Slow growth"] --> F["Look for capacity: disk, network, partitions per broker"]
    G["One location or one topic"] --> H["Look for shared path or hot data"]
        

Short, direct answers to the questions that come up most often.

No. Brief rises during restarts or planned moves are normal. What matters is whether the count returns to zero, whether any partition is under its minimum ISR, and whether you caused the rise on purpose.

Because the count is reported by the leader of each partition. Healthy leaders notice that a follower on some other broker has fallen out of sync. Look for the broker that is missing from the in-sync lists.

An under-replicated partition has a leader and some missing in-sync copies, so it still works. An offline partition has no leader at all, so it cannot serve reads or writes. Offline is far more serious.

Under-replicated only means fewer in-sync copies than assigned. Under minimum ISR means fewer than the safety threshold, and then acks=all producers are rejected. A partition can be under-replicated without being under the minimum.

Rarely from slowness, since followers easily reach the leader’s latest data. If it happens, suspect a truly unreachable or stuck follower, or a control plane problem.

Only after finding the real cause, and only modestly. Raising it hides trouble and lets slow followers hold back writes and the high watermark.

The broker may be performing log recovery or copying a large amount of missed data. Check its startup progress, replication lag, and any throttles that might be limiting catch-up.

Not by themselves. Consumers read committed data below the high watermark. However, if the leader fails while few copies remain, uncommitted or thinly replicated data is at risk, and reads may be delayed because the high watermark advances slowly.

Only if the cause is capacity, and only after partitions are moved onto the new brokers. New brokers alone hold no data. If the cause is a fault or a shared network problem, more brokers will not help.

Repeating times point to scheduled work or predictable load: batch jobs, compaction, backups, cloud burst credits running out, or a daily traffic peak. Compare the times with your schedules and resource limits.

More copies add durability but also cost storage, network, and write latency, and increase the number of things that can lag. Three is the common balance; go higher only for a clear reason.

Treat the healthy baseline as exactly zero under-replicated partitions outside planned work, alert on any lasting deviation, and always wait for zero before performing the next maintenance step.

An under-replicated partition is Kafka’s honest way of saying that some promised copies are missing or behind. It is a symptom that can come from a dead broker, a slow disk, a jammed network, a paused Java process, a stressed controller, an overloaded cluster, or simply a maintenance task in progress. The skill of diagnosis is not memorizing every cause. It is following a calm method: size the risk, find the common missing broker, classify the pattern by scope and timing, read the resource evidence, fix the cause instead of the number, and then improve your alerts so the next event is smaller.

Remember that the alert is always about a gap between promise and reality, and Kafka closes that gap only when followers can keep up again. Everything in this guide is a way of finding out what stopped them.