Before we meet the ISR, we need to understand the problem it was invented to solve.

Every computer eventually fails. Disks wear out, power supplies burn, network cables get unplugged, and whole data centers lose cooling. If a Kafka partition lived on only one server, then on the day that server died every message stored on it would become unavailable, and possibly be gone forever. For a company that moves payments, rides, or video events through Kafka, that is not acceptable.

So Kafka keeps several copies of each partition on different servers. Each copy is called a replica. The number of copies is called the replication factor. In production, a replication factor of 3 is the most common choice: one copy can be lost, and two more remain.

Copies create a new problem: agreement

Making copies sounds easy, but keeping them identical is the hard part. When a producer sends a message, which copy receives it first? When is it safe to tell the producer “your message is stored”? What if one copy is slow, or unreachable, or half-asleep because its disk is struggling? And if the server holding the main copy dies, which of the remaining copies takes over, and how do we know it did not miss anything?

These questions are exactly what the ISR answers. To see why Kafka’s answer is clever, look at the three obvious designs that came before it.

The rest of this article unpacks that one sentence: “wait for everyone who is keeping up.” Each word in it hides a precise rule, and understanding those rules is what separates people who use Kafka from people who can run it safely at scale.

A short tour of the parts of Kafka that the ISR lives inside. Skip ahead if you already know them well.

Brokers, topics, and partitions

A broker is one Kafka server. A group of brokers working together is a cluster. Data in Kafka is organized into topics, which are named streams such as orders or ride-locations. Because one machine cannot hold or serve an entire busy topic, each topic is split into partitions. A partition is an ordered, append-only list of messages. Each message in it receives a number called an offset, starting at 0 and always increasing.

Replicas, leaders, and followers

Replication happens per partition, not per topic and not per broker. If the topic orders has 12 partitions and a replication factor of 3, then there are 36 replicas spread over the cluster. For every partition, Kafka picks one replica as the leader. The other replicas are followers. Producers send messages to the leader. Consumers normally read from the leader. Followers do one job only: they keep asking the leader for new messages and append them to their own copy.

The full list of brokers that hold a copy of a partition is called the assigned replicas. The order of that list matters. The first broker in the list is the preferred leader, the one Kafka would like to be leader when everything is healthy. We will return to this when we talk about balancing.

The controller and cluster metadata

Someone has to keep the official record of who leads what and who is in each ISR. That job belongs to the controller. In modern Kafka, the controller role is played by a small group of nodes running Kafka’s own Raft-based consensus system, known as KRaft. Every change to leaders and ISRs is written to a replicated metadata log that all brokers read. Older Kafka versions stored this information in Apache ZooKeeper instead, and one broker acted as the controller. Kafka 4.0 removed ZooKeeper support entirely, so KRaft is the world you will meet in new clusters. The ISR ideas in this article are the same in both worlds; only the place where the record is stored differs.

flowchart LR
    PR["Producer applications"] -->|"send messages to the leader"| B1
    subgraph CL["Kafka cluster"]
        B1["Broker 1: LEADER of orders-0"]
        B2["Broker 2: FOLLOWER of orders-0"]
        B3["Broker 3: FOLLOWER of orders-0"]
        CTRL["Controller quorum: stores leader and ISR records"]
    end
    B2 -->|"fetch new messages"| B1
    B3 -->|"fetch new messages"| B1
    B1 -->|"serve reads up to the high watermark"| CO["Consumer applications"]
    CTRL -.->|"metadata updates"| B1
    CTRL -.->|"metadata updates"| B2
    CTRL -.->|"metadata updates"| B3

Notice the direction of the arrows between followers and the leader. The leader does not push data to followers. Followers pull it, using the same kind of fetch request that consumers use. This is a deliberate choice, and it will explain a lot of behavior later, from how the leader learns about follower progress to why slow followers do not overwhelm a busy leader.

A small glossary

The definition is short. The consequences fill the rest of this article.

The In-Sync Replica set of a partition is the set of replicas, always including the current leader, that are sufficiently caught up with the leader. “Sufficiently caught up” has a precise meaning that we will define in a moment. What matters first is what the ISR is used for. It is used for three decisions:

• When is a message committed? A message counts as committed once every replica in the ISR has stored it.
• Who may become the next leader? Under safe settings, only a member of the ISR may be elected.
• How many copies must be alive to accept writes? The producer setting acks=all together with the topic setting min.insync.replicas is checked against the current ISR size.

Three groups of replicas

It helps to think of a partition’s replicas in three overlapping groups:

The ISR is always a subset of the assigned replicas. The assigned list changes only when someone moves the partition or changes its replication factor. The ISR, by contrast, changes constantly and automatically in response to the health of the brokers.

Defining “keeping up”

Early Kafka versions used two rules: a follower was out of sync if it was behind by more than a set number of messages, or if it had not contacted the leader for a set time. The message-count rule turned out to be a bad judge. During a sudden burst of traffic, healthy followers were briefly many messages behind and were wrongly thrown out, then let back in, over and over. The message-count rule was removed in Kafka 0.9.

Today there is a single, time-based rule controlled by the broker setting replica.lag.time.max.ms, which defaults to 30,000 milliseconds. In everyday words: a follower stays in the ISR as long as, within every 30-second window, it has managed to fetch everything the leader had at some recent moment. If a follower goes 30 seconds without being caught up, the leader removes it from the ISR.

Read that carefully, because it contains a subtle and very useful idea. The follower does not need to be perfectly level with the leader at every instant. That would be impossible when producers are constantly writing. It only needs to show that it is keeping pace: that its last fetch reached the point where the leader’s log ended at some recent moment. A follower that is always a few milliseconds behind but never falls further behind is perfectly healthy.

Why a clock is a better judge than a counter

To see why the message-count rule failed, picture a topic receiving 50,000 messages per second. A limit of, say, 4,000 messages sounds generous, yet at that rate it represents only eighty milliseconds of traffic. A healthy follower that pauses for a blink to flush its disk would instantly break the limit, be evicted, and then be readmitted a moment later. Now picture the opposite: a quiet topic receiving three messages an hour. A follower could be frozen for a full day and still never be more than a handful of messages behind, so a counter would never notice it had died.

A clock has neither problem. It asks one question that works at any traffic level: “Has this follower shown, within the last 30 seconds, that it can reach the end of the leader’s log?” Busy topic or sleepy topic, a follower that answers yes is useful, and a follower that answers no is not. This is why the time-based rule survived and the counting rule was retired.

A worked example

Take a partition named orders-0 with a replication factor of 3, assigned to brokers 1, 2, and 3, where broker 1 is the leader. Watch how the three groups change as trouble arrives and passes:

Notice that in the last row, the assigned list still names broker 1. Kafka does not forget that broker 1 is supposed to hold a copy. It simply records that broker 1 is not currently in sync, and it will bring it back into the ISR once it restarts and catches up.

Two position markers, and one rule about who can see what, sit at the heart of how the ISR is used.

Log End Offset (LEO)

Every replica, leader or follower, keeps its own log on its own disk. The Log End Offset, or LEO, is the offset that the next new message will receive in that replica. If a replica holds messages numbered 0 through 9, its LEO is 10. In other words, the LEO points just past the last message that replica has stored. Each replica has its own LEO, and they differ whenever followers are behind.

High Watermark (HW)

The High Watermark, or HW, is a single number per partition kept by the leader. It is the smallest LEO among all the replicas in the ISR. Every message with an offset below the high watermark has been stored by every in-sync replica. Those messages are called committed.

Why consumers stop at the high watermark

Here is the rule that makes the whole design safe: consumers can only read messages below the high watermark. Even though the leader already has offsets 8 and 9 on its disk, it does not show them to consumers.

The reason is protection against a rewind. Suppose a consumer could read offset 9, and then the leader crashed before any follower copied it. A follower would become the new leader, and offset 9 would not exist anymore. The consumer would have acted on a message that history now says never happened. By exposing only committed messages, Kafka guarantees that anything a consumer has seen will survive a leader failure, as long as at least one in-sync replica survives.

How the leader learns each follower’s position

Because followers pull, the leader learns their progress from their requests. Each fetch request contains the offset the follower wants next. If a follower asks for offset 8, the leader can infer that the follower already has everything below 8, so that follower’s LEO is 8. The leader records this for every follower, recalculates the high watermark as the minimum, and if the high watermark moved forward, it releases any waiting producers and makes the newly committed messages visible to consumers.

The leader then tells the followers the new high watermark by attaching it to its replies to their next fetch requests. This detail, that followers learn the high watermark one round trip later than the leader, turns out to matter greatly during failures, and we will meet it again in the chapter about leader epochs.

Two neighbors you may also hear about

Every few seconds, by default every five, each broker also writes its current high watermarks to a small checkpoint file on disk. When a broker restarts, this file tells it how much of its own log it can still trust. That habit is the seed of a real problem, which we will solve in Chapter 8.

Let us follow one message from the producer to the moment it is safely committed. Every idea from the last four chapters plays a part.

Assume the setup from before: partition orders-0, three replicas, broker 1 as leader, all three brokers in the ISR, and a producer using acks=all. That setting tells the leader, “do not answer me until every in-sync replica has this message.”

sequenceDiagram
    participant P as Producer
    participant L as Leader Broker 1
    participant F2 as Follower Broker 2
    participant F3 as Follower Broker 3
    P->>L: Produce request with acks all
    L->>L: Append to local log and move the LEO forward
    Note over L: The request now waits in a holding area
    F2->>L: Fetch request asking for the next offset
    L-->>F2: New messages plus the current high watermark
    F3->>L: Fetch request asking for the next offset
    L-->>F3: New messages plus the current high watermark
    F2->>L: Next fetch reports its new LEO
    F3->>L: Next fetch reports its new LEO
    Note over L: Every ISR member has the message, so the high watermark moves forward
    L-->>P: Acknowledgement of success

The journey, one step at a time

Why pulling is a smart choice

Making followers pull instead of having the leader push brings several benefits at once. First, each follower controls its own speed, so a slow follower simply fetches less often and never overloads the leader or blocks its siblings. Second, followers naturally send requests in batches, which improves throughput. Third, the leader learns each follower’s position for free from the request itself. Fourth, the leader can hand followers the exact bytes already stored on its disk, including compressed batches, without decompressing or re-encoding anything. On Linux, Kafka can even hand these bytes from the page cache to the network card without copying them through the application, a trick called zero-copy.

What the latency really contains

With acks=all, the time a producer waits is roughly the leader’s append time, plus one fetch cycle by the slowest in-sync follower, plus the network trips. In a well-tuned cluster inside one data center this is typically single-digit to low double-digit milliseconds. If followers sit in different availability zones, add a couple of milliseconds for the extra distance. This is the price of safety, and it is a small one. The important consequence is that the slowest member of the ISR sets the pace for committed writes. That is precisely why Kafka removes replicas that fall too far behind, the topic of the next chapter.

What consumers experience in the meantime

While all this happens, consumers are sending fetch requests to the same leader. A consumer asking for the newest data receives messages only up to the high watermark, and if nothing new has been committed, the leader holds the request for a short while, exactly as it does for followers, and answers when the watermark moves. So the very moment a producer receives its acknowledgement, consumers waiting at the end of the log can be handed that message. From the outside, the pipeline from producer to consumer is only as fast as replication allows. This is one of the reasons a clean, fast ISR contributes directly to end-to-end latency, not just to durability.

There is a useful consequence for testing. If you measure the time between sending a message and receiving it in a consumer, and you see that time jump whenever a broker misbehaves, the cause is often a follower that is holding back the high watermark, not a slow consumer or a slow network to the consumer.

The ISR is a living list. Replicas leave when they fall behind and return when they recover. Here is exactly how.

stateDiagram-v2
    [*] --> Assigned : replica is created on a broker
    Assigned --> CatchingUp : starts fetching from the leader
    CatchingUp --> InSync : reaches the high watermark
    InSync --> OutOfSync : no catch-up within the lag time limit
    OutOfSync --> CatchingUp : broker healthy again and fetching
    InSync --> [*] : partition moved away by reassignment

How a replica gets removed (shrinking)

How a replica comes back (expanding)

When the troubled broker recovers, its replica starts fetching again. It first reconciles its log with the leader (Chapter 8 explains how), then copies everything it missed. Once its LEO reaches the leader’s high watermark, the leader considers it caught up and proposes the larger ISR to the controller. When the controller approves, the leader logs “Expanding ISR from 1,2 to 1,2,3” and the replica counts again toward commits.

Two details deserve attention. First, the follower only has to reach the high watermark, not the leader’s current end of log, because the leader is constantly writing and a moving target would be impossible to hit. Second, while the leader waits for the controller’s answer, it uses the more conservative view of the ISR for safety decisions, so the risky guess can never make committed data look safer than it is.

The settings that control membership

A worked example: the garbage-collection pause

Kafka brokers run on the Java virtual machine, which periodically pauses the program to clean up memory. Usually these pauses last a few milliseconds. Now imagine a broker with a badly sized heap suffering a pause of 40 seconds. During that time it sends no fetch requests, sends no heartbeats, and answers nobody. Because no heartbeats arrive, the controller fences the broker after about nine seconds and removes it from every ISR it belonged to. (Had only its fetching stalled while heartbeats kept flowing, the leaders would have removed it at the 30-second mark instead.) When the pause ends, the broker wakes up believing everything is normal. It learns from the controller that its state is stale, truncates its logs against the new leader epochs, and starts catching up. To the outside world the incident looks like a short blip of write latency followed by a burst of ISR expansions. Understanding this sequence explains why experienced operators keep heaps modest and let the operating system’s page cache do the heavy lifting.

When the ISR flaps

Sometimes a follower is thrown out, comes back, and is thrown out again in a loop. This is called flapping, and it usually signals a broker running near its limits: disks saturated by a big reassignment, network links full, memory pressure causing long garbage-collection pauses, or too few fetcher threads for the number of partitions. The fix is almost never to remove the safety net. It is to give the follower enough resources, spread load more evenly, or slow down a heavy background job. Raising replica.lag.time.max.ms modestly can stop harmless flapping from short pauses, but a large value makes real failures take longer to detect, and every second of delay is a second of possible frozen writes.

When a leader disappears, the ISR is the list of candidates who are guaranteed to hold every committed message.

The invariant that makes it safe

A message is only acknowledged and shown to consumers when every replica in the ISR has stored it. Therefore, at every moment, every member of the ISR holds every committed message. If the controller always picks the next leader from the ISR, the new leader is guaranteed to have all committed data. Nothing that was promised to a producer or shown to a consumer can be lost by a normal leader change. Everything else in Kafka’s replication design is machinery to preserve this one sentence.

flowchart TD
    A["Controller notices the leader is gone"] --> B{"Is any live replica in the ISR?"}
    B -->|"Yes"| C["Choose the first live ISR member in the assigned list"]
    C --> D["Increase the leader epoch and record the new leader and ISR"]
    D --> E["Clients refresh metadata and continue"]
    B -->|"No"| F{"Is unclean leader election enabled?"}
    F -->|"No"| G["Partition stays offline until an ISR member returns"]
    F -->|"Yes"| H["Choose any live out-of-sync replica"]
    H --> I["Committed messages may be lost"]
    I --> D

How the failure is noticed

In KRaft mode, every broker sends a heartbeat to the controller every couple of seconds. If heartbeats stop for longer than broker.session.timeout.ms, which defaults to 9 seconds, the controller declares the broker fenced. It removes that broker from every ISR it belonged to, and for every partition the broker was leading it elects a new leader from the remaining ISR members. Older ZooKeeper-based clusters used session expiry in ZooKeeper for the same purpose.

The kinds of election

Graceful versus sudden

When a broker is stopped politely, it tells the controller first. The controller moves leadership to other ISR members one partition at a time while the broker is still alive, so clients see almost no interruption. When a broker crashes, the same election happens only after the failure is detected, which takes several seconds. This is why rolling restarts should always use the graceful path, and why abrupt process kills in production deserve a postmortem.

Balance: the preferred leader

After a failure, leadership shifts to surviving brokers. When the failed broker returns and rejoins the ISR, it does not automatically take its leaders back. Left alone, the cluster drifts into an uneven state where one or two brokers carry most of the leadership and therefore most of the traffic. Kafka fixes this with automatic preferred-leader rebalancing, controlled by auto.leader.rebalance.enable (on by default). Every five minutes, the controller checks each broker, and if more than 10 percent of the partitions that should be led there are led elsewhere, it moves leadership back, but only to replicas that are currently in the ISR.

How the controller records these decisions

In a KRaft cluster, the controller nodes maintain a replicated metadata log. Every change to a partition is one record in that log, carrying the new leader, the leader epoch, the new ISR list, and a partition epoch, a version number that increases with every change to that partition. Brokers read the log and keep an up-to-date copy of the whole map of the cluster in memory.

The partition epoch is what makes the process strict. When a leader asks to change its ISR, it must quote the leader epoch and partition epoch it believes are current. If either number is out of date, the controller rejects the request and the leader must refresh its view. This is a compare-and-swap pattern, the same idea as saving a shared document only if nobody else has saved a newer version since you opened it. It ensures two parties can never overwrite each other’s decisions by accident.

Because the metadata log itself is replicated by majority vote among the controller nodes, a decision that has been recorded survives the failure of any single controller. If the active controller crashes, the remaining nodes elect a new one, typically within a second or a few seconds, and the new one already holds the entire log. Older designs had to reload the state of every partition from ZooKeeper after a controller change, which took noticeably longer on clusters with very many partitions. That difference is one reason large clusters recover faster on KRaft.

The zombie leader problem

Consider a leader that becomes cut off from the controller but is still connected to some producers. The controller, hearing nothing, elects a new leader. The old leader does not know yet and continues to accept writes. Does this cause corruption? With acks=all, no. The old leader’s followers have moved on and no longer fetch from it, so it cannot get acknowledgements from them. Its attempt to shrink the ISR down to itself is rejected by the controller, because its epoch is stale. It cannot acknowledge anything. Once it learns about the new epoch, it steps down, truncates any unreplicated tail of its log, and becomes a follower. Producers using acks=1 in the same situation may receive acknowledgements for messages that are later discarded. This is one of the clearest demonstrations of why acks=all matters.

Unclean election: the last resort

What if every ISR member is dead? The partition has no leader who is guaranteed to have the committed data. Kafka gives you a choice, set by unclean.leader.election.enable, which is off by default. If it is off, the partition stays unavailable until an ISR member comes back. If it is on, Kafka promotes an out-of-date replica so that service resumes, accepting that messages the old ISR had committed may be lost, and that consumers may see offsets jump backward or messages reappear.

Electing a new leader is only half the story. The other replicas must also fix their logs so that everyone tells the same history.

Why logs can disagree after a failure

When a follower restarts or becomes a follower of a new leader, its log might contain messages the new leader never had, for example messages the old leader wrote to it just before crashing. Those uncommitted extras must be removed, a step called truncation, so that the follower’s log becomes a true prefix of the leader’s log. Then it can resume fetching normally.

The question is where to truncate. In early Kafka versions, the answer was: cut back to the follower’s own high watermark. This sounded reasonable but had a hidden flaw. Remember that a follower learns the high watermark one fetch round later than the leader does. So a follower’s idea of the high watermark can lag behind the truth.

Two things that went wrong in the early design

These problems were fixed in Kafka 0.11 by KIP-101, and a related fast-failover corner case was closed by KIP-279 in Kafka 2.0. The solution introduced a small, powerful idea: the leader epoch.

The leader epoch

A leader epoch is simply a number that goes up by one every time a new leader is elected for a partition. Think of it as the generation of leadership. Each leader remembers, in a small checkpoint file, a table of the form “epoch 3 began at offset 1,204.” Every message batch written to the log is stamped with the epoch of the leader that wrote it.

How a follower uses it

Applied to the two failure stories above, the follower in story A would ask the leader where its epoch ended and discover that m2 belongs to the shared history, so it would keep m2. In story B the returning broker would find that its epoch ended at offset 1 according to the new leader, so it would discard its own m2 and fetch the new m3. Both problems disappear.

What lives on disk

To do all of this, every broker keeps a small set of files beside the message data. Knowing them helps when you read a broker’s data directory during an incident.

When a broker shuts down cleanly, it leaves a marker so that the next start can skip the expensive checking work. After a hard crash there is no marker, so the broker walks through the segments after the recovery point, validates them, rebuilds indexes, and only then joins as a follower and starts its epoch-based truncation. That is why a broker that crashed can take noticeably longer to return than one that was stopped politely, and why very large partitions increase recovery time.

The ISR tells Kafka which copies are healthy. Three settings decide how much you rely on them.

The three levels of acknowledgement

The producer setting acks controls when the leader tells the producer “success.”

A hidden trap: the ISR can shrink to one

Here is a subtle question. With acks=all, the leader waits for “all the replicas in the ISR.” But what if the ISR has shrunk until it contains only the leader? Then “all” means “just the leader,” and acks=all silently behaves like acks=1. The message would be acknowledged after being written in only one place.

To close this trap, Kafka has the topic and broker setting min.insync.replicas. It defines the smallest ISR size for which the leader will accept an acks=all write. If the ISR falls below this number, the leader rejects new writes with a “not enough replicas” error. If the ISR shrinks after the message was appended but before it could be fully replicated, the producer receives a related error, “not enough replicas after append.” Both errors are retriable, so a well-configured producer keeps trying until the cluster recovers or the delivery timeout expires.

The famous trio: 3, 2, all

The most widely recommended production baseline is a replication factor of 3, min.insync.replicas=2, and acks=all. Let us see why it is a sweet spot:

• Every acknowledged message exists on at least two brokers, so any single broker can fail without losing it.
• If one broker is down, the ISR has two members, which still meets the minimum, so writes continue without interruption.
• If two brokers are down, the ISR has one member, below the minimum, so writes are refused. Kafka chooses to stop accepting data it cannot protect rather than pretend.

Choosing your numbers

Two simple rules sum up the table. Writes stay open as long as the number of failures is no larger than the replication factor minus min.insync.replicas. And acknowledged data survives as long as fewer than min.insync.replicas brokers holding it fail at the same moment.

Retries, duplicates, and ordering

When a leader changes or the ISR is too small, produce requests fail temporarily, and producers retry. Retries could create duplicates if the first attempt actually succeeded but the acknowledgement was lost. Since Kafka 3.0 the producer enables idempotence by default. Each producer gets an identity and numbers its messages per partition, and the leader discards anything it has already seen. Combined with acks=all, this gives safe retries with no duplicates and preserved order within a partition. The overall time limit for a send, including retries, is delivery.timeout.ms, two minutes by default, and it should comfortably exceed the time a leader election or an ISR recovery takes.

Transactions and the ISR

Kafka transactions let a producer write to several partitions and have all of the writes appear together, or not at all. They rely on the same replicated foundations. The special transaction log is itself a replicated topic, and markers that say “this transaction committed” or “aborted” are written into the data partitions like ordinary messages, so they must be replicated to the ISR before consumers can act on them. Consumers reading in the committed mode stop at the last stable offset, which can never pass the high watermark. The practical lesson is that weak ISR settings weaken transactional guarantees as well. A transaction is only as durable as the replication underneath it.

Internal topics deserve care too

Kafka stores consumer positions and transaction state in its own internal topics. Their replication is governed by separate settings, such as offsets.topic.replication.factor and transaction.state.log.min.isr, whose defaults are already three and two. On small test clusters people sometimes lower them to make things start; forgetting to raise them again for production leaves the cluster’s memory of who read what protected by a single copy.

Every design gives something up. Here is what the ISR approach buys and what it costs.

Where ISR sits in the CAP trade-off

The CAP theorem says that when a network splits a system into groups that cannot talk to each other, you must choose between staying consistent (never showing wrong or missing data) and staying available (continuing to answer). With its default settings, Kafka chooses consistency. If too few in-sync replicas exist, it refuses writes, and if no in-sync replica exists, it keeps the partition offline. If you enable unclean leader election or set min.insync.replicas=1, you lean toward availability instead. The ISR model lets you pick your point on that line, topic by topic.

ISR compared with majority voting

The saving matters at Kafka’s scale. To tolerate two simultaneous failures, a quorum system needs five copies of every message, while the ISR approach can get by with three, storing and sending 40 percent less data. When you move petabytes a day, that difference is a large cloud bill. The price is the sensitivity to slow ISR members described earlier, which the lag limit bounds but does not eliminate. Interestingly, Kafka’s own metadata layer, KRaft, uses a true majority-vote protocol, because the metadata log is small and needs the strictest guarantees. Data partitions use ISR because they are big and need to be efficient.

Counting the traffic

Imagine producers sending one gigabyte per second into a topic with a replication factor of 3. The cluster must absorb that gigabyte from producers, then the leaders forward it to two followers each, so an extra two gigabytes per second of replication traffic moves between brokers. Disk writes across the cluster total three gigabytes per second, because every byte is stored three times. If two consumer groups read the data, another two gigabytes per second leaves the cluster. Spread across 12 brokers, each broker receives about 250 megabytes per second and sends about 333 megabytes per second, which fits inside a 10-gigabit network card with room to spare. This kind of arithmetic is how architects decide how many brokers a workload needs.

Tail latency and the slowest replica

Average produce latency can look wonderful while the slowest requests are terrible. The reason is that every acks=all write waits for the slowest current member of the ISR, so the more members there are, the more chances that one of them is having a bad moment. Imagine each follower is slow for one request in a hundred. With one follower, roughly one write in a hundred is slow. With six followers, the chance that at least one of them is slow on any given write rises to about six in a hundred. This effect, common to all systems that wait for several helpers, is one reason teams do not simply add more and more replicas. A replication factor of 3 gives strong protection while keeping the tail latency manageable, and going higher should be a conscious decision for special data.

Let us walk through the most common trouble spots, assuming a replication factor of 3, min.insync.replicas=2, producers using acks=all, and unclean election disabled.

flowchart TB
    subgraph AZA["Availability Zone A"]
        BA["Broker 1: leader of orders-0"]
        KA["Controller node 1"]
    end
    subgraph AZB["Availability Zone B"]
        BB["Broker 2: follower of orders-0"]
        KB["Controller node 2"]
    end
    subgraph AZC["Availability Zone C"]
        BC["Broker 3: follower of orders-0"]
        KC["Controller node 3"]
    end
    BB -->|"fetch"| BA
    BC -->|"fetch"| BA
    KA --- KB
    KB --- KC
    KA --- KC

A timeline of a leader crash

The exact delay before a crash is detected depends on your session timeout settings. Graceful shutdown skips the detection wait entirely, which is why it is so much smoother.

Replication is the largest hidden consumer of resources in most Kafka clusters. Understanding it prevents surprises.

How replication scales

Followers behave like special consumers, and a broker that follows many partitions runs fetcher threads that pull from each broker that leads them. The setting num.replica.fetchers controls how many threads work per source broker. On brokers with many partitions, one thread can become a bottleneck, causing followers to fall behind even though the hardware is idle. Raising this value spreads partitions across more threads. A broker’s ability to keep followers up to date is also limited by its network, its disks, and the share of CPU spent on TLS encryption and compression handling.

Balancing leaders and replicas

Leaders do more work than followers, since they serve producers and consumers. An uneven leader distribution creates hot brokers. Automatic preferred-leader rebalancing helps, but for bigger imbalances, tools such as LinkedIn’s open-source Cruise Control can plan and execute moves that even out disk, network, and leader counts across brokers.

Moving data safely

When you add a broker or drain one, partitions are reassigned. A new replica is created, starts as out-of-sync, copies the whole log from the leader, and joins the ISR when it reaches the high watermark. That copying can saturate the network and cause the very ISR shrinks you want to avoid. Kafka lets you throttle reassignment traffic with replication throttle settings on leaders and followers, so that moves finish gradually without starving normal work.

Batching, compression, and replication

Producers send messages in batches, and brokers store and forward those batches as they arrive. A leader does not open a compressed batch to look inside it, and followers copy the same bytes. This is why compression saves not only disk space but also replication bandwidth: a batch compressed to a third of its size costs a third as much to send to each follower. Larger batches also mean fewer requests and fewer fetch round trips, which helps followers keep up. The trade-off is that a producer waiting to fill a large batch adds a little delay before the message is even sent, so teams tune the waiting time and batch size together. When followers struggle to keep up on a busy cluster, checking that producers use efficient compression and reasonable batch sizes is often cheaper than buying more hardware.

Rack and zone awareness

Each broker can declare its location with the broker.rack setting. Kafka then spreads a partition’s replicas across different racks or availability zones whenever it creates or reassigns partitions. This turns “the ISR survives a broker failure” into “the ISR survives the loss of an entire zone.” In the cloud, the cost is that replication traffic crosses zone boundaries and is billed per gigabyte for self-managed clusters. Since Kafka 2.4, a feature called follower fetching (KIP-392) also lets consumers read from a follower in their own zone, which cuts consumer cross-zone traffic while still respecting the high watermark, so consumers never see uncommitted data.

Latency versus safety knobs

Replication is a security topic because durability settings are also safety settings, and because every replica is another copy of your data.

Protect the replication traffic

Followers fetch data from leaders over the network, using the same protocol as consumers. If that traffic is unencrypted, anyone who can watch the network between brokers can read every message. Production clusters therefore secure the inter-broker listener with TLS encryption, and authenticate brokers to each other with mutual TLS or SASL. The controller nodes should have their own protected listener, since they hold the authoritative record of who leads what and who is in every ISR.

Kafka’s authorization system also has a special permission for this. A follower needs the ClusterAction permission on the cluster to perform replica fetches. Only broker identities should hold it. If an ordinary application account had it, that account could impersonate a follower and pull data from any partition.

Guard the durability settings

Suppose someone with the right to change topic configurations lowers min.insync.replicas to 1, or switches on unclean leader election, to “fix” a noisy alert. Nothing breaks immediately. The cluster simply becomes quietly capable of losing acknowledged data. That is why these settings deserve the same protection as a firewall rule.

Protect the cluster from noisy clients

A single misbehaving producer that floods a leader with data can starve the fetch requests from followers. Followers fall out of the ISR, and suddenly the whole partition is at risk because of one client. Kafka’s client quotas let you cap bytes per second per client or per user, so that the ISR is protected from overload. Similarly, replication throttles cap the traffic of background moves.

Remember that every replica is a copy of the data

• Encrypt broker disks or volumes, since a replication factor of 3 means three physical copies of sensitive data.
• Apply retention and deletion policies with all replicas in mind. Data expires on each replica through the same rules, but old copies can linger on disks that are detached or backed up.
• Restrict the shell and file-system access on broker machines, because the log files are readable directly from disk.

The ISR is the heartbeat of a partition. These are the instruments that show whether it is beating steadily.

The metrics that matter most

Two of these deserve extra attention. UnderMinIsr is the metric that tells you producers are being rejected right now, so it should page a human. AtMinIsr is the early warning that you are one failure away from that state, and it often reveals problems a day before they become incidents. On the producer side, watch the error rate, the retry rate, and any “not enough replicas” errors, since applications feel ISR trouble first.

What the logs say

Brokers write clear messages whenever an ISR changes. In the leader’s server log you will see lines of the form “Shrinking ISR from 1,2,3 to 1,2” and “Expanding ISR from 1,2 to 1,2,3,” along with the partition and reason. Controller logs record elections and fencing decisions. Send these logs to a central system, and build a view that answers three questions: which broker was removed, how long did it stay out, and how often has it happened this week? A single broker that appears in every shrink message is your problem broker.

Signals from the client side

The producers and consumers using the cluster are also instruments. Producer clients expose their own metrics for request latency, retry rate, and error rate. A rise in retries with “not enough replicas” or “not the leader” errors is often the first sign that an ISR or an election is in trouble, sometimes minutes before broker-side alerts catch up. Consumer lag can tell a similar story: if lag on a group suddenly grows across many partitions that share the same leader broker, the cause may be a replication problem on that broker rather than a slow consumer. Correlating client metrics with broker metrics in one dashboard shortens investigations enormously.

A simple runbook for an alert

Habits that experienced teams share, and traps that repeatedly cause outages.

Patterns that work

Anti-patterns to avoid

A compact checklist you can print, followed by the mistakes that surprise even experienced teams.

Mistakes that surprise people

• Expecting client code to manage the ISR. Applications only need to know the leader. The ISR is a broker-side concern, and clients experience it only through results: successes, delays, or retriable errors.
• Expecting a write to fail the instant a follower dies. The dead follower stays in the ISR until the lag limit passes, so writes may stall briefly first.
• Forgetting about the ISR when replacing hardware. A new broker with the same identity but an empty disk must copy everything before it rejoins, and while it does, your effective redundancy is lower.
• Testing with tiny messages only. Replication behavior and lag look very different with large messages and compression on. Test with realistic sizes.
• Not testing leadership changes under load. Failover works in a quiet lab and misbehaves in production traffic when timeouts are set too tightly.

How well-known companies and platforms lean on the ideas in this article.

Starting points by workload

The table below shows illustrative starting points, not universal rules. Adjust them to your own tolerance for loss and delay.

Short answers to the questions people ask most often.

If you remember only one page of this article, make it this one.

The ISR is Kafka’s answer to a hard question: how do you keep several copies of a growing log identical and available when machines fail all the time? Rather than waiting for everyone, or voting on every message, Kafka waits for the replicas that are provably keeping up, and it keeps a strict list of who they are. From that one idea come the high watermark, the rules for electing leaders, the epochs that keep logs identical, and the three settings that decide your durability: the replication factor, min.insync.replicas, and acks.