Start with the plain-English idea. The technical details come in the next chapters.

Apache Kafka keeps several copies of every piece of data. One copy is the boss, called the leader. The other copies are helpers, called followers. All new messages go to the leader first, and the followers copy them from the leader. When the leader disappears because its machine crashed, lost power or lost its network, Kafka must quickly pick a new leader from the followers. Choosing the new leader is called a leader election.

Most of the time Kafka is picky. It only promotes a follower that is fully caught up with the leader. That is a clean leader election, because the new leader is guaranteed to hold every message that was confirmed to the sender. An unclean leader election happens when Kafka has no caught-up follower left and is allowed to promote a follower that is behind. The word “unclean” is a warning label. The new leader’s copy is not clean, because it may be missing messages that Kafka had already promised were safe.

Rule A is a clean election. Rule B is an unclean election. Kafka lets you pick the rule with a single setting called unclean.leader.election.enable. When it is false, Kafka follows Rule A and simply keeps the partition offline until a trustworthy copy returns. When it is true, Kafka follows Rule B. The partition comes back fast, but some confirmed messages may be lost forever.

Where the name and the default come from

In the early years of Kafka the setting was switched on by default. The thinking at the time was that people used Kafka mainly to move logs and activity events, where staying up mattered more than keeping every single record. As Kafka grew into a system that carries payments, orders and financial events, that default became risky. Starting with version 0.11.0, released in 2017, the default flipped to false. Kafka now protects your data first and your uptime second, unless you tell it otherwise.

A common myth to clear up early

Many people believe that turning the setting on makes Kafka run unclean elections all the time. It does not. The setting is only a permission. Kafka will use that permission in one very narrow situation: the leader is gone and no caught-up follower is alive. On a healthy cluster with good replication, this situation may never happen, and the setting will sit unused. The danger is not in the permission itself. The danger is in what the permission allows on the worst day of the year.

A quick tour of the moving parts. If you already know Kafka well, skim the cards and jump to the diagram.

Kafka is a system for moving streams of events between programs. An event is a small record that says something happened, such as “user 42 placed order 9001” or “sensor 7 reported 21 degrees”. Programs that create events are called producers. Programs that read events are called consumers. In the middle sits a group of servers called a cluster, and each server in it is a broker. Kafka was created at LinkedIn to carry activity data such as page views and clicks. It became an Apache project, and LinkedIn has publicly described its own clusters as handling trillions of messages per day.

The partition is an append-only log

Inside a broker, a partition is stored as a log: a file that only grows at the end. New events are added at the tail. Old events are never edited. This simple design is the secret of Kafka’s speed. Writing to the end of a file is very fast on ordinary disks, and reading a range of events in order is fast too. Old data is removed later by a retention rule, for example “keep seven days” or “keep 500 gigabytes”, or compacted so that only the latest value for each key remains.

Ordering is guaranteed inside one partition, but not across partitions. If two events must stay in order, they must go to the same partition. Producers usually achieve this by attaching a key, such as the customer ID, so that every event for the same customer lands in the same partition.

flowchart LR
  P1["Producer App A"] --> B1
  P2["Producer App B"] --> B1
  subgraph CL["Kafka Cluster"]
    B1["Broker 1
Leader of Partition 0"]
    B2["Broker 2
Follower of Partition 0"]
    B3["Broker 3
Follower of Partition 0"]
    CT["Controller
decides who leads"]
    B1 -->|"replicates"| B2
    B1 -->|"replicates"| B3
    CT -.->|"assigns roles"| B1
    CT -.->|"assigns roles"| B2
    CT -.->|"assigns roles"| B3
  end
  B1 --> C1["Consumer Group X"]
  B1 --> C2["Consumer Group Y"]

Figure 1 shows a single partition, but a real topic has many partitions, and each partition has its own leader. Broker 1 may lead Partition 0 while Broker 2 leads Partition 1. This spreads the load. It also means that when one broker dies, only the partitions it was leading need new leaders. The other partitions keep running without a pause. Kafka clients do not need a load balancer in front of the brokers. They ask any broker for the cluster map, learn which broker leads which partition, and send each request straight to the right leader.

Machines fail. Kafka’s answer is to keep more than one copy of every partition.

If a partition lived on just one broker, that broker’s death would take the partition down with it, and if its disk broke, the data would be gone. Big clusters make this worse. With hundreds of brokers, hardware failures stop being rare surprises and become a normal weekly event. So Kafka keeps several copies of each partition on different brokers. Each copy is called a replica. The number of copies is the replication factor. A replication factor of 3 means every partition exists on three brokers. Most production clusters choose 3, because it survives the loss of one broker with room to spare.

One leader, several followers

Among the replicas of a partition, exactly one is the leader. Producers write only to the leader. Consumers normally read from the leader too. The remaining replicas are followers. A follower does not talk to producers at all. Its only job is to copy. It sends the leader a fetch request that says, “Please send me everything after offset N”, and appends whatever comes back to its own log, in the same order. This pull-based design is the same mechanism consumers use, which keeps the system simple.

Why only one leader? Because one writer per partition makes ordering easy. If two brokers accepted writes for the same partition at the same time, they would have to argue about which event comes first. Kafka avoids that argument by giving the pen to a single broker at a time.

Where the replicas live

Kafka spreads replicas across brokers so that no two copies of the same partition share a machine. Good clusters go one step further and tell Kafka which rack or cloud availability zone each broker sits in, using the broker.rack setting. Kafka then tries to place the copies in different racks or zones, so that one power failure or one zone outage cannot take out every copy at once. Here is a small example for a topic with three partitions and a replication factor of 3 on three brokers.

Notice two things. First, each broker leads one partition and follows two, so the work is shared fairly. Second, the order of the assigned replicas matters. The first replica in the list is called the preferred leader. Kafka tries to move leadership back to it when things are calm, so that the load stays balanced. The same order is also used as the queue of candidates when a new leader is needed. You will see why that detail matters in the chapters ahead.

This is the most important idea for understanding unclean elections. Read it slowly.

Not every follower keeps up. A follower might sit on a slow disk, pause for a long garbage-collection cycle, lose its network for a while, or simply be restarting. Kafka needs to know which followers are close enough to the leader to be trusted. So the leader keeps a list called the ISR, which stands for in-sync replicas. The ISR always includes the leader itself, plus every follower that has been keeping up.

What does “keeping up” mean? Kafka uses time, not a count of messages. A follower stays in the ISR as long as it has caught up to the end of the leader’s log at least once within the last replica.lag.time.max.ms milliseconds. In recent Kafka versions, the default for this setting is 30 seconds. If a follower has not fully caught up within that window, the leader removes it from the ISR. When the follower catches up again, the leader lets it back in. So the ISR shrinks and grows as conditions change.

How a follower drops out of the ISR

Why the ISR is a promise

The ISR is not just a status list. It is the promise that makes Kafka’s safety rules work. A message counts as committed only after every replica currently in the ISR has stored it. So if a message is committed, then every member of the ISR has it. That means any ISR member can safely become the next leader without losing a committed message. This is the core reason why clean elections choose only from the ISR.

Now look at the danger hidden in this design. The ISR can shrink all the way down to just the leader. If two followers are slow or down, the ISR is the leader alone. At that moment, messages are being committed by a group of one, and your “three copies” have quietly become a single copy. The cluster still looks healthy from the outside. Producers still get their confirmations. But if that lone leader dies now, there is no in-sync copy left anywhere.

Three small terms that decide which messages are safe and which are only “probably fine”.

Every replica has a log end offset, shortened to LEO. This is simply the offset of the next message the replica will write, so it tells you how far that copy has grown. If a replica holds offsets 0 through 9, its LEO is 10. The leader also keeps a number called the high watermark, shortened to HW. The high watermark is the smallest LEO among all replicas in the ISR. Every message with an offset below the high watermark is committed, meaning all in-sync copies hold it.

In this example the smallest LEO among the three in-sync replicas is 8, so the high watermark is 8. Offsets 0 to 7 are committed. Offsets 8 and 9 exist on the leader, but they are not committed yet, because Broker 2 has not copied them. Consumers are only allowed to read below the high watermark, so they see offsets 0 to 7 and nothing more. This rule stops consumers from reading a message that might vanish if the leader crashed a moment later.

flowchart LR
  L["Broker 1 - Leader
holds offsets 0 to 9
LEO = 10"]
  F1["Broker 2 - Follower
holds offsets 0 to 7
LEO = 8"]
  F2["Broker 3 - Follower
holds offsets 0 to 8
LEO = 9"]
  HW["High Watermark = 8
offsets 0 to 7 are committed
consumers read only these"]
  L --> HW
  F1 --> HW
  F2 --> HW

What the producer is told: acks and min.insync.replicas

The producer chooses how much certainty it wants before it treats a write as done. This choice is called acks.

A second setting, min.insync.replicas, sets a floor for acks=all. With a replication factor of 3 and min.insync.replicas=2, the leader refuses to accept a write unless at least two replicas, itself included, are in the ISR. If the ISR shrinks to one, producers get an error instead of a false sense of safety. Recent Kafka clients use acks=all by default, and that is the setting you want for data you care about.

Before we study elections, let us watch one message travel through the cluster. The election rules make much more sense afterwards.

Most explanations of Kafka skip the small steps in the middle, but those steps are exactly where the safety promises are made. When you know who does what, and in which order, the phrase “committed message” stops being abstract. It becomes a picture you can draw on a whiteboard. Follow the journey of a single write below.

The life of one write

sequenceDiagram
  autonumber
  participant P as Producer App
  participant L as Leader Broker
  participant F1 as Follower 1
  participant F2 as Follower 2
  P->>L: Produce request with acks=all
  Note over L: Check epoch and ISR size, then append at next offsets
  F1->>L: Fetch from my log end
  L-->>F1: New records
  F2->>L: Fetch from my log end
  L-->>F2: New records
  F1->>L: Next fetch shows the new log end
  F2->>L: Next fetch shows the new log end
  Note over L: All ISR members have the records, high watermark moves up
  L-->>P: Acknowledgement
  L-->>F1: High watermark in the next fetch reply
  L-->>F2: High watermark in the next fetch reply

How the controller knows a leader is gone

Each broker regularly tells the controller that it is alive. In ZooKeeper-based clusters this happened through a session that expired when heartbeats stopped. In KRaft clusters, brokers send heartbeats to the controller quorum, and the broker is fenced if they stop for longer than the session timeout, which is a matter of seconds by default. When the controller declares a broker gone, it walks through every partition that broker was leading and elects a new leader for each one, using the rules that the next chapters describe in detail.

The controller also guards the ISR. A leader cannot change its own ISR unilaterally. It sends a request to the controller that says, “I would like to shrink or grow the ISR to this set”. The controller checks the request against the current leader epoch and partition epoch, and only then records the change. This prevents an outdated leader from shrinking the ISR in a way that could hide data from the rest of the cluster.

Where the time goes in a failover

A failover has several phases, and each adds a little delay. First comes detection, because the controller must be sure that the broker is really gone and not just slow. Second comes the decision, when the controller picks a new leader for each affected partition. Third comes propagation, when the new leader and epoch are sent to all brokers. Finally, clients must notice that their old requests failed, refresh their cluster map and reconnect to the new leader. In a well-tuned cluster this takes a few seconds. In a cluster with tens of thousands of partitions on the failed broker, it may take noticeably longer, because each partition needs its own decision.

Before we study the risky path, let us see what Kafka does by default.

Leader elections are run by the controller. In older Kafka clusters the controller was one broker chosen with the help of ZooKeeper, a separate coordination service. In modern clusters that use KRaft mode, the controller role belongs to a small quorum of servers that keep the cluster’s metadata in their own replicated log. Kafka 4.0 and later run in KRaft mode only. The mechanics differ a little, but the rules for choosing a leader are the same in spirit.

The controller watches which brokers are alive. When it learns that a partition’s leader is gone, it walks through the partition’s list of assigned replicas in order and picks the first one that is both alive and in the ISR. It then bumps a counter called the leader epoch, records the new leader and the remaining ISR, and tells the other brokers about the change. Producers and consumers notice that their requests to the old leader fail, refresh their view of the cluster, and continue with the new leader. In a healthy cluster, this whole process takes a matter of moments.

Why this path never loses committed data

Remember the promise from chapter 4: a message is committed only after every ISR member has it. The new leader is an ISR member, so it has every committed message. Nothing that was confirmed under acks=all can be missing. Some uncommitted messages, those that were on the old leader but not yet copied everywhere, may be lost, but the producer was never told those were safe. A well-written producer simply retries them.

Kafka has other clean ways to change leaders as well. During a planned shutdown of a broker, called a controlled shutdown, the controller moves leadership away in an orderly manner before the broker stops. A preferred leader election moves leadership back to the first replica in the list once it has caught up. Both choose only from the ISR, so both are safe. They are not the subject of this guide, but it helps to know they exist, because people sometimes confuse “leader election” with “unclean leader election”. Only one of them can lose data.

What Kafka does when no safe candidate exists and the setting allows it to take a risk.

An unclean leader election happens when three conditions come together. First, the partition has no leader, because the old leader failed. Second, every replica that was in the ISR is dead or unreachable. Third, the setting unclean.leader.election.enable is true for that topic. When all three are true, the controller stops insisting on the ISR. It looks again at the full list of assigned replicas and picks the first one that is alive, even though that replica was outside the ISR and may be missing recent messages.

The controller then does one more thing that is easy to miss. It resets the ISR to contain only the new leader. The other replicas are no longer trusted, so they must prove themselves again by catching up. From this moment on, the new leader’s log is the official history of the partition. Whatever the old leader had beyond that point is no longer part of the story.

flowchart TD
  A["Leader of a partition fails"] --> B{"Is any ISR replica alive?"}
  B -->|"Yes"| C["Clean election
promote first live ISR replica
no committed data lost"]
  B -->|"No"| D{"Is unclean.leader.election.enable true?"}
  D -->|"No"| E["Partition stays OFFLINE
no reads and no writes
wait for an ISR replica to return"]
  D -->|"Yes"| F{"Is any other replica alive?"}
  F -->|"Yes"| G["Unclean election
promote first live replica
ISR becomes only the new leader
data loss is possible"]
  F -->|"No"| E

Clean and unclean side by side

Three ways the permission can be granted

The setting can be applied at the broker level, so that it becomes the default for all topics, or at the topic level, so that it overrides the default for one topic only. Kafka also allows these values to be changed while the cluster is running, without a restart. That flexibility is useful, because it lets you enable the permission only for the topics that truly need it. It is also dangerous, because a change made in a hurry during an incident can be forgotten and left in place for months.

Recent Kafka versions also provide a way to run an unclean election on demand. An administrator can use the kafka-leader-election tool, or the matching admin API, to ask for an unclean election on a partition that currently has no leader. This lets an operator make a deliberate, one-time decision during an outage without switching the topic to permanent “unclean” behavior. It is a much better habit than leaving the setting on “just in case”.

One concrete story, with real offsets, from the first slow follower to the final loss.

Numbers make the idea easier to hold, so let us follow one partition with a replication factor of 3. The assigned replicas are Broker 1, Broker 2 and Broker 3, in that order. Broker 1 is the leader. The producer uses acks=all, but the topic has min.insync.replicas=1, which is a common accidental setting. Unclean leader election is enabled on the topic.

sequenceDiagram
  autonumber
  participant P as Producer App
  participant B1 as Broker 1 Leader
  participant B2 as Broker 2
  participant B3 as Broker 3
  participant C as Controller
  P->>B1: Send messages for offsets 8 to 12
  B1-->>P: Confirmed, ISR holds only Broker 1
  Note over B1: Offsets 8 to 12 exist only here
  Note over B1: Broker 1 fails permanently
  C->>C: No ISR replica is alive
  C->>B2: Unclean election, become leader with epoch 1
  Note over B2: Log end is 8, five messages are missing
  B2-->>C: Leader ready, ISR is only Broker 2
  P->>B2: Send new messages
  B2-->>P: Confirmed, they receive offsets 8 and up
  B3->>B2: Fetch from offset 5

The two ingredients that made this possible

Look closely at what had to go wrong. The story needed both a shrunken ISR that let a single machine confirm writes, and the permission to elect an out-of-sync replica. If min.insync.replicas had been 2, Broker 1 would have rejected the writes for offsets 8 to 12 after the ISR fell to one, and no message would have been confirmed with a single copy. If unclean election had been disabled, the partition would have gone offline at step 5 and waited for Broker 2 to be recognized as the last trusted copy. The loss needs both weaknesses together, which is why each defence matters.

What happens when the old leader comes back and its diary no longer matches the new one.

Suppose Broker 1 in our story is repaired and restarted, with its disk intact. Its log still contains offsets 8 to 12, the five messages that the rest of the cluster has never seen. Meanwhile the new leader has written different messages at offsets 8, 9, 10 and so on. Two copies now disagree about what lives at the same offsets. Kafka cannot keep both. The new leader’s history wins, and Broker 1 must cut its log back to the point where the two histories split. This cut is called log truncation.

How a returning replica finds the split point

Early versions of Kafka used the high watermark to decide where to cut. That approach had gaps, and in some rare timings it could leave replicas with different data at the same offset, which is a serious problem. Modern Kafka uses a more reliable tool called the leader epoch. An epoch is a counter that goes up by one every time a partition gets a new leader. Each batch of messages is stamped with the epoch of the leader that wrote it, and every broker keeps a small table that says which epoch started at which offset.

When Broker 1 returns, it does not simply start fetching. It first asks the new leader, “For the epoch I last knew about, at which offset did that epoch end?” The new leader answers using its table. In our story it answers that epoch 0 ended at offset 8. Broker 1 then truncates its log so that it ends at offset 8, throwing away offsets 8 to 12, and starts copying the new leader’s messages from there. After that, all replicas agree again.

flowchart LR
  subgraph BEFORE["Before Broker 1 returns"]
    direction TB
    OB1["Broker 1 old log
offsets 0 to 12
epoch 0"]
    OB2["Broker 2 new leader
offsets 0 to 7 from epoch 0
then new data from epoch 1"]
  end
  subgraph STEP["Recovery steps"]
    direction TB
    Q["Broker 1 asks:
where did epoch 0 end?"]
    A["Broker 2 answers:
offset 8"]
    T["Broker 1 truncates
offsets 8 to 12"]
    Q --> A --> T
  end
  subgraph AFTER["After recovery"]
    direction TB
    AB1["Broker 1 log
offsets 0 to 7 old
then copy of the new leader"]
    AB2["Broker 2 leader
same history as Broker 1"]
  end
  BEFORE --> STEP --> AFTER

Fencing: why an old leader cannot do harm

The epoch has a second job. It is used to fence out leaders that are outdated. Imagine a broker that was cut off from the network and does not know it has been replaced. It may still believe it is the leader and accept writes. Every request in Kafka carries the epoch it expects, and replicas reject requests that carry an older epoch than they know of. So the outdated leader cannot slip new writes into the cluster, and its ISR changes are refused by the controller. This stops a dangerous “two leaders” situation known as split brain.

Applications feel the election in different ways, and some of the effects are silent.

The producer’s side

During the moments when a partition has no leader, a producer receives retriable errors such as “leader not available” or “not leader or follower”. Well-configured producers retry automatically, and the messages wait in a memory buffer until a leader appears or the delivery timeout expires. With unclean election off, an offline partition can keep producers waiting for a long time, the buffer can fill up, and the application may start blocking or failing. With unclean election on, the wait is short, because a new leader appears quickly.

The important part is what the producer does not see. Messages that were confirmed before the election and then lost never send an error afterwards. The confirmation was true when it was given. The producer moves on with no clue that five of its messages no longer exist. Modern producers also enable idempotence by default, which prevents duplicate writes during retries, but idempotence does nothing to bring back messages that a lost leader held alone.

The consumer’s side

Consumers keep a position, which is the offset they will read next. That position is saved in Kafka in an internal topic named __consumer_offsets. After an unclean election, the log may be shorter than a consumer’s saved position. In our story, a consumer may have read up to offset 12 and saved position 13, but the new leader’s log ends at offset 8. When the consumer asks for offset 13, the broker replies that the offset is out of range.

The consumer then follows its auto.offset.reset policy. If the policy is latest, it jumps to the end of the log and skips whatever it has not yet seen. If the policy is earliest, it goes back to the start of the retained log and reprocesses a great deal. If the policy is none, it throws an error and lets the application decide. There is also a sneakier case. If new messages arrive fast enough that the log grows past the consumer’s saved position before the consumer asks again, the position looks valid, and the consumer would carry on from the wrong place with no error at all. Kafka closed this gap in version 2.1 by letting consumers send the leader epoch with their fetches, so that a truncation can be detected and either handled or reported.

Effects on stateful systems

Stateful applications are hurt the most. A stream processing job may keep its running totals in a local store that is backed by a Kafka changelog topic. If that changelog loses its tail, the rebuilt store can differ from what the job computed earlier. Kafka’s exactly-once features assume that a message, once confirmed, stays confirmed. An unclean election breaks that assumption, so any guarantee built on top of it can quietly weaken. The internal topics that hold consumer offsets and transaction state deserve extra care for the same reason, and are normally configured with a high replication factor and strict settings.

Every distributed system faces this choice. Unclean election is Kafka’s version of it.

The CAP theorem says that when a network splits or machines fail, a distributed system cannot promise both consistency and availability at the same time. Consistency means everyone sees the same, correct data. Availability means every request receives an answer. When failures strike, you must lean one way. Kafka’s default settings lean toward consistency for committed data: if it cannot guarantee that the next leader has everything, the partition waits. The unclean setting leans toward availability: the partition answers again quickly, even if that answer is built on a shorter history.

A simple way to decide

Ask three questions about each topic. First, can the data be recreated from another source, such as the database that produced it? Second, what is the cost of a missing record, in money, trust or legal risk? Third, what is the cost of an hour of downtime for this stream? If the data can be recreated and downtime is costly, the permission may be reasonable. If the data cannot be recreated and a missing record is costly, keep the setting off. In many companies the answer differs for different topics, so the setting should be chosen topic by topic.

The best way to avoid a bad election is to make sure the ISR rarely shrinks in the first place.

Unclean elections are the last domino in a row. The first domino is almost always a follower that cannot keep up. So performance work and safety work are really the same job. A cluster with enough headroom keeps its followers close to the leader, keeps its ISR full, and rarely faces the hard choice at all. The following ideas show how size and speed connect to the safety of your data.

What replication costs

Every byte written to a topic with a replication factor of 3 is stored three times and sent across the network twice more, from the leader to two followers. That means the cluster’s internal traffic is about twice its incoming traffic, on top of what consumers read. Disks must absorb the same amount of writing, three times over. If you plan capacity by looking only at the producer traffic, you will under-size the network and the disks, and the first sign of trouble will be followers dropping out of the ISR under load.

Common design choices and their prices

Habits that keep followers close to the leader

• Leave headroom. Plan for the cluster to run comfortably below its limits on normal days, so that a spike or a lost broker does not push it over the edge.
• Tame pauses. Use a modern garbage collector, size the memory of the broker process sensibly, and leave plenty of memory for the operating system’s page cache, which Kafka depends on for fast reads and writes.
• Use fast disks and enough network. Replication traffic competes with producer and consumer traffic, so separate or generous links help.
• Throttle heavy maintenance. Moving partitions between brokers creates a lot of copy traffic. Kafka provides replication throttle settings, such as leader.replication.throttled.rate and follower.replication.throttled.rate, so that a rebalance does not starve normal replication and push followers out of the ISR.
• Keep partition counts reasonable. A broker that leads or follows an enormous number of partitions has more to copy, more files to open and a slower failover.
• Restart gently. Restart one broker at a time and wait for the under-replicated count to reach zero before moving on.

Scaling the number of brokers

Adding brokers gives you more room, but new brokers start empty. Kafka does not move partitions on its own, so you must reassign some partitions to the new brokers. While that copying happens, the cluster is under extra load and the ISR of moving partitions is briefly bigger than usual. Do this in quiet hours, use throttles, and check the under-replicated metric between steps. Growing the cluster in small, planned steps is always safer than a large rebalance done in a hurry.

Unclean election never works alone. Five settings work together, and you must read them as a team.

Whether you lose data in a bad moment depends on a handful of settings that interact with each other. Looking at unclean.leader.election.enable alone is like checking one lock on a door that has four locks. The table below lists the settings you should review together for every important topic.

The durability triangle

Three settings form a triangle that protects confirmed data: a replication factor of 3, min.insync.replicas of 2, and acks=all. Together they say, “A write is only confirmed when at least two brokers hold it.” With this triangle in place, losing any single broker cannot lose a confirmed message, and the write path pauses rather than confirming a message that lives on one machine. Add unclean.leader.election.enable=false and you have closed the fourth lock: a stale replica can never become the leader.

Two mistakes with min.insync.replicas

Setting min.insync.replicas equal to the replication factor sounds extra safe, but it means that a single broker failure blocks every write to the partition. This is rarely what people want. Setting it to 1 removes the protection, and lets one machine confirm writes on its own. The balanced choice for a replication factor of 3 is 2, which tolerates one failure for both reads and writes.

Client settings that make trouble visible

Broker settings decide whether loss can happen. Client settings decide whether your applications notice it, survive it and recover from it. It is worth reviewing them as carefully as the broker side, because the same failure can be harmless for a well-built client and painful for a careless one.

A word about auto.offset.reset. Choosing latest means that after an out-of-range error, the consumer skips to the newest data and may miss records. Choosing earliest means it replays everything still retained, which can be huge and can cause duplicates. Neither is right for every application. A payment processor may prefer to stop and raise an alarm, so that a human decides. A dashboard may prefer to jump to the latest data. The important point is to make the choice on purpose, write it down, and test it.

Different topics deserve different answers. Here is how to think about the common kinds.

Rather than using one cluster-wide answer, sort your topics into groups by how much a lost record and an hour of downtime would cost. The cards below describe the usual groups and the direction each one normally takes.

A short method for deciding

How well-known organizations use Kafka, and what each one teaches about this trade-off.

A durability setting is also a security control. Someone who can change it can quietly weaken your guarantees.

It may seem strange to talk about security in a chapter on leader election, but the two are close neighbours. A configuration change that turns the unclean permission on for a sensitive topic does not raise any alarms and changes nothing you can see. It just makes a future failure more damaging. That makes it an attractive target for accidents, and also for anyone who wants to cause harm without being noticed. Treat these settings with the same care as a firewall rule.

Who may change these settings

Kafka supports access control lists, called ACLs, that decide which users may perform which actions. Changing topic configuration and triggering leader elections are powerful administrative actions. They require permissions such as altering configuration on a topic, or altering the cluster as a whole. Give these permissions to a small group of platform engineers and to automation accounts, not to every application team. Application service accounts should only need to produce and consume.

Security effects of a rolled-back history

Losing or rewinding data can be a security problem too. Imagine a topic that carries “this access token was revoked” events, or “this account was locked” events. If an unclean election loses the tail of that topic, the consumers may never learn that a token was revoked, and a door that should be closed stays open. The same is true for compacted topics that hold the latest permission state for each user. A rolled-back value can bring back an old permission. For this reason, security-related event streams belong firmly in the strict group.

You cannot fix what you cannot see, and unclean elections are quiet by nature.

Because an unclean election does not raise an error for producers or consumers, the only reliable way to notice it is to watch the cluster itself. Kafka brokers and controllers publish metrics that describe leader elections, ISR changes and offline partitions. Set up alerts on the following ones. Names can differ slightly between Kafka versions, so confirm them in the documentation for the version you run.

flowchart LR
  BR["Brokers and Controller
publish metrics and logs"] --> MC["Metrics Collector
Prometheus or similar"]
  BR --> LG["Log Aggregator
controller and state change logs"]
  MC --> AL{"Offline partitions or
unclean elections above zero?"}
  LG --> AL
  AL -->|"Yes"| PG["Page the on-call engineer"]
  AL -->|"No"| DB["Dashboards and daily review"]
  PG --> RB["Follow the runbook
scope, recover, reconcile"]

What to look for in logs

The controller log and the state change log record every change of leader and ISR. During an incident, search for partitions that moved from offline to online with an ISR of just one member, since that is the fingerprint of an unclean election. On the returning broker, look for lines that mention truncating a partition to a particular offset. They tell you exactly where the log was cut, and they help you work out the size of the lost window.

Watch the early signs, not only the disaster

An unclean election is the last step in a chain. Long before it, you will usually see the ISR shrink and grow again, followers lag more than usual, and disks or networks show stress. Alerting on those early signs gives you time to act. A very useful habit is to keep a dashboard that shows, for each important topic, the number of partitions whose ISR is smaller than the replication factor.

Test for loss from the outside

Broker metrics show what Kafka believes. A small test program shows what applications experience. Write a canary that produces numbered messages at a steady pace into a strict topic and consumes them back, watching for gaps or repeats in the numbers. If a number ever goes missing, you have found data loss even if every metric looked clean. Some teams also add sequence numbers to their real events, so that downstream systems can detect gaps on their own.

Real failures come in many shapes. Here are the ones that lead to unclean elections, and what to do afterwards.

The recovery playbook

When a partition goes offline, the choice is not between “enable the permission” and “do nothing”. There is a better order of steps, and it helps to have it written down before you need it.

flowchart TD
  S["Partition offline"] --> Q1{"Can the last ISR broker
be brought back in time?"}
  Q1 -->|"Yes"| R1["Restore it and let a clean election happen
no data loss"]
  Q1 -->|"No"| Q2{"Can the business accept
possible loss on this topic?"}
  Q2 -->|"No"| W["Keep waiting and escalate
restore from backup or upstream source"]
  Q2 -->|"Yes"| U["Preserve old logs, then run a targeted
unclean election for this topic only"]
  U --> V["Restore strict setting after recovery"]
  V --> Z["Reconcile lost range and fix consumers"]

Habits that keep data safe, and habits that cause the loss stories you read about.

Patterns that work

Anti-patterns to avoid

A checklist you can take to your next design review.

Common mistakes in more detail

Mistake one: believing that acks=all is enough. As you saw in chapter 5, acks=all only waits for the members of the ISR. If the ISR is small and the permission is on, a confirmed message can still disappear. The protection comes from the whole set of settings, not from one of them.

Mistake two: not testing multi-failure cases. Most teams test the loss of one broker, which Kafka handles gracefully. Unclean elections come from the second and third failures, which are rare and rarely tested. Include double failures in your drills.

Mistake three: forgetting the internal topics. The topics that store consumer offsets and transaction state are just as vulnerable. Losing consumer offsets can make an entire consumer group reprocess or skip data. Check that their replication factor and minimum ISR are set as strictly as your important data topics.

Mistake four: ignoring the consumer side. Even with strict settings, an unexpected loss can happen through hardware fault or human error. Consumers should have a deliberate policy for out-of-range offsets, and applications should be able to detect gaps through sequence numbers or reconciliation jobs.

Mistake five: measuring availability without measuring durability. A dashboard that reports 100 percent uptime says nothing about lost records. Track both, and give durability the same seriousness as availability.

The mechanics of who leads a partition keep evolving, though the core trade-off stays the same.

For many years, Kafka relied on ZooKeeper to store cluster information and to help pick a controller. Modern Kafka replaces it with KRaft, where a small group of controller servers keep the cluster’s metadata in their own replicated log and use the Raft consensus method to agree on changes. Kafka 4.0 and later run in KRaft mode only. In this design, the active controller records leader changes as entries in the metadata log, and brokers read those entries to learn about new leaders. Failovers can be quicker in large clusters with many partitions, because the metadata is already replicated and ready.

The rules for unclean election did not change with KRaft. The controller still prefers an ISR member, still resets the ISR to the new leader when it has to go outside it, and the same setting still controls the permission. What improves is the speed and the clarity of the process, not the underlying trade-off between keeping data and staying available.

Ideas to make recovery smarter

The Kafka community has been exploring ways to lower the need for risky elections. One such proposal, known as eligible leader replicas (KIP-966), aims to keep track of which replicas are known to hold all committed data, even after the ISR has shrunk. With that memory, the controller could pick a safe leader in more situations, and leave fewer partitions stuck offline. The status of ideas like this changes between releases, so read the release notes for your version before relying on them.

Other trends help indirectly. Follower fetching lets consumers read from a nearby replica, which can reduce cross-zone traffic. Tiered storage moves old data to cheaper storage, which makes brokers lighter and faster to recover. Better tools for rolling restarts and rebalancing reduce the number of times that a cluster is exposed to risk. None of these removes the need to understand unclean elections, but together they make the dangerous situation less common.

Let us put everything together on a made-up company, so that the ideas turn into real decisions.

Imagine a fictional shop called FreshCart that sells groceries online. It has a website, a mobile app, a warehouse system and a small data team. Its Kafka cluster has six brokers spread over three availability zones, two brokers in each. The platform team wants clear rules for the topics that different teams create. Instead of arguing about each topic, they define two named profiles. The critical profile uses a replication factor of 3, a minimum ISR of 2 and unclean election off. The telemetry profile uses a replication factor of 3, a minimum ISR of 2 and unclean election on. Both use zone-aware placement, so the difference is only in how they behave when almost everything has failed.

Why these choices make sense

The first five topics are either money, stock counts, permissions or legal records. A lost record here creates a wrong answer that can last for a long time, so these topics would rather wait than guess. The last four topics describe things that are either high in volume, quickly out of date or only useful in bulk. If one partition of delivery-van-locations loses ten seconds of pings, the map on a customer’s phone simply shows a slightly older position until the next ping arrives. Blocking the whole location pipeline to protect ten seconds of pings would be a bad trade.

A bad night in the data centre

Now imagine that a power fault takes out one whole zone during a sale, and at the same moment a broker in another zone is paused by a long garbage-collection cycle. Some partitions have lost two of their three copies, and the last one is behind. Here is what each group of topics experiences.

A tabletop exercise you can run with your own team

You can practise the same story without touching a real cluster. Gather the platform team and a few application owners in one room, and read out a failure one step at a time: “A zone is down. Two brokers are gone. A third is paused. Which of your topics are offline? Which came back through an unclean election? What do your consumers do next?” For each topic, ask the owner to say aloud what they would check and what they would tell the business. Gaps in the answers show you where the runbook, the alerts or the topic settings need work. Repeat the exercise every quarter, with a different failure each time.

Everything in this guide can be held in a few sentences.