Before any technical words, let us understand the simple problem that leader election solves.

A distributed system is a group of computers that work together so they look like one big computer. The group gives you more power and more safety, because if one computer breaks, the others can carry on. But there is a catch. When many computers share one job, they must agree on the facts. If computer A believes “the file lives on machine 7” and computer B believes “the file lives on machine 9”, users will get errors, lost data, or worse.

The easiest way to keep everyone in agreement is to have one computer make the decisions and tell the others. That computer is called the leader. The others are called followers. Followers copy what the leader decides and stand ready to take over. This pattern is used almost everywhere in big software systems, from databases to messaging systems to cloud control planes.

What is leader election?

Leader election is the process by which a group of computers chooses exactly one of them to be the leader. It runs when the group starts for the first time, and again whenever the current leader stops responding. The most important rule is that the group must never end up with two leaders at the same time. That dangerous situation has a name, split brain, and a good election design makes it impossible.

Two different elections in Kafka

Here is the part that confuses many beginners. Kafka actually has two layers of leaders. The first layer is the controller leader. It is one node, chosen by the KRaft controller quorum, that manages the whole cluster’s metadata. The second layer is the partition leader. Every slice of every topic has its own leader, and that leader is chosen by the controller leader. This guide focuses on the first layer, the KRaft election, and then shows how it powers the second layer.

Why this matters to real businesses

Large companies run Kafka to move things like payment events, ride requests, video views, and sensor readings. When Kafka pauses even for a short time, apps that depend on it can stall. A fast and safe election means that the “brain” of Kafka recovers by itself, without a human waking up at three in the morning. That is why the election design is one of the most carefully engineered parts of the whole system.

A short tour of the Kafka words we will use again and again.

Kafka is a system for storing and moving streams of messages. Apps that send messages are called producers. Apps that read messages are called consumers. In the middle sits the Kafka cluster, made of computers called brokers. A broker is simply a server that stores messages on disk and serves them to consumers.

What is “metadata”?

Metadata means “data about data”. In Kafka it includes the list of topics, the number of partitions, which brokers hold each replica, who the current partition leaders are, access rules, and settings. It is small compared with the messages themselves, but it is extremely important. If the metadata is lost or confused, Kafka does not know where anything is, even though all the messages are still sitting on disk.

Why the metadata needs its own careful home

Many brokers must see the same metadata, and they must see changes in the same order. Suppose two admins create a topic with the same name at the same moment. Exactly one must win, and every broker must agree on which. Somewhere there must be a single trusted “source of truth” with a strict order of events. In Kafka, the KRaft controller quorum is that source of truth.

To appreciate the new design, we first look at the old one and the pain it caused.

For more than a decade after Kafka was born at LinkedIn, Kafka relied on a separate tool called Apache ZooKeeper. ZooKeeper is a small, well-known coordination service. It stores tiny pieces of data in a tree of nodes and lets many programs watch those pieces for changes. Kafka used it as its filing cabinet for metadata and as the referee for choosing a controller.

How the old election worked

In the ZooKeeper design, every broker was allowed to become the controller. All brokers raced to create a special short-lived entry in ZooKeeper. This entry could only exist once, so the first broker to create it won. The winner became the controller. If the controller died, its entry vanished automatically, ZooKeeper told the other brokers, and they raced again.

The pain points

Push instead of pull

In the old design, the controller pushed updates to every broker after each change. When a cluster had many brokers and many partitions, one change could trigger a flood of messages. Brokers that missed an update could hold a stale picture of the cluster until the next push. A better idea was for brokers to pull a shared log of changes at their own pace, which is exactly what KRaft does.

KRaft is Kafka’s own built-in way of agreeing on metadata, without any outside helper.

The community proposed the move away from ZooKeeper in a design document known as KIP-500 (a KIP is a “Kafka Improvement Proposal”). The plan was simple to say: let Kafka store its metadata inside Kafka itself, and let a small group of Kafka nodes agree on it using a well-known agreement method called Raft. The name KRaft is short for “Kafka Raft”.

The rollout happened in stages. KRaft first appeared as an early preview in the Kafka 2.8 line, became approved for production use in the 3.3 line, and ZooKeeper support was fully removed in the Kafka 4.0 line. Today, KRaft is the only way to run new Kafka clusters on current versions.

What is Raft?

Raft is a consensus algorithm. Consensus is a fancy word for “a group of computers agreeing on one answer, even when some of them crash or lose their network.” Raft was designed in the 2010s as an easier-to-understand alternative to older methods. It is now used inside many famous tools, such as etcd (the memory of Kubernetes), Consul, and CockroachDB.

The big idea: one log, many copies

KRaft keeps every cluster change in a single ordered list called a log. Every change, such as “create topic orders”, “broker 4 joined”, or “partition 2 now led by broker 5”, is added to the end of this log as a small record. The log lives in a special internal topic named __cluster_metadata, which has exactly one partition. Every controller keeps a copy of it, and every broker reads it too.

Because the log is ordered, everyone who reads it in order ends up with exactly the same picture. This is the heart of the whole design: the state of the cluster is simply the result of replaying the log. If you lose a node, you can rebuild its picture by replaying the log from the start, or from a saved snapshot plus the newer records.

Why a leader is required

Raft allows only one node, the leader, to add records to the log. That single-writer rule guarantees that records have a clear order and never conflict. The other controllers are followers: they copy the leader’s log and wait. When the leader vanishes, they elect a new one from among themselves, and the new leader continues writing from exactly where the old one left off.

A Kafka process can wear one hat, or two. Let us see what each hat means.

Every Kafka server in KRaft mode has a setting called process.roles. It can say broker, controller, or both. This one setting decides what the server does in the cluster.

Why dedicated controllers are safer

Imagine the head librarian also had to carry boxes of books all day. If a huge delivery arrives, the librarian gets tired and slow, and the catalog stops being updated. In the same way, if a controller also serves heavy message traffic, a traffic spike or a slow disk can make the controller late for its own heartbeats. Late heartbeats look like a crash, and that triggers unnecessary elections. Keeping controllers on separate small machines protects the brain from the muscle.

Voters and observers

Not every controller has to vote. In Raft, a node that copies the log but does not vote is often called an observer. Brokers act like observers of the metadata log: they read it, but they never vote for a leader and never count toward a majority. The nodes that do vote are the voters, and together they form the quorum.

Here is the whole picture: who talks to whom, and why.

flowchart TB
  APP["Producers and Consumers"]
  subgraph BF["Broker Fleet - data plane"]
    B1["Broker 1"]
    B2["Broker 2"]
    B3["Broker 3"]
  end
  subgraph QU["KRaft Controller Quorum - control plane"]
    C1["Controller 1 - ACTIVE LEADER"]
    C2["Controller 2 - Follower"]
    C3["Controller 3 - Follower"]
  end
  LOG[("Metadata Log - cluster_metadata topic")]
  ADM["Admin Tools and Clients"]
  APP -->|"send and read messages"| BF
  ADM -->|"create topics and change settings"| C1
  C2 -->|"fetch new records"| C1
  C3 -->|"fetch new records"| C1
  C1 -->|"appends records"| LOG
  B1 -->|"register and heartbeat"| C1
  B2 -->|"register and heartbeat"| C1
  B3 -->|"register and heartbeat"| C1
  B1 -.->|"fetch metadata"| C1
  B2 -.->|"fetch metadata"| C1
  B3 -.->|"fetch metadata"| C1
    

Let us walk through the figure from the top. Producers and consumers talk only to brokers. They never talk to controllers. Admin tools, such as the ones used to create topics, send their requests to the cluster, and those requests end up at the active controller, because only the active controller is allowed to change metadata.

The components, one by one

Why the controllers keep things in memory

Every controller, not only the leader, builds a full picture of the cluster in memory by replaying the log. When a follower becomes the new leader, it already has that picture. It does not need to go off and load the whole cluster state from some outside place first. This is one of the biggest reasons why failover in KRaft is much faster than in the ZooKeeper design.

Five simple ideas explain almost everything about how the election stays safe.

Idea 1: Terms, called epochs in Kafka

Time in Raft is chopped into numbered periods. Kafka calls each period an epoch. Each epoch has at most one leader. When a new election starts, the epoch number goes up by one. A bigger epoch always beats a smaller one. If a node ever sees a message with a bigger epoch than its own, it knows it is out of date and immediately steps back.

Idea 2: A majority decides everything

A candidate wins only if more than half of all voters say yes. With three voters, that means two. With five voters, that means three. Because two different majorities of the same group must overlap by at least one member, two candidates can never both win in the same epoch. This overlap trick is what makes split brain impossible.

Idea 3: One vote per epoch

A voter may give only one vote in each epoch, and it writes that vote to disk before replying. Even after a crash and restart, it remembers its vote and cannot vote for a second candidate in the same epoch.

Idea 4: Only well-informed candidates win

A voter refuses to vote for a candidate whose log is behind its own. It compares the epoch and position of the last record. This guarantees that any new leader already holds every record that the previous leader had told anyone was committed. So a committed change is never lost by an election.

Idea 5: Nodes are always in one of a few states

At any moment a controller is in one of a small set of states. The simplified picture below shows the main ones.

stateDiagram-v2
  [*] --> Follower
  Follower --> Candidate : leader silent too long
  Candidate --> Leader : wins majority of votes
  Candidate --> Follower : another leader appears or a higher epoch is seen
  Candidate --> Candidate : split vote then retry with a new epoch
  Leader --> Follower : sees a higher epoch or loses the majority
    

The real Kafka code has a few extra helper states, such as a node that has voted but has not yet heard from a leader, and a leader that is in the middle of gracefully resigning. They are refinements of the same simple picture.

How Kafka differs slightly from textbook Raft

In the original Raft paper, the leader pushes new entries to followers. KRaft flips this to match Kafka’s style: followers pull new records from the leader by sending fetch requests, just like Kafka replicas and consumers already do. Elections still use vote requests that candidates send out. This pull design reuses well-tested Kafka code and makes it easy for brokers, which are not voters, to follow the same log.

Follow one full election from the moment the old leader disappears to the moment the new leader starts working.

Let us use a cluster with three controllers: Controller 1, Controller 2, and Controller 3. Controller 1 is the active leader in epoch 5. Everything is calm. Controllers 2 and 3 keep sending fetch requests to Controller 1, and Controller 1 keeps answering. Then, suddenly, Controller 1’s machine loses power.

sequenceDiagram
  participant C1 as Controller 1 - old leader
  participant C2 as Controller 2
  participant C3 as Controller 3
  Note over C1: Machine crashes in epoch 5
  C2->>C1: Fetch request - no answer
  C3->>C1: Fetch request - no answer
  Note over C2: Fetch timeout expires and random wait ends
  C2->>C2: Becomes Candidate in epoch 6 and votes for itself
  C2->>C3: VoteRequest epoch 6 with last log position
  C3-->>C2: VoteResponse granted
  Note over C2: 2 of 3 votes so it becomes Leader
  C2->>C3: Leader change record appended to the log
  C3->>C2: Fetch request - gets the new record
  Note over C2,C3: Cluster metadata writes resume
    

The graceful path

Not every leadership change is a crash. If an operator shuts down the active controller on purpose, it can politely tell the other voters that it is stepping down. The followers then know the leader is leaving and can start the election right away instead of waiting for a timeout. Planned restarts, such as rolling upgrades, therefore cause even shorter pauses than crashes.

What if two followers both decide to run at the same time? Kafka has a simple answer.

A split vote happens when two candidates start at nearly the same moment, and each collects its own vote but neither gets a majority. In a three-node group, if Controller 2 and Controller 3 both become candidates, each votes for itself, and each refuses the other. Nobody wins that round.

Raft’s answer is delightfully simple: randomness. Each candidate waits a random amount of time before trying again. Almost always, one candidate wakes up first, asks for votes first, and wins before the other even starts. The random wait is controlled by settings such as the election timeout and the maximum election backoff, and their defaults are on the order of one second.

Why the timers must be tuned as a family

The “disruptive server” problem and pre-vote

Imagine one controller whose network cable is loose. It cannot hear the leader, so it keeps starting elections with bigger and bigger epoch numbers. When its network comes back, it announces a huge epoch, and the perfectly healthy leader is forced to step down because a bigger epoch always wins. The whole cluster suffers because of one flaky node.

To prevent this, newer Kafka releases added a pre-vote step. Before a follower raises its epoch, it first asks the others: “If I ran, would you vote for me?” Only if a majority says yes does it start a real election. A node that is cut off from everyone simply gets no positive answers and stays quiet, without pushing the epoch number up. Healthy leaders are no longer toppled by a lone troublemaker.

Winning the vote is only the first half. The new leader must also get the group back in sync.

The moment a candidate wins, it becomes the leader for the new epoch and adds one special record to the log. This record says “epoch 6 began, and I am the leader.” Followers replicate it like any other record. It also serves as a checkpoint: after this record is copied to a majority, the leader knows its authority is real.

Cleaning up the tail of the log

The old leader might have written some records that only it received before it crashed. Those records were never confirmed as committed. If a follower has such an extra tail, the new leader’s log wins. When that follower fetches, the leader tells it where the two logs first differ, and the follower throws away its extra records and copies the leader’s version. This is safe because uncommitted records were never promised to anyone.

The high watermark

The high watermark is a number that marks the last position in the log which is safely stored on a majority of voters. Records at or below it are committed, which means they will survive any future election. The leader raises the high watermark only after enough followers report that they have copied the records. Only committed records are shown to brokers and applied to the official cluster state.

Raft is trusted because it proves a small set of promises that always hold, no matter what crashes.

The people who designed Raft did not just test it. They wrote down a short list of properties that must always be true, and then showed why the rules make them true. KRaft inherits these properties. Understanding them in plain words will give you a deep feeling of trust in the whole system.

How old leaders are shut out

Every request between controllers carries the sender’s epoch. A node that receives a request with a lower epoch than its own simply rejects it and tells the sender about the newer epoch. So if an old leader wakes up after a long pause, such as a freeze caused by a garbage-collection stall, it will quickly learn it has been replaced and step down. This idea of using a growing number to lock out stale actors is often called fencing, and Kafka uses it in many places: controller epochs, partition leader epochs, and broker epochs.

How the metadata log stays identical on every controller, and how it stays small enough to manage.

The write path for one metadata change

Notice the pattern: the leader never tells the admin “success” until the change is safe on a majority. That is what makes it safe for the leader to crash right after answering. The change is already on at least two of the three machines, and any new leader must come from a group that includes at least one of them.

Why snapshots exist

A cluster can create millions of metadata records over years. Replaying all of them after every restart would be slow and would waste disk space. So each node periodically writes a snapshot, a compact copy of the entire cluster state at a certain log position. After a snapshot is safe, all older log segments can be deleted. On restart, a node loads the latest snapshot, then replays only the few records after it.

Snapshots are taken automatically after a certain amount of new metadata or after some time has passed, whichever comes first. If a follower or a new broker has fallen so far behind that the leader has already deleted the records it needs, the leader simply sends the latest snapshot first and then the newer records. This is how a brand new controller or a broker that was offline for days catches up quickly.

The log is just a long list of small records. Knowing the common ones makes the whole system feel less mysterious.

Every record in the metadata log is a tiny, typed message that describes one change to the cluster. Each type has a name and a clear purpose. When a controller or broker replays the log, it reads the records in order and updates its in-memory picture, one change at a time. Here are the most common kinds.

The metadata version

Kafka evolves, and new releases sometimes add new kinds of records. If a brand new controller wrote a record type that an older broker could not understand, the older broker would be confused. To prevent this, the cluster has a shared setting called the metadata version. It says “the cluster as a whole agrees to use record types up to this level.” During an upgrade, you first replace the software everywhere and only then raise the metadata version. Until you raise it, the cluster keeps writing records that old and new software both understand, which keeps rollbacks possible.

Why the log is a good design

Because every change is a record in an ordered list, several useful things come for free. A new broker can catch up by reading from a snapshot and then the tail. An auditor can see who changed what and in which order. A test can replay the log and check that it produces the same state. And because every node reads the same list, arguments about “who has the newest version of the truth” simply do not arise.

Controllers elect a leader among themselves, but the brokers must also learn who it is.

Brokers are not voters, yet they depend on the active controller all the time. Each broker is configured with the addresses of the controller nodes. When a broker starts, it finds the current leader by asking. If it asks a follower, the follower answers with who the current leader is and which epoch it is in.

Registration and heartbeats

A broker first registers with the active controller, saying “I am broker 4, here are my addresses, and here are my supported features.” The controller writes this into the log. From then on, the broker sends a heartbeat every couple of seconds. The heartbeat carries the broker’s progress in reading the metadata log. The controller uses heartbeats to know which brokers are alive.

Fencing: keeping sick brokers out

If a broker’s heartbeats stop for longer than its session timeout (the default is about nine seconds), the controller fences it. A fenced broker is treated as unavailable: it loses any partition leadership it held, and it must catch up on metadata before it is allowed back in. This protects clients from talking to a broker that has lost touch with the cluster and might hold stale information.

Pull-based metadata updates

Brokers fetch new metadata records from the controller leader in the same way followers do, and apply them to their own in-memory view. When a new controller leader is elected, brokers’ fetches to the old leader start failing. They look for the new leader, reconnect, and continue from where they stopped. Because they pull from a position, they never miss an update, and they never apply one twice.

The controller leader’s most visible job is choosing who leads each partition.

Remember that every partition has one leader replica and some follower replicas. When a broker dies, all the partitions it led suddenly have no leader. Producers and consumers cannot use them until a new leader is picked. This is where the active controller earns its keep.

flowchart TD
  A["Broker 2 stops sending heartbeats"] --> B["Active controller fences Broker 2"]
  B --> C["Find every partition led by Broker 2"]
  C --> D{"Is another In-Sync Replica alive?"}
  D -->|"Yes"| E["Pick the first live replica from the In-Sync list"]
  D -->|"No"| F{"Is unclean election allowed?"}
  F -->|"No"| G["Partition stays offline until an In-Sync Replica returns"]
  F -->|"Yes"| H["Pick any live replica and accept possible data loss"]
  E --> I["Write partition change record to the metadata log"]
  H --> I
  I --> J["Majority of controllers commit the record"]
  J --> K["Brokers fetch the record and update their roles"]
  K --> L["Clients refresh metadata and use the new leader"]
    

The In-Sync Replica list is the safety net

A replica is in sync when it has copied everything the leader has, or is very close. By default, Kafka only lets an in-sync replica become the new leader. That way, the messages that were confirmed to producers survive the failover. When producers ask for the strongest guarantee, “acknowledge only after all in-sync replicas have the message,” a message that was confirmed is guaranteed to exist on whichever in-sync replica is picked next.

Who updates the In-Sync list?

The partition leader itself notices which followers are keeping up. When it wants to shrink or grow the list, it sends a request to the active controller. The controller checks the request against its records, including the leader’s epoch, and only then writes the change. This stops an old, confused leader from changing the list after a new leader has already taken over.

Unclean election: availability versus safety

If every in-sync replica is gone, Kafka has a hard choice. It can keep the partition offline and wait for one to return, which protects data but hurts availability. Or, if the setting for unclean leader election is on, it can promote an out-of-date replica. The partition comes back sooner, but some already-confirmed messages may be lost. The default is safety first: unclean election is off.

Preferred leaders and rebalancing

The first replica in a partition’s list is called the preferred leader. Kafka spreads preferred leaders evenly across brokers on purpose. After a failure, leadership may concentrate on a few brokers. A background task in the controller periodically checks the balance and, when the imbalance passes a threshold, moves leadership back to the preferred replicas.

A design is only as good as the way it behaves when things go wrong. Let us walk through the common failures.

Let us put approximate times on a full failure so you can picture how long each phase takes.

The numbers below use typical default timers and are meant as a teaching picture, not a promise. Real clusters differ with network speed, machine load, and settings. The point is to see which phases dominate.

Case 1: The active controller machine loses power

Add these up and you get a total pause of a few seconds for tasks that need the controller. Notice that almost all of the time is spent noticing the failure and waiting the random pause, not in the vote itself.

Case 2: A broker machine loses power

In this case, the pause for the affected partitions is dominated by the broker session timeout. If the broker is shut down politely, it can tell the controller first, and leadership moves before the broker leaves, so clients often see almost no pause.

How many voters should you run, and where should you put them?

The size of the quorum decides how many failures you can survive. The rule is simple: the group keeps working as long as a majority is alive. The table shows what that means for common sizes.

Why not just add more voters?

Every commit has to wait for a majority to write to disk and reply. With more voters, the “slowest member of the fastest majority” tends to be slower, and there are more messages moving around. More voters also means more machines to secure and keep healthy. Most clusters do very well with three. Choose five when you want to keep full protection even while one controller is being upgraded or repaired.

Where to place the voters

Put each voter in a different failure zone. In a public cloud, that usually means different availability zones. With three voters in three zones, a whole zone can go dark and the quorum survives. Do not stretch the voters across faraway regions. Every commit would then pay the round-trip delay between regions, and long delays trigger false elections.

Changing the voter set later

Early KRaft releases used a fixed list of voters written in each node’s settings, and changing it needed careful, coordinated edits and restarts. Newer releases support a dynamic quorum, where an operator can add or remove a voter using a command while the cluster is running, and the change itself is recorded in the log. This makes replacing a broken controller machine much less risky.

Different teams need different layouts. Here is a simple guide to picking one.

There is no single perfect layout. The right choice depends on how much a pause would cost you, how big your cluster is, and how many people you have to run it. Use the three situations below as starting points and adjust from there.

Why the small quorum can look after very large clusters.

It may seem odd that three little controller nodes can manage a cluster with dozens of brokers and a huge number of partitions. The reason is that metadata is tiny compared with message data. A partition needs only a few small records: where it lives, who leads it, and which replicas are in sync. Even with a very large number of partitions, the metadata fits comfortably in the memory of a controller.

What limits the speed of metadata changes?

How the design scales

Adding brokers does not add voters. Brokers only read the log, so the quorum size stays fixed while the number of brokers grows. Reads scale easily because a broker fetches at its own pace and fetching is cheap. In the old push design, the controller had to send an update to every broker after each change, so its work grew with the size of the cluster. In KRaft, the leader only serves fetch requests, which are simple and can be answered from data already in memory or in the operating system’s cache.

Startup and recovery speed

When a node restarts, it loads its last snapshot and replays only the newer records, so it is ready quickly. A failover is fast because standbys are already up to date. The Kafka community has reported far shorter controller failover and shutdown times, especially for clusters with very many partitions, compared with the ZooKeeper design. Exact figures depend on hardware and configuration, so measure your own environment.

Where the real limits still live

Metadata is not the only limit. Each partition still costs brokers memory, file handles, and replication work. Very large partition counts also mean longer recovery of the data itself after a broker restart. KRaft removes the controller as the main bottleneck, but good planning of partition counts is still an important part of running Kafka well.

No design is perfect. Here is an honest look at what KRaft gives you and what it asks of you.

ZooKeeper design versus KRaft design

Trade-offs to understand

The controllers hold the keys to the whole cluster, so they deserve special protection.

Whoever controls the metadata log controls Kafka. They can create or delete topics, change who is allowed to read data, and decide which brokers are trusted. This is why the controller quorum should be treated like the vault of the building, not like a normal room.

A bonus of the new design

Since the old external coordination service is gone, there is one less system to secure, one less set of passwords and certificates, and one less network path to guard. Security teams often count that as a real gain.

What to watch so you notice trouble long before your users do.

You can think of monitoring as three questions: Is there a leader? Is it stable? Is everyone keeping up? Kafka exposes many measurements that answer each one. Names and details can vary a little between versions, so treat this list as a guide to the ideas.

Useful command-line checks

Kafka ships with a tool that describes the state of the metadata quorum. It reports the current leader, the epoch, the high watermark, each voter’s position in the log, and how far behind each one is. Running it after every maintenance step is a cheap and powerful habit. Comparing the positions across voters quickly reveals a node that is quietly falling behind.

Logs to read

Controller logs tell a story: “fetch timeout expired,” “became candidate,” “granted vote,” “became leader.” If you see this story repeat every few minutes, something is wrong with a machine or a network. Send controller logs to your central logging system, and set an alert on leadership changes that happen more often than your normal maintenance schedule explains.

When something looks wrong, this table helps you go from symptom to likely cause to first check.

A calm method for any incident

How KRaft clusters are actually created, moved, and run in real environments.

Starting a fresh cluster

A new KRaft cluster begins by choosing a unique cluster ID, a random label shared by every node. Each node’s storage folder is then prepared using a formatting tool that writes this ID and the initial metadata files. On first start, the voters find each other, hold their first election, and the winner begins writing the log. After that, brokers register and the cluster is ready.

Moving an existing ZooKeeper cluster

Teams with old clusters can migrate without downtime. The process has clear phases: first upgrade to a version that supports migration, then add a new KRaft controller quorum next to the old setup. The new controllers copy the existing metadata from ZooKeeper. Then brokers are restarted one at a time so they talk to the new controllers. During a middle period, the new controllers keep writing changes to both places, which allows a rollback. Finally, once everyone is happy, the old coordination service is switched off for good.

Running on virtual machines, containers, and managed services

On virtual machines, give each controller its own instance with a persistent disk and a fixed name. On Kubernetes, controllers usually run as stateful workloads with stable names and persistent volumes, because a controller must come back with its old disk and its old identity. Operators such as Strimzi support this pattern. Managed services, including Amazon MSK on newer Kafka versions and Confluent’s cloud offering, run the controller quorum for you, which removes most of the operational work, though the ideas in this guide still explain what you see.

Habits that keep the quorum calm, and traps that keep waking people up at night.

Best practices

Useful design patterns

Anti-patterns to avoid

Common mistakes in short

• Using an even number of voters and believing it adds safety
• Setting timeouts by copy-and-paste without measuring your own network and disks
• Ignoring repeated small elections until a big outage happens
• Leaving the controller listener open to too many hosts
• Upgrading all controllers at once

The same ideas show up in famous systems that millions of people use every day.

Everything you learned, told as one simple story.

Imagine a food-delivery company called QuickBite. Its Kafka cluster has three dedicated controllers, called Ctrl-A, Ctrl-B, and Ctrl-C, placed in three different availability zones, and nine brokers spread around them. Ctrl-A is the active leader in epoch 12. Every few seconds the brokers send heartbeats to Ctrl-A, and Ctrl-B and Ctrl-C keep fetching new records from it.

A quick reference you can return to whenever a word slips your mind.

Quick answers to the questions that beginners ask most often.

The whole story in a few short lines.