To understand the controller quorum, first understand the problem it was built to solve.

What “metadata” actually means

A Kafka cluster stores your messages in topics, and every topic is split into partitions. But besides the messages themselves, the cluster must remember a large amount of information about itself. Which brokers are alive right now? Which topics exist? How many partitions does each topic have? Which broker is the leader of each partition, and which brokers hold up-to-date copies? What are the configuration overrides, the quotas, and the access control rules? All of this “data about data” is called metadata.

Metadata is small compared to message data, but it is extremely important. If brokers disagree about who leads a partition, two brokers might both accept writes for it and your data would diverge. So Kafka needs one trusted place that stores the metadata and one trusted process that changes it.

The ZooKeeper era

For its first decade, Kafka kept its metadata in a separate system called Apache ZooKeeper, a small coordination service that runs as its own group of servers. One of the Kafka brokers was chosen as the controller by racing to create a special temporary entry in ZooKeeper. The winner became the controller, watched ZooKeeper for changes, made decisions such as electing new partition leaders, and then pushed those decisions to every other broker through direct network calls.

This design worked, and it powered huge deployments for years. But as clusters grew, its weaknesses became painful.

The road to KRaft

The Kafka community answered these problems with a proposal known as KIP-500, “Replace ZooKeeper with a Self-Managed Metadata Quorum”. The idea was bold and simple: stop storing metadata in an outside system. Instead, treat metadata as just another log, the very thing Kafka is best at, and let a small set of Kafka nodes agree on that log using a consensus protocol. The result is called KRaft, short for “Kafka Raft”.

One sentence summary: a few Kafka nodes keep a replicated log of cluster decisions, and every broker reads that log.

The three-part idea

KRaft rests on three ideas that fit together neatly.

First, metadata is an event log. Instead of storing “the current state” in a database, Kafka records every change as an event: “topic orders was created”, “broker 4 registered”, “partition 7 of orders now has leader broker 2”. The current state of the cluster is simply what you get by replaying all those events in order.

Second, a small group of nodes owns the log. These nodes are called controllers. They replicate the log among themselves and use the Raft consensus algorithm so that they always agree on its exact contents, even when machines crash or networks misbehave.

Third, everyone else follows the log. Regular brokers do not vote. They simply pull new events from the controllers and apply them to their own in-memory picture of the cluster.

flowchart TD
    subgraph Quorum["Controller quorum - voters"]
        C1["Controller 1 - active leader"]
        C2["Controller 2 - follower"]
        C3["Controller 3 - follower"]
    end
    C1 --- C2
    C1 --- C3
    C2 --- C3
    B1["Broker 1 - observer"]
    B2["Broker 2 - observer"]
    B3["Broker 3 - observer"]
    B4["Broker 4 - observer"]
    B1 -->|"fetch metadata"| C1
    B2 -->|"fetch metadata"| C1
    B3 -->|"fetch metadata"| C1
    B4 -->|"fetch metadata"| C1
    P["Producers and consumers"] -->|"data traffic"| B1
    P -->|"data traffic"| B2
        

Notice that producers and consumers never talk to controllers for their normal work. The controllers are the “brain”, and the brokers are the “hands”. Clients keep working with brokers, and brokers ask the controllers for decisions only when something about the cluster itself changes.

What lives in the metadata log

The metadata log is stored as an ordinary Kafka topic-style log with one partition. Its internal name is __cluster_metadata. Each entry in it is a small record with a type. Some examples are records that register a broker, records that create or delete a topic, records that change partition leadership or the set of in-sync replicas, records that store configuration and quota changes, records that hold access control entries, and records that store feature versions.

Every record has an offset, which is just its position in the log: 0, 1, 2, and so on. Offsets give the cluster a shared clock. When a broker says “I have applied the metadata up to offset 48,210”, every other node knows exactly what that broker knows.

Every new word below is explained once here and then used consistently for the rest of the tutorial.

A KRaft cluster has just a few moving parts. Knowing each one makes everything else easy.

Process roles

Every Kafka node declares what job it does through a setting called process.roles. The value can be broker, controller, or both. A node with only the controller role stores the metadata log and takes part in elections but does not host any topic data. A node with only the broker role serves clients and follows the metadata log. A node with both roles does both jobs in one process, which is called combined mode.

The components inside a controller

How nodes find each other

In the classic static setup, every node is given the exact list of voters through controller.quorum.voters, written as a set of node id, host, and port entries. Newer versions also support controller.quorum.bootstrap.servers, which is just a list of addresses used to discover the current quorum, and this is what makes changing the quorum at runtime possible. Each node also has a unique node.id and, before first use, the storage directory must be formatted with a shared cluster id so that nodes from different clusters can never be mixed by mistake.

Before Raft makes sense, you need to feel the problem it solves.

The problem in plain words

Suppose three computers must all remember the same list of decisions. Computers can crash, restart, freeze for a few seconds, or lose their network cable. Messages between them can be delayed, duplicated, or dropped. Yet the list must never contain two different versions of the same decision, and a decision that was announced as final must never disappear. This challenge is called distributed consensus.

Two safety promises

Why a majority solves it

Here is the trick that makes Raft work. Any decision requires the approval of a majority. Any two majorities of the same group must overlap in at least one member. That overlapping member has seen both decisions, so it can prevent two conflicting decisions from both succeeding. It is the same reason two different winning coalitions in an election can never both include entirely different sets of voters.

The three jobs of Raft

Raft breaks consensus into three understandable jobs. Leader election chooses one node to be in charge. Log replication makes that leader copy its log to the others. Safety rules make sure that only a node with the most complete log can win an election, so committed entries are never lost. The next chapters show how Kafka performs each job in its own particular style.

Kafka did not copy Raft blindly. It bent the protocol to match the tools Kafka already trusts.

Push versus pull

In the original Raft paper, the leader pushes new entries to followers with a message usually called “append entries”. Kafka’s version turns this around. Followers pull data from the leader by sending ordinary fetch requests, exactly like a Kafka consumer or a Kafka replica follower does. The leader never has to remember what each follower has or chase after it.

Why pull made sense for Kafka

Elections are how the quorum picks the one node allowed to write to the metadata log.

The states a controller can be in

At any moment, a voter is playing one of a few roles. It helps to think of them as costumes that a node changes as events happen.

What triggers an election

A follower expects to hear from the leader regularly, because it keeps sending fetch requests and getting answers. If the leader stops responding for longer than the fetch timeout, which defaults to a couple of seconds, the follower assumes the leader is gone. It then waits for a small random delay and becomes a candidate. The random delay is called the election backoff, and it is the secret that prevents all followers from becoming candidates at the exact same moment and splitting the vote.

The election in five steps

sequenceDiagram
    participant C1 as Controller 1 (old leader, down)
    participant C2 as Controller 2
    participant C3 as Controller 3
    Note over C2,C3: Fetches to Controller 1 time out
    Note over C2: Random election backoff expires first
    C2->>C2: Epoch 5 to 6, vote for self
    C2->>C3: Vote request (epoch 6, last log epoch and offset)
    C3->>C3: Check epoch, no earlier vote, candidate log is complete enough
    C3-->>C2: Vote granted
    Note over C2: 2 of 3 votes is a majority
    C2->>C3: Begin quorum epoch 6, I am leader
    C3->>C2: Fetch from offset N
    Note over C2,C3: Controller 1 later returns, learns epoch 6, becomes follower
        

The “most complete log wins” rule

This rule is the heart of Raft’s safety. A voter refuses to support a candidate whose log is behind its own. Since a committed entry has been stored on a majority, and a winning candidate needs a majority of votes, at least one voter in every winning vote has the committed entry. That voter will refuse anyone who lacks it. So a candidate that is missing committed entries can never win.

Graceful stepping down

When an operator stops the active controller in a planned way, the leader sends an end-quorum-epoch message that names the preferred successors. The other voters can then start an election immediately instead of waiting for a full timeout. This is why planned restarts of controllers cause a much smaller pause than crashes do.

The epoch is the single most important safety device in the whole protocol.

What an epoch does

Every message between controllers carries an epoch number. The rule is short and strict: if you receive a message with an epoch lower than your own, you reject it and tell the sender the current epoch. If you receive a message with a higher epoch, you immediately accept that a newer era has started, and you adjust yourself. That is all. But this tiny rule blocks a surprising number of dangerous situations.

The zombie leader problem

Imagine the leader is not dead, only frozen for a few seconds by a long garbage collection pause or a network hiccup. The others give up, elect a new leader in a higher epoch, and continue. Then the frozen leader wakes up and still believes it is in charge. If it tried to write, we would have two leaders. This is dangerous, and it is called a zombie leader, or split brain.

Kafka’s design stops this in several layers:

flowchart TD
    A["Leader in epoch 5 freezes"] --> B["Others time out and elect new leader in epoch 6"]
    B --> C["Old leader wakes up, still thinks epoch 5"]
    C --> D["Old leader tries to replicate an entry"]
    D --> E{"Do followers accept epoch 5?"}
    E -->|"No, epoch too old"| F["Old leader receives epoch 6 in the rejection"]
    F --> G["Old leader becomes follower, drops uncommitted entries"]
    E -->|"Never happens"| H["Split brain avoided"]
        

Epochs also fence brokers and partitions

The same idea appears at other layers of Kafka. Every broker registration carries a broker epoch, so a broker that crashed and restarted cannot be confused with its former self. Every partition has a leader epoch, so a stale partition leader cannot get away with old decisions. Once you notice this pattern, Kafka’s design feels consistent: wherever there is a “who is in charge”, there is a number that grows with each new answer.

This chapter follows one decision, from a request to a committed record that the whole cluster can trust.

The life of a write

Let us follow the creation of a new topic. An administrator sends a create-topic request to any broker. The broker forwards it to the active controller. The controller checks that the request is valid, decides how many partitions and which brokers should host replicas, and turns all of that into metadata records. Only then does the log come into play.

sequenceDiagram
    participant Admin as Admin client
    participant B as Any broker
    participant L as Active controller (leader)
    participant F1 as Standby controller 1
    participant F2 as Standby controller 2
    Admin->>B: Create topic request
    B->>L: Forward request
    L->>L: Validate and build metadata records
    L->>L: Append to own log
    F1->>L: Fetch from offset N
    L-->>F1: Records N to N+3
    F2->>L: Fetch from offset N
    L-->>F2: Records N to N+3
    F1->>L: Fetch from offset N+4 (proves it stored N to N+3)
    Note over L: Majority stored, high watermark moves to N+3
    L->>L: Apply records to state
    L-->>B: Success
    B-->>Admin: Topic created
    F2->>L: Fetch, learns new high watermark
        

Why the fetch request doubles as an acknowledgement

Notice something clever in the diagram. A follower never sends a special “I got it” message. When the follower asks for the records starting at offset N+4, it is implicitly saying “I have safely stored everything before N+4”. The leader reads that number from each fetch and uses it to compute the majority position. The same message thus serves as a heartbeat, a request for data, and an acknowledgement. Fewer message types mean fewer places for bugs.

What “committed” really means

A record is committed when it is stored on a majority of voters. From that moment, the protocol guarantees it will appear in the log of every future leader. The high watermark is simply the offset of the last committed record. Observers, such as brokers, only ever apply records up to the high watermark, because anything beyond it might still be thrown away if the leader fails.

Batching for speed

The controller does not write one record at a time. It gathers records into batches, and one batch can carry many changes, such as the creation of a topic with fifty partitions. Batching reduces the number of disk flushes and network round trips. This is one reason a KRaft controller can handle changes to very large numbers of partitions far faster than the old design.

After a leader change, some nodes may hold entries that never made it to safety. Here is how they get cleaned up.

How logs drift apart

Suppose the leader in epoch 5 appends entries 100 to 103 to its own log, but crashes before any follower fetches them. A new leader is elected in epoch 6 from one of the followers, whose log ends at entry 99. The new leader then appends its own entries 100 and 101 with epoch 6. Now the old leader’s disk holds different data at positions 100 and 101 than the rest of the cluster. Those old entries were never committed, so throwing them away breaks no promise.

The fetch-based fix

When the returning node fetches from the new leader, its fetch request includes two extra facts: the offset it wants next, and the epoch of the last entry it holds. The leader compares this against its own log. If they match, it simply returns the next entries. If they do not match, the leader replies with a special note that says, in effect, “your log diverges from mine, and the last point where we agree is at this epoch and offset”. The follower truncates its log back to that point and fetches again from there.

flowchart TD
    A["Returning node sends fetch: next offset 104, last epoch 5"] --> B{"Does leader log agree at that position?"}
    B -->|"Yes"| C["Leader returns the next records"]
    B -->|"No"| D["Leader replies with the largest agreeing epoch and offset"]
    D --> E["Node truncates its log to that point"]
    E --> F["Node fetches again from the agreed offset"]
    F --> C
        

This is the same technique Kafka uses for ordinary partition replication, which is why the code could be shared. It jumps to the correct point in one or two round trips instead of stepping back one entry at a time.

Why committed data is never truncated

The truncation only ever removes entries that were not stored on a majority. Committed entries exist on a majority, and the election rule guarantees the new leader has them, so the agreed point is always at or after the last committed entry. A follower will never be asked to delete something that was reported as committed.

Raft only agrees on a log of records. Something must give those records meaning. That something is the controller’s state machine.

A single-threaded brain

The active controller processes every request through a single event queue, one event at a time, on one thread. This may sound slow, but it is the opposite. Because only one thing ever changes the state at any moment, there are no locks, no race conditions, and no confusing partial states. The speed comes from batching many changes into each event and from keeping the whole state in memory.

Two speeds of truth

The controller keeps two versions of the truth. The first is the latest state including uncommitted records, which it uses to validate the next request. This lets it accept a burst of requests without waiting for each one to be committed before checking the next. The second is the committed state, which reflects only what a majority has stored. The controller never answers a client with success until the relevant records are committed, so clients never see a decision that later vanishes.

If the leader loses its position, the uncommitted version is discarded and rebuilt from the committed log. This is safe, because nobody was ever told those changes had succeeded.

What the controller is responsible for

Standby controllers are always warm

Every voter, leader or not, replays committed records into its own in-memory image. So a standby is never cold. When the leader fails and an election completes, the new leader already knows the entire cluster state. It only needs to be careful about uncommitted entries at the tail of its log. This is the main reason controller failover dropped from many seconds in the old world to a very brief pause today.

Brokers do not vote, but they are the reason the quorum exists. Here is how they stay in step.

Joining the cluster

When a broker starts, it first reads its local copy of the metadata log, if any, and then contacts the controllers through the controller listener. It sends a registration request that includes its id, its network endpoints, its rack, the features it supports, and the ids of the storage directories it owns. The active controller writes a record for this registration into the log. Once the record commits, the broker officially exists and receives a fresh broker epoch.

Staying alive: the heartbeat

After registering, a broker sends periodic heartbeats to the active controller. Each heartbeat carries the broker’s current metadata offset, meaning how far it has applied the log. The controller uses this to decide when the broker is caught up. Only a caught-up broker is allowed to move from the fenced state into the active state, where it can lead partitions.

flowchart LR
    A["Starting"] --> B["Registered and fenced"]
    B -->|"caught up with metadata log"| C["Active"]
    C -->|"heartbeat lease expires"| B
    C -->|"controlled shutdown requested"| D["Shutting down"]
    D -->|"partition leadership moved away"| E["Stopped"]
    B -->|"never catches up"| B
        

Fencing

If a broker misses its heartbeat for longer than the session timeout, which defaults to about nine seconds, the controller fences it. A fenced broker is removed from leadership and from every in-sync replica set. The controller writes records for these changes, and other brokers learn about them from the log. A fenced broker that is actually still running will notice that it can no longer reach the controller or has been told it is fenced, and it will stop acting as a leader.

How metadata reaches brokers

Brokers do not receive pushes. Each broker runs a component that fetches new records from the controller quorum, applies them to its own image, and hands changes to local subsystems, such as the replica manager, the group coordinator, and the authorizer. Because everything flows through the log, each broker’s state is always “the log up to offset X”, a clean, comparable fact.

If a broker has fallen very far behind, or is brand new with no local copy, it does not need to replay millions of old records. It asks for the latest snapshot and then continues from there. The next chapter explains snapshots in detail.

A log that only grows would eventually swallow the disk and make new nodes wait forever. Snapshots solve both problems.

The problem with an endless log

Every change ever made to the cluster is in the metadata log. After months or years there may be millions of records, many of them describing things that no longer matter, such as a topic that was created and deleted, or a leader that changed a hundred times. Replaying all of that on every restart would be wasteful.

What a snapshot is

A snapshot is a single file that captures the complete cluster state as it was at a particular offset and epoch. Think of it as the “current picture”, rather than the “history of every change”. Once a snapshot exists, all log entries before its offset are no longer needed to rebuild the state, and can be deleted to free space.

When snapshots are taken

Each node, leader or follower, takes its own snapshots from its own committed state. Two settings guide the timing: one based on how many bytes of new records have accumulated since the last snapshot, and one based on elapsed time. Whichever condition is met first triggers a new snapshot. Because the snapshot is produced from committed data only, it is always safe.

Serving snapshots to slow or new nodes

Suppose a controller or broker asks for records that the leader has already deleted because a snapshot replaced them. The leader responds with the identity of its latest snapshot, telling the follower, in effect, “those entries are gone, here is the snapshot that includes them”. The follower then sends fetch-snapshot requests to download the snapshot in pieces, loads it, and resumes normal fetching right after the snapshot’s offset.

sequenceDiagram
    participant N as New or lagging node
    participant L as Leader
    N->>L: Fetch from offset 10
    L-->>N: Offset 10 is gone, use snapshot at offset 500000, epoch 7
    N->>L: Fetch snapshot piece 1
    L-->>N: Snapshot bytes
    N->>L: Fetch snapshot piece 2
    L-->>N: Snapshot bytes
    Note over N: Snapshot loaded, state rebuilt
    N->>L: Fetch from offset 500001
    L-->>N: Newest records
        

Where it is stored

Snapshots live in the same directory as the metadata log, as files named after the offset and epoch they represent. Older snapshots are cleaned up once newer ones exist and are safely in place. The size of a snapshot is proportional to the amount of current cluster state, not to the amount of history, so it grows with the number of partitions, topics, configurations, and access rules, but not with age.

How many controllers should you run, and what exactly happens when some of them fail?

The rule of thumb

Most production clusters run three dedicated controllers. Clusters that must survive two simultaneous controller failures, or that plan maintenance while staying resilient, use five. More than five is rarely useful, because each additional voter adds coordination work and slightly slows commits while only marginally improving availability.

Placing voters across failure domains

Fault tolerance is only real if failures are independent. Three controllers on three virtual machines that share one physical host fail together. A good design places each voter in a different failure domain, such as separate racks or availability zones. With three zones, one voter per zone tolerates the loss of an entire zone. With only two zones, the zone holding the majority becomes a single point of failure for metadata writes, so two-zone deployments need careful thought.

flowchart TD
    subgraph ZA["Zone A"]
        CA["Controller 1"]
        BA["Brokers 1 and 2"]
    end
    subgraph ZB["Zone B"]
        CB["Controller 2"]
        BB["Brokers 3 and 4"]
    end
    subgraph ZC["Zone C"]
        CC["Controller 3"]
        BC["Brokers 5 and 6"]
    end
    CA --- CB
    CB --- CC
    CA --- CC
    BA --> CA
    BB --> CB
    BC --> CC
        

Latency between voters

Every commit needs acknowledgements from a majority. If voters are in far apart locations, each commit waits for the round trip to the nearest majority. Metadata changes are infrequent compared to data traffic, so moderate latency between zones in one region is fine. Stretching a quorum across continents is a different story: elections and commits become slow, and timeouts may need careful tuning. For multi-region resilience, most teams run independent clusters and replicate data between them rather than one giant quorum.

Understanding what breaks, and what keeps running, is what makes a good on-call engineer.

The two planes

Kafka has a control plane, which is the controller quorum and its decisions, and a data plane, which is the brokers moving message data. They are loosely coupled. A healthy data plane can run for a while without the control plane, because leaders and replica lists already exist. But many normal events require a control plane decision, so the data plane cannot run forever without it.

What still works when the quorum has no majority

What stops working

flowchart TD
    A["Quorum loses majority"] --> B["Metadata writes stop"]
    B --> C["Healthy partitions keep serving clients"]
    B --> D["Topic changes fail"]
    B --> E["Broker failure cannot be handled"]
    E --> F["Affected partitions become unavailable"]
    B --> G["ISR changes cannot be committed"]
    G --> H["Strict acks may stall producers"]
    A --> I["Majority restored"]
    I --> J["New leader elected, backlog processed"]
    J --> K["Normal operation resumes"]
        

Recovery when a majority is lost for good

If two of three controllers are permanently destroyed, restarting the survivors will not help, because a majority is impossible without them. The safe recovery approach is to bring back replacement voters with a copy of the surviving node’s metadata, following the documented procedure for your Kafka version, or to restore from a known-good backup of the metadata directory. This is why keeping copies of the controller data directory, and rehearsing the procedure, is a valuable insurance policy.

Early KRaft had a fixed list of voters. Newer versions make the group flexible and more stable.

The limit of a static quorum

In the first design, the voter list was written into every node’s configuration. To replace a failed machine or to grow from three to five voters, an operator had to edit configs and restart nodes in a delicate order. A mistake could break the majority arithmetic.

Dynamic quorum reconfiguration

Newer Kafka releases allow controllers to be added or removed while the cluster runs. The current voter set is itself stored in the metadata log as special control records, so the quorum agrees on its own membership using the very same Raft rules. Nodes use a bootstrap server list only to discover the current quorum, and each voter has a unique directory identifier in addition to its node id, so a machine that lost its disk cannot be mistaken for its previous self.

sequenceDiagram
    participant Op as Operator
    participant L as Leader
    participant V as Existing voters
    participant N as New controller
    N->>L: Fetch log as observer
    L-->>N: Records or snapshot
    Op->>L: Add voter request for N
    L->>L: Append voter set record
    V->>L: Fetch, store the record
    Note over L: Majority stored, record committed
    L-->>Op: Voter added
    Note over N: N now counts toward majorities
        

Pre-vote: avoiding disruptive elections

A subtle weakness in basic Raft is that a node cut off from the network keeps timing out and raising its epoch. When it reconnects, its high epoch forces the healthy leader to step down, causing a needless election even though the cluster was fine. The remedy, available in newer releases, is a pre-vote phase: before increasing its epoch, a would-be candidate first asks the others “would you vote for me if I ran?”. Voters that still hear from a healthy leader say no, so the isolated node never raises the epoch and never disturbs the cluster.

Where the controllers run, and how a cluster travels from the old world to the new one.

Isolated versus combined deployment

Controller machines are usually modest. They need reliable low-latency disks, stable networking, and enough memory to hold the metadata image, but not the large disks and heavy network of a data broker.

Migrating a ZooKeeper cluster

Existing ZooKeeper-based clusters can move to KRaft without downtime through a staged migration. The idea is to add a KRaft quorum beside the running cluster, let it take over the role of controller while both worlds hold the same metadata, and then finish by removing ZooKeeper.

The controllers hold the keys to the entire cluster, so they deserve the tightest protection.

Why controllers are sensitive

Whoever can write to the metadata log can create topics, change access rules, alter configurations, and reassign partitions. A stolen or forged controller identity would give an attacker full control. Access to the metadata log files also reveals the structure of the entire cluster, including access rules and configuration values.

A quorum is quiet when it is healthy, so you need the right signals to notice trouble early.

Questions your dashboards should answer

Good monitoring for the controller quorum answers a handful of simple questions. Is there exactly one active controller? Do all voters agree on who the leader is and on the current epoch? Is every voter keeping up with the leader? Are elections rare? Are brokers applying the latest metadata? Are disks fast enough?

Tools that ship with Kafka

Logs worth watching

Controller logs record every state change of the Raft manager: becoming candidate, granting a vote, learning about a new leader, resigning, and truncating a diverging log. Repeated messages about vote requests, timeouts to a particular peer, or a node repeatedly becoming a candidate are early warnings of network problems or overloaded machines. Messages about fetch timeouts from brokers point to slow links between brokers and controllers.

flowchart TD
    A["Alert: no active controller"] --> B{"Do most voters respond?"}
    B -->|"No"| C["Check network, machines, disks on voters"]
    B -->|"Yes"| D{"Do voters agree on epoch?"}
    D -->|"No"| E["Look for partitioned or isolated voter, check timeouts and firewall rules"]
    D -->|"Yes"| F["Check election timing and pauses on the candidate"]
    C --> G["Restore majority, watch the election complete"]
    E --> G
    F --> G
        

Some choices repeatedly prove wise, and some repeatedly cause outages.

Patterns worth copying

Anti-patterns to avoid

A checklist of habits that keep controller quorums calm, and the errors that disturb them.

A safe rolling restart, explained

Tuning timeouts carefully

The defaults for election and fetch timeouts suit most networks in a single region. Raising them makes elections rarer but leaves the cluster leaderless longer during a real failure. Lowering them speeds up failover but risks false elections during a garbage collection pause or a brief network stall. Change them only with measurements in hand, and keep the relationship sensible: the fetch timeout must comfortably exceed the time it takes a healthy voter to make and answer a fetch, including jitter.

Opening the metadata directory of a controller shows how the ideas of this tutorial become real files.

Three kinds of files

A controller’s metadata directory holds three families of files that work together. The first is a set of log segments, which contain the metadata records in order. The second is a set of snapshot files, each one a full picture of the cluster state at a given offset and epoch. The third is the tiny quorum state file, which remembers the node’s voting history. Together, they are everything the node needs to rejoin the cluster after a restart.

Why the quorum state file must be trusted

Raft’s safety depends on a simple promise: a voter grants at most one vote per epoch. If a voter could forget its vote after a crash, it might vote for one candidate, restart, and then vote for another in the same epoch. Two candidates could each collect a majority, and two leaders would share an epoch. That is why the quorum state is written to disk before the vote is announced, and why manual edits or restores of old copies of this file are so dangerous.

Durability and the meaning of “stored”

When this tutorial says that a follower “has stored” a record, it means the record was written in a way that survives a crash of that machine. The majority argument only holds if the records genuinely exist on stable storage when the acknowledgement is sent. If a node acknowledged data that lived only in memory, then a simultaneous power loss on several nodes could erase a record that was already announced as committed. This is why the choice of disks, the flushing behavior of the file system, and the reliability of the underlying hardware matter far more for controllers than raw disk size.

How the log shrinks

Log segments are retained until a snapshot covers them and until every voter and the known observers no longer need them. The leader keeps enough of the log to serve normally lagging followers with plain fetches, and relies on snapshots for anyone who has fallen further behind. Deleting whole segments is a cheap file operation, and it happens in the background without blocking the write path.

flowchart LR
    A["Records arrive in log segments"] --> B["High watermark advances"]
    B --> C["State machine applies committed records"]
    C --> D{"Enough bytes or time since last snapshot?"}
    D -->|"Yes"| E["Write snapshot to temporary file"]
    E --> F["Rename to final snapshot name"]
    F --> G["Old segments covered by snapshot become deletable"]
    D -->|"No"| A
        

A controller is small, but it is not free. This chapter shows how to size it and what to watch as the cluster grows.

What actually grows with the cluster

The demands on the controller tier do not grow with the amount of message traffic. They grow with the amount of metadata. More partitions mean a bigger in-memory image, bigger snapshots, and more records to replay. More brokers mean more heartbeats to process. More topics, more access rules, and more configuration overrides all add entries. A cluster that moves enormous amounts of data across a modest number of partitions barely troubles its controllers, while a cluster with a very high partition count and frequent topic churn keeps them busy.

Sizing checklist

The timeout family

Several timeouts govern the behavior of the quorum, and it helps to see them as a small family that must be consistent with each other.

Capacity signals that say “grow now”

The same ideas play out in very different kinds of organizations.

Yes. Whichever voter currently wins the election is the Raft leader and also plays the role of active controller. The other voters are standby controllers that follow the log.

No. Brokers are observers. They read the metadata log but never vote and never count toward a majority. Only nodes listed as voters in the quorum can vote.

Ordering is essential. Every decision must be applied in exactly the same order everywhere, and a single ordered log gives that guarantee without any extra coordination between partitions.

Usually nothing noticeable. Clients talk to brokers for data, and brokers keep serving existing partitions. Only actions that need a metadata change, such as creating a topic, may wait briefly until the new leader is elected.

You can, but it is worse than one. A majority of two is two, so losing either node stops all metadata writes. Two voters double the chance of an outage without adding any tolerance.

Because an even number raises the majority bar without raising the count of failures you can survive, and a perfectly even network split leaves neither side with a majority.

An offset is a position in the log, counting records. An epoch is a counter for leader reigns, increasing with each new election. Together, an epoch and offset pair identifies a precise point in the history of the log.

Those systems follow the paper closely, with the leader pushing entries out. Kafka’s version has followers pull entries using fetch requests, reuses Kafka’s own log storage, treats non-voting observers as a natural part of the design, and uses the word epoch instead of term.

Neither. Recovery and scaling of the control plane are considerably better, and the safety rules come from a well-studied consensus algorithm. The data path for producing and consuming is unchanged.

The node will fail to append or snapshot, become an unreliable voter, and may crash. Give the metadata directory enough space and alert on free space early, since a lost voter reduces your safety margin.

In newer releases, yes. The voter set is stored in the log and can be changed one member at a time while the cluster continues to work. Older static setups required config edits and restarts.

Recent major releases no longer support it, so staying on it means staying on older versions that will stop receiving fixes. Planning a migration to KRaft is the sensible path.

The current epoch, the leader the node knows about, and whom the node voted for. Remembering these across restarts prevents a node from voting twice in one epoch, which is a crucial safety rule.

If a follower falls behind or dies, the leader must ask the controller to remove it from the in-sync list. Without a working controller, the list cannot change, and strict acknowledgement settings keep waiting for the missing follower.

A node has a fixed set of jobs chosen through its process roles. To make a machine a voter, run it with the controller role, let it catch up as an observer, and then add it to the voter set through the supported membership change in newer releases. A pure broker does not become a voter by itself.

A fresh leader is careful about entries from earlier epochs that it holds but that were never confirmed as stored on a majority. It first appends a marker record in its own epoch, and only once a majority stores that marker does everything before it become committed. This small rule avoids a rare but real safety hazard described in the Raft research.

As soon as the change is committed and the next fetch from each broker arrives, which is normally a small fraction of a second in a healthy cluster. Brokers apply records strictly in order, so a broker never sees a later change before an earlier one.

After the broker misses its heartbeats for longer than the session timeout, the controller fences it, removes it from in-sync replica lists, and elects new leaders for the partitions it led, choosing from the remaining in-sync replicas. All of these changes are written as records so every broker learns them.

A slow voter is not fatal, because only a majority is needed. But it reduces your safety margin, since the cluster then depends on the other voters staying healthy. Investigate a persistently slow voter quickly rather than living with it.

Because followers already need to contact the leader constantly to get new data. Using those same requests to show that both sides are alive avoids extra message types and keeps the protocol small, which means fewer opportunities for bugs.

The controller quorum manages cluster-wide facts, such as brokers, topics, and partition leaders. A group coordinator is a role run by a broker, which manages membership and offsets for one consumer group. They are separate systems that both rely on the metadata to know where things live.

Ask the metadata quorum tool for the current status. Look for a single leader, the same epoch reported everywhere, a high watermark that advances, and a small, steady lag for every voter and every observer. Then compare with your monitoring dashboards for election counts and commit latency.

Kafka replaced an outside coordination service with a small group of its own nodes that keep one shared, ordered record of every cluster decision. Those nodes, the controller quorum, agree on that record using a pull-based version of the Raft algorithm. One voter leads at a time, others copy its log by fetching, a majority must store each record before it counts as committed, and epochs make sure that an old leader can never override a new one. Brokers simply follow the log, snapshots keep it small, and newer releases let the group change its own membership and resist disruptive elections. Understanding these few ideas lets you size, secure, monitor, upgrade, and troubleshoot a Kafka cluster with confidence.