Before comparing two tools, it helps to understand the job both of them are doing.

Many machines, one system

Apache Kafka is a distributed event streaming platform, meaning it runs across many computers, called brokers, that work together as one system instead of relying on a single machine. Spreading the work across many brokers gives Kafka the ability to handle huge volumes of data and to keep working even if one broker fails. But this teamwork creates a new problem: someone has to keep track of the team itself.

Questions like which brokers are currently alive, which broker is responsible for which piece of data, and who should take charge if the current leader disappears, all need trustworthy, up-to-date answers that every broker agrees on. Getting a group of independent computers to agree on shared facts, especially when some of them might fail or lose connection at any moment, is a famously difficult problem in computer science, generally called distributed consensus.

The piece that does this job

The part of Kafka responsible for this coordination is often called the control plane, separate from the data plane that actually carries the events. Over Kafka’s history, two very different technologies have filled this control plane role: an external tool called ZooKeeper, used for many years, and a newer, built-in system called KRaft, which is what modern Kafka clusters use today.

The original coordinator Kafka relied on for over a decade.

A general-purpose coordination service

Apache ZooKeeper is a separate open-source project, not built specifically for Kafka, designed to help distributed systems coordinate with each other. It stores small amounts of shared information in a tree-like structure, similar to files and folders, and it lets many different applications read and write to that structure safely, even when several of them try at once.

For many years, Kafka used ZooKeeper to store its cluster metadata, which is information such as the list of active brokers, which broker leads each partition, topic configuration, and access control rules. Kafka brokers would constantly talk to ZooKeeper to check and update this information as the cluster operated.

Running a ZooKeeper ensemble

ZooKeeper itself runs as a cluster of its own machines, usually called an ensemble, so that it does not become a single point of failure. A typical production setup might run Kafka brokers on one set of machines and a separate ZooKeeper ensemble of three or five machines alongside it. This means operating a Kafka cluster actually meant operating two different distributed systems side by side, each with its own configuration, monitoring, and upgrade process.

flowchart LR
  subgraph KC[Kafka Cluster]
    B1[Broker 1]
    B2[Broker 2]
    B3[Broker 3]
  end
  subgraph ZE[ZooKeeper Ensemble]
    Z1[ZooKeeper Node 1]
    Z2[ZooKeeper Node 2]
    Z3[ZooKeeper Node 3]
  end
  B1  Z1
  B2  Z2
  B3  Z3
        

A working solution can still carry real costs.

Two systems to run, not one

Running Kafka with ZooKeeper meant a team had to become skilled at operating and troubleshooting two entirely separate pieces of software. Each had its own configuration files, its own memory and disk needs, its own security setup, and its own upgrade schedule. Mistakes or mismatches between the two could cause confusing failures that were hard to diagnose, because the problem might live in Kafka, in ZooKeeper, or in the connection between them.

A ceiling on how big a cluster could grow

As a Kafka cluster grew to hold more topics and partitions, the amount of metadata ZooKeeper had to track grew too. Beyond a certain point, usually somewhere in the range of a few hundred thousand partitions, ZooKeeper itself became a bottleneck, slowing down how quickly the cluster could start up, recover from failures, or apply changes. This placed a practical ceiling on how large a single Kafka cluster could comfortably grow.

Slower recovery and extra network hops

Because brokers had to communicate with an entirely separate system to check or update metadata, every coordination step involved extra network trips. When something went wrong, such as a broker crashing, the process of noticing the failure and reassigning its responsibilities took longer than it needed to, since it depended on round trips to ZooKeeper rather than being handled natively inside Kafka.

A fresh approach that keeps coordination inside Kafka itself.

What KRaft stands for

KRaft stands for Kafka Raft, named after Raft, a well-known and well-studied algorithm for reaching agreement across a group of machines, which is the same kind of consensus problem ZooKeeper was solving. Rather than reaching outside to a separate tool, Kafka’s own developers built consensus directly into Kafka, using the Raft algorithm as its foundation.

No more second system

With KRaft, the cluster metadata that used to live in ZooKeeper is instead stored and managed by a small set of Kafka nodes themselves, called controllers. These controllers are still Kafka processes, running the same software as the brokers, just configured to take on the coordination role. There is no separate ensemble to install, patch, or monitor. One system now handles both the data and the coordination.

flowchart LR
  subgraph KC[Kafka Cluster]
    C1[Controller 1]
    C2[Controller 2]
    C3[Controller 3]
    B1[Broker]
    B2[Broker]
    B3[Broker]
  end
  C1  C2
  C2  C3
  C1  C3
  C1 -.manages.-> B1
  C2 -.manages.-> B2
  C3 -.manages.-> B3
        

A short, plain-English look at the idea powering KRaft.

Electing a leader

Among the group of controllers, Raft has them elect one of themselves as the leader, while the others become followers. Only the leader accepts new changes to the shared metadata, such as “broker 2 now leads partition 5”. The leader records the change and then makes sure the majority of followers have also recorded it before treating the change as confirmed.

Majority agreement, not everyone

Raft does not need every single controller to confirm a change, only a majority. In a group of three controllers, two agreeing is enough. This matters because it means the system can keep working even if one controller is temporarily unreachable, as long as a majority remains available and able to communicate.

What happens if the leader disappears

If the leader controller crashes or becomes unreachable, the remaining controllers notice and hold a new election among themselves, choosing a new leader, usually within a very short time. Because the followers were already kept up to date, the new leader can take over with almost no loss of information.

sequenceDiagram
  participant L as Leader Controller
  participant F1 as Follower Controller 1
  participant F2 as Follower Controller 2
  L->>F1: Replicate metadata change
  L->>F2: Replicate metadata change
  F1-->>L: Acknowledge
  F2-->>L: Acknowledge
  L-->>L: Majority reached, change confirmed
        

One of the cleverest parts of KRaft is that it reuses an idea Kafka already knew well.

The metadata log

Rather than inventing a brand-new storage mechanism, KRaft stores all of its cluster metadata as a special internal topic, following the exact same append-only log design that Kafka already uses for ordinary event data. Every change to the cluster, such as a new topic being created or a partition’s leader changing, becomes an event appended to this metadata log, in order, with each one building on the last.

Why this design is elegant

This means Kafka’s engineers did not need to build a second, unrelated storage system just for coordination. The same reliable, well-tested log mechanism that carries millions of ordinary events also carries the cluster’s own internal history. Brokers can replay this metadata log to reconstruct the current state of the cluster, in the same way a consumer can replay an ordinary topic to catch up on events it missed.

A direct comparison clears up the most common confusion.

A closer look at the roles inside a KRaft cluster.

Dedicated controller nodes

In many production deployments, a small number of machines, often three or five, are set up to run only as controllers, focusing entirely on managing cluster metadata and leaving the data-handling work to separate broker machines. This separation keeps the coordination work isolated from the heavy lifting of handling producer and consumer traffic.

Combined mode for smaller setups

KRaft also allows a single machine to act as both a broker and a controller at the same time, which is called combined mode. This is popular for smaller clusters, development environments, and learning setups, since it reduces the number of machines needed to get started. Larger, production-grade clusters generally prefer separating the two roles for better stability and predictability.

Big changes to widely used software tend to happen carefully, in stages.

What teams actually notice once they move to KRaft.

Practical guidance for anyone setting up or migrating a Kafka cluster today.

Best practices

Mistakes to avoid

No, not for a new deployment. Modern Kafka releases use KRaft by default and no longer support running with ZooKeeper at all.

No. KRaft went through a long period of testing and gradual rollout before becoming the default, and it is generally faster for tasks like leader elections precisely because it avoids the extra network hops that ZooKeeper required.

Raft is a method for a group of machines to elect a leader and agree on a shared, ordered sequence of changes, even if some machines fail or lose connection temporarily.

Only temporarily, during a supported, staged migration process. A cluster is not meant to run in a mixed mode permanently; it is meant to fully transition from one mode to the other.

The event data itself, stored in topic partitions, is not restructured by the migration. The migration moves the cluster metadata, such as topic configuration and leadership information, from ZooKeeper into the new KRaft metadata log.

Raft was designed to be easier to understand and implement correctly than some earlier consensus algorithms, while still being well proven, which made it a practical and trustworthy foundation to build directly into Kafka.

It can be, for smaller or lightly loaded clusters. For larger or business-critical deployments, separating controller and broker roles onto dedicated machines is generally the safer and more predictable choice.

Every Kafka cluster needs a trustworthy way to coordinate its brokers, track cluster metadata, and recover quickly from failures. For many years this job belonged to ZooKeeper, a capable but separate distributed system that added operational overhead and placed a ceiling on how large a cluster could comfortably grow. KRaft replaced that arrangement by building coordination directly into Kafka itself, using the well-studied Raft consensus algorithm and reusing Kafka’s own trusted log design to store metadata as just another topic. The result is a single, simpler system that starts faster, recovers faster, and scales further, and it is now the standard way every modern Kafka cluster runs.