Before learning what a cluster is, let us see why a single Kafka server cannot carry a serious business.

The limits of one machine

Apache Kafka is a system for storing and moving streams of events, such as orders, payments, clicks, or sensor readings. You can run Kafka on a single computer, and that is perfectly fine for learning. But a single computer has hard limits. Its disk can only hold so much. Its network card can only carry so much traffic. Its memory and processor can only serve so many readers and writers at once.

There is a second, more serious problem. A single machine is a single point of failure. If its power supply breaks, if the operating system needs a restart, or if someone trips over a cable, everything stops. Every application that depends on it stops too.

What a cluster gives you

A cluster is a group of Kafka servers that cooperate as one logical system. It gives you three gifts: more capacity because the work is divided, more speed because many machines work in parallel, and more safety because your data is copied to several machines. The rest of this guide explains how Kafka achieves each of these.

A Kafka cluster is a team of servers, called brokers, that together store events and serve the applications that write and read them.

The simple definition

Each Kafka server is called a broker. A cluster is simply several brokers that know about one another and share the job. Production clusters commonly have three brokers at the very least, and large companies run clusters with dozens or even hundreds of brokers. Each broker has its own disks, its own network address, and its own share of the data.

What the cluster is responsible for

The cluster accepts events from producers, which are applications that write data. It stores those events safely on disk. It delivers them to consumers, which are applications that read data. It also keeps copies of the data, decides which broker is in charge of which piece, and repairs itself when a broker disappears.

A handful of words describe almost everything about a cluster. Learn these and the rest becomes easy.

A first picture

The diagram below shows a small cluster with three brokers holding one topic that has three partitions. Notice that every partition has a leader on one broker and followers on the others.

flowchart TB
  subgraph B1["Broker 1"]
    P0L["Partition 0 - Leader"]
    P1F["Partition 1 - Follower"]
    P2F["Partition 2 - Follower"]
  end
  subgraph B2["Broker 2"]
    P1L["Partition 1 - Leader"]
    P0F["Partition 0 - Follower"]
    P2F2["Partition 2 - Follower"]
  end
  subgraph B3["Broker 3"]
    P2L["Partition 2 - Leader"]
    P0F2["Partition 0 - Follower"]
    P1F2["Partition 1 - Follower"]
  end
        

Partitions are the secret to how a cluster shares its load.

Slicing a topic

When you create a topic, you choose how many partitions it has. Kafka then places these partitions across the brokers of the cluster. A topic with twelve partitions in a cluster of three brokers will typically have four partitions on each broker as leaders. Because each partition is an independent, ordered log of events, different brokers can serve different partitions at the same time without getting in each other’s way.

Why this creates speed

Suppose a producer sends ten thousand events each second. If everything went to a single partition on a single broker, that broker would carry all the load. With twelve partitions spread across three brokers, each broker carries roughly a third of the traffic. The same idea works for readers, because a group of consumers can each read different partitions in parallel.

Order inside a partition

Kafka guarantees that events inside one partition are stored and delivered in the order they were written. To keep related events together, producers attach a key, such as a customer number, and Kafka sends all events with the same key to the same partition. Order across different partitions is not guaranteed, and this is a trade-off you accept in exchange for parallel speed.

Replication is what turns a group of machines into a reliable cluster.

Leaders and followers

Every partition has one leader replica and zero or more follower replicas, each on a different broker. All producers write to the leader, and by default consumers read from the leader as well. The followers continuously fetch new events from the leader and append them to their own copies. They are like students copying notes from the teacher’s board in real time.

The replication factor

The replication factor is the total number of copies. A factor of three is the most common in production, meaning one leader and two followers. With three copies, the cluster can lose one broker and still have complete data, and it can even survive losing two brokers without losing information, though it may stop accepting writes depending on your safety settings.

Different brokers, always

Kafka never places two copies of the same partition on the same broker. Putting two copies on one machine would defeat the purpose, because a single failure would remove both. This is why the replication factor can never be larger than the number of brokers in the cluster.

sequenceDiagram
  participant P as Producer
  participant L as Leader Broker
  participant F1 as Follower Broker 1
  participant F2 as Follower Broker 2
  P->>L: Send event
  L->>L: Append to own log
  F1->>L: Fetch new events
  L-->>F1: Return event
  F2->>L: Fetch new events
  L-->>F2: Return event
  L-->>P: Confirm write
        

Not every copy is equally up to date. Kafka keeps track of which ones can be trusted.

What is an in-sync replica?

The set of replicas that are fully caught up with the leader is called the in-sync replica set, often shortened to ISR. The leader is always a member. A follower stays in the set as long as it keeps up with the leader within a limited delay. If a follower becomes slow, perhaps because its broker is overloaded or its network is weak, Kafka removes it from the set until it catches up again.

Why the set matters

When the leader fails, Kafka only promotes a new leader from the in-sync set. This guarantees that the new leader has every event that was confirmed to the producer. A follower that had fallen behind might be missing recent events, and promoting it would silently lose data.

Confirmation levels for producers

Producers can choose how many copies must be safely stored before Kafka confirms a write. This gives a dial between speed and safety.

The minimum in-sync setting

Clusters also have a rule for the minimum number of in-sync replicas required to accept a write. With three replicas and a minimum of two, the cluster keeps accepting writes even if one broker is down, because two copies still confirm each event. If two brokers fail, the cluster refuses new writes rather than risk keeping only one copy. This is Kafka choosing safety over availability, and it is a wise choice for important data.

Someone must keep the master list and make decisions when things change. In Kafka, that job belongs to the controller.

What the controller does

The controller tracks which brokers are alive, which partitions exist, who leads each partition, and which replicas are in sync. When a broker disappears, the controller notices and chooses new leaders for every partition that lost its leader. When a new topic is created or partitions are added, the controller assigns them to brokers. It also spreads this updated information to the other brokers so everybody shares the same view.

The older way and the newer way

In earlier years, Kafka relied on a separate coordination service called ZooKeeper to store its metadata and elect the controller. That meant running and maintaining two different systems. Modern Kafka replaces it with a built-in mechanism called KRaft, short for Kafka Raft. In KRaft mode, a small group of nodes called the controller quorum stores the cluster’s metadata inside Kafka itself, using an agreement method known as the Raft protocol. Recent major versions of Kafka have removed ZooKeeper completely, so new clusters use KRaft.

The controller quorum

A quorum means a majority vote. The controller nodes elect one active controller among themselves, and the others stay ready as standbys that hold identical copies of the metadata. A majority must agree on any change. This is why controller groups have an odd number of members, usually three or five. With three, the group tolerates one failure. With five, it tolerates two.

flowchart LR
  subgraph Q["Controller Quorum"]
    C1["Controller 1 - Active"]
    C2["Controller 2 - Standby"]
    C3["Controller 3 - Standby"]
  end
  Q --> B1["Broker 1"]
  Q --> B2["Broker 2"]
  Q --> B3["Broker 3"]
  Q --> B4["Broker 4"]
        

The true test of a cluster is how it behaves on a bad day. Let us walk through a failure step by step.

flowchart TD
  A["Broker 2 stops responding"] --> B["Controller detects the failure"]
  B --> C["Find partitions led by Broker 2"]
  C --> D["Promote an in-sync follower to leader"]
  D --> E["Publish new leadership to all brokers"]
  E --> F["Clients redirect to new leaders"]
  F --> G["Broker 2 returns and catches up"]
        

Sometimes every in-sync copy is gone. Kafka then faces a hard choice, and understanding it prevents nasty surprises.

The dilemma

Imagine a partition with three replicas where two brokers fail in quick succession, and the third replica had fallen behind just before. There is now no in-sync replica available. Kafka can either wait for an in-sync replica to return, which keeps data safe but leaves the partition unavailable for a while, or it can promote the out-of-date replica, which restores service immediately but permanently loses the events that replica never received.

Kafka’s default stance

By default, Kafka prefers safety and waits. Allowing an out-of-date replica to become leader is called an unclean election, and it is disabled by default in modern versions. Teams enable it only for data where availability matters more than completeness, such as certain metrics or logs that can be regenerated.

Having three copies helps only if they do not all sit in the same danger zone.

Failures come in groups

Machines rarely fail alone. A whole rack of servers can lose power. A network switch can cut off every machine connected to it. In the cloud, an entire availability zone, which is a separate data-centre area within a region, can experience trouble. If all three copies of a partition live in the same rack, a single rack failure removes them all.

Rack awareness

Kafka brokers can be labelled with the rack or zone they live in. With this information, Kafka places the copies of each partition in different racks whenever possible. A cluster spread over three zones with a replication factor of three will keep one copy in each zone, so even a whole zone going dark leaves two complete copies alive.

flowchart LR
  subgraph Z1["Zone A"]
    BA["Broker 1"]
  end
  subgraph Z2["Zone B"]
    BB["Broker 2"]
  end
  subgraph Z3["Zone C"]
    BC["Broker 3"]
  end
  BA --- BB
  BB --- BC
  BA --- BC
        

With many brokers, how does an application know which one to talk to?

Bootstrap servers

Every producer and consumer is given a short list of broker addresses, called bootstrap servers. The application contacts one of them and asks for the cluster’s map, called metadata. The map lists all the brokers, all the topics, and which broker currently leads each partition. The application does not need the full list of brokers up front, only enough addresses to get started. Giving two or three is wise, so that the client can still start if one is down.

Talking directly to the right leader

Armed with the map, the producer sends each event straight to the broker leading the target partition. Consumers likewise fetch from the right brokers. If leadership changes after a failure, the client’s request is answered with a gentle “not the leader anymore” message, and the client refreshes its map and tries the new leader. This is why failures are mostly invisible to well-written applications.

sequenceDiagram
  participant App as Application
  participant Any as Any Broker
  participant Lead as Partition Leader Broker
  App->>Any: Ask for cluster metadata
  Any-->>App: Brokers, topics, and leaders
  App->>Lead: Send or fetch events
  Lead-->>App: Response
        

Clusters are meant to grow. Adding capacity is possible, but it needs a little thought.

Adding a new broker

When your traffic grows, you add more brokers. There is one important detail: a new broker does not automatically take over existing data. It joins the cluster empty. Only new topics and partitions may be placed on it. To make it useful for existing topics, you must move some partitions onto it, an activity known as partition reassignment.

Reassignment and its cost

During reassignment, Kafka copies the partition’s data from the old broker to the new one, then switches the replica over. This copying uses network and disk, and it can slow down normal traffic if done carelessly. Careful teams limit the copying speed, move a few partitions at a time, and choose quieter hours. Tools exist to automate the balancing of load, and many managed services do it in the background.

There is no single right size, but a few simple questions guide the decision.

A worked idea in words

Suppose your applications write one hundred megabytes of events every second, and you keep them for three days with three copies. That is a very large amount of storage, several hundred terabytes, spread over the brokers. Now imagine ten different consumer groups all reading that stream. Outgoing traffic is roughly ten times the incoming traffic. Seen this way, it becomes clear that disks and networks, not processors, are usually the first limits you meet.

One cluster is not always enough. Larger organisations often run many, and connect them.

Why more than one cluster?

A company may run separate clusters for different regions, so that European customers’ data stays in Europe and users everywhere enjoy low delay. Others keep a standby cluster in a distant location for disaster recovery, in case an entire region is lost. Some separate clusters by business area or by importance, so that a noisy experimental workload can never disturb the payments system.

Copying between clusters

Kafka provides tools, such as MirrorMaker and similar replication services, that continuously copy topics from one cluster to another. This copy is separate from the internal replication inside a cluster. It is usually slightly delayed, so after a regional disaster the standby cluster may be missing the last few seconds of events. Teams must plan for this gap and decide how applications resume in the other location.

A cluster carries valuable business facts, so it must be guarded and watched.

Guarding the doors

Kafka can encrypt traffic between applications and brokers, and between the brokers themselves, so that nobody listening on the network can read the events. It can require every client to prove its identity, which is called authentication. It can then apply rules about who may write to or read from each topic, which is called authorisation. Large organisations also encrypt data stored on disk and keep records of who accessed what.

Watching the vital signs

Good teams also watch offline partitions, which have no leader at all and are therefore unavailable. They watch request delays, network use, and the health of the controller. They set alerts so that a small trouble, like a broker falling out of sync, is fixed long before it grows into an outage. Dashboards showing these numbers over time also help to plan growth.

You can run a cluster yourself or let a provider do it. Each path has its own comfort and cost.

Many teams also run Kafka on container platforms such as Kubernetes, using specialised operators that automate much of the routine work. This offers a middle path, but it still requires a team that understands both Kafka and the platform underneath it.

Learning from common errors is the cheapest education there is.

Best practices at a glance

Three is the usual starting point. It allows three copies of each partition on different machines and lets the cluster survive the loss of one broker while still accepting safe writes.

No. Each copy of a partition must live on a different broker, so you cannot have more copies than brokers.

No. Each broker holds a share of the partitions. Only through replication does a piece of data live on several brokers, and even then not on all of them unless the cluster is very small.

New clusters use KRaft, where the cluster stores its own metadata. Recent major Kafka versions have removed ZooKeeper support, so older clusters must migrate before upgrading past the last version that supports it.

Another member of the controller quorum takes over, because all standbys hold the same metadata. This usually happens within moments, and data traffic keeps flowing in the meantime.

Not if the data was confirmed with sensible safety settings. Confirmed events exist on several in-sync replicas, so a follower with the complete data becomes the new leader.

Not by itself. The new broker starts empty. You must move partitions onto it before it shares the existing load.

Inside a cluster, replication keeps followers in step with leaders and is part of the write confirmation. Copying between clusters is a separate tool that mirrors topics to another location, usually with a small delay, mainly for disaster recovery or regional access.

No. Choose settings per topic based on how valuable the data is. Critical business events deserve the strongest safety, while throwaway telemetry can accept lighter settings for speed.

We began with a city library whose branches share books and back one another up. A Kafka cluster is that same idea for events. Several brokers divide topics into partitions, keep replicated copies on different machines, and elect leaders to handle traffic. A controller, backed by a quorum in modern KRaft mode, keeps the master map and reacts when brokers fail. Clients simply learn the map and talk to the right leaders. By spreading copies across racks and zones, tuning safety settings, growing carefully, and watching a few key signals, a cluster can lose machines without losing data or trust. The cluster is not magic; it is a well-organised team with clear roles and plans for bad days.