Scaling is not one thing. It is a family of answers to different kinds of pressure.

Apache Kafka is a platform that stores and moves streams of events, such as payments, clicks, or sensor readings. A group of Kafka servers working together is a cluster, and each server in it is called a broker. Producers write events to the cluster, and consumers read them back. When we say a cluster must scale, we mean it must handle more of something without getting slower or less reliable.

The important word is something. Growth can arrive in several forms, and each form stresses a different part of the system. A cluster that handles a million small events per second may fall over when asked to store a year of history. A cluster that stores huge amounts of data may struggle when ten thousand consumers connect. Naming the pressure correctly is the first scaling skill.

You cannot scale what you cannot picture. Here is the minimum anatomy you need.

A topic is a named stream, such as orders. To spread the work, a topic is split into partitions, and each partition is an ordered log stored on disk. Every event in a partition has an offset, a sequence number that grows by one. Ordering is guaranteed only inside a single partition, not across the whole topic.

Each partition is stored on several brokers for safety. The number of copies is the replication factor, and a common choice is three. One copy is the leader, which handles all reads and writes for that partition, and the others are followers that copy the leader’s log. The followers that are fully caught up are called the in-sync replicas. If the leader fails, one of the in-sync followers takes over.

The Controller and Cluster Metadata

Someone must decide which broker leads which partition, and record the answer. This work belongs to the controller. In older Kafka versions, the controller depended on a separate coordination service called ZooKeeper. Newer versions use a built-in quorum of controllers called KRaft, and modern releases have removed ZooKeeper entirely. Either way, the controller keeps the cluster’s metadata, meaning the list of topics, partitions, leaders, and configuration.

flowchart TD
  P["Producers"] --> B1["Broker 1"]
  P --> B2["Broker 2"]
  P --> B3["Broker 3"]
  B1  B2
  B2  B3
  B1  B3
  B1 --> C["Consumer group"]
  B2 --> C
  B3 --> C
  K["Controller quorum: metadata and leader decisions"] -.-> B1
  K -.-> B2
  K -.-> B3
        

Good scaling is triggered by evidence, not by hunches or anxiety.

A healthy cluster tells you when it is running out of room, if you know where to listen. The trick is to watch trends and headroom rather than waiting for failures. Scaling under pressure is riskier, because moving data adds load at exactly the moment the cluster is already struggling.

The Main Health Indicators

Also watch growth, not just level. A cluster at fifty percent disk that is growing ten percent a month has a very different future from one that is stable. Multiply the growth rate by the time it takes your organization to buy hardware, get approval, and complete data moves. If that total is longer than the time until you hit the limit, you are already late.

The simplest idea is to give each broker more power. It helps, but only up to a point.

Vertical scaling means upgrading the machines you already have: more processor cores, more memory, faster disks, or a faster network card. It is attractive because the architecture stays the same. The number of brokers does not change, and neither does the way partitions are placed.

What Each Resource Buys You

Adding brokers is Kafka’s natural way to grow, with one surprise that catches nearly everyone.

Horizontal scaling means adding more brokers to the cluster and spreading the work across them. It increases total throughput, storage, and network capacity together, and it does not run into a single machine’s ceiling. This is why Kafka is described as a scale-out system.

Now for the surprise. When you add a new broker to a running cluster, it joins the cluster but holds nothing. Existing partitions stay exactly where they were. The new broker will sit idle, while the old brokers stay as busy as before. Kafka does not automatically move data onto new servers, because moving data is expensive and it wants the operator to choose when.

Newly created topics are different. When a topic is created after the new brokers join, Kafka can place its partitions across all brokers, including the new ones. That is why some teams grow the cluster gradually just by letting new workloads land on the new machines. However, existing hot topics still need to be moved explicitly.

Moving a partition is a careful copy-then-switch dance, not a simple cut and paste.

Partition reassignment is the operation that changes which brokers hold a partition’s replicas. It is the workhorse of all horizontal scaling. Understanding its steps helps you predict its cost and avoid surprises.

Suppose a partition currently lives on brokers one, two, and three, and you want it on brokers one, two, and four. Broker four must first receive a full copy. It fetches the entire log from the current leader, just like an ordinary follower, and the amount of data can be huge for busy topics. During this phase the partition temporarily has four replicas, and the new one is not yet in sync.

Once broker four has caught up and joined the in-sync set, the controller removes broker three from the replica list. Broker three then deletes its now-unneeded copy. At no point during the move does the partition lose its safety, because the old copies remain until the new one is ready. That is the design’s great strength, and its cost is the temporary extra load.

sequenceDiagram
  participant Op as Operator
  participant Ctl as Controller
  participant L as Leader broker
  participant N as New broker
  participant O as Old broker
  Op->>Ctl: Request new replica placement
  Ctl->>N: Become follower of the partition
  N->>L: Fetch full log history
  L-->>N: Stream data until caught up
  N->>Ctl: Now in sync
  Ctl->>O: Remove replica from the set
  O->>O: Delete old copy
        

The Hidden Costs of Moving Data

Throttling and Batching

To protect production traffic, operators apply a replication throttle, which caps the bandwidth used for moving data. A low throttle is safe but slow, and a high throttle is fast but can hurt clients. The right value comes from watching latency during a small trial move. Operators also split a large rebalance into batches, finishing and checking each one before starting the next.

Partitions are the unit of parallelism, so getting their number right is the most important scaling decision.

Partitions decide how widely a topic’s work can be spread. A partition is handled by one leader at a time and read by only one consumer in a group at a time. Therefore, the number of partitions caps both how many brokers can share a topic’s load and how many consumer instances can read it in parallel.

A Simple Way to Size a Topic

A well-known method is to measure the throughput a single partition can deliver on the producing side and on the consuming side. Then divide the target throughput by each number and take the larger result. For example, if a partition can accept ten megabytes per second from producers, but a consumer can process only five megabytes per second from a partition, and the goal is a hundred megabytes per second, you need at least twenty partitions because the consuming side is the limit. Add headroom for growth so that you do not need to change the count soon.

Increasing Partitions Later

Kafka allows the partition count of an existing topic to be increased. It does not allow it to be decreased. Growing a topic seems harmless, but it has an important side effect for keyed data. Events with the same key are normally placed in a partition chosen by a formula that depends on the number of partitions. When that number changes, the formula sends the same key to a different partition. New events for a key may land somewhere different from its old events, so ordering for that key is no longer guaranteed across the change.

Sometimes the cluster is fast enough but simply too full.

Storage need is easy to predict. It equals the data rate, multiplied by the retention time, multiplied by the replication factor, with headroom on top. Each of those terms is a lever. You can store less by reducing the rate, shortening retention, lowering replication for non-critical topics, or compressing data. You can store more by adding disks, adding brokers, or moving old data to cheaper storage.

Levers That Reduce Storage Pressure

Tiered Storage

Tiered storage is a feature that separates recent data from old data. The newest segments stay on the brokers’ local disks for fast access, while older segments are copied to a cheaper, larger remote store such as object storage. Consumers reading recent data are unaffected, and consumers replaying old data read from the remote tier, usually with higher latency. It became production-ready in recent Kafka releases, but availability and limits depend on the version and on the plugin used for the remote store.

The network often runs out before the disks do, because one byte in becomes many bytes moving.

Every byte a producer writes is copied to each follower, and then may be read by every consumer group. If the replication factor is three, one megabyte in creates two megabytes of copying between brokers. If four separate consumer groups read the topic, that megabyte is sent out four more times. A modest-looking ingress rate can therefore produce many times as much total traffic.

Rack Awareness

Brokers can be given a rack label, which usually represents a zone or a power and network failure domain. When rack awareness is used, Kafka spreads a partition’s copies across different racks. A failure of one rack then never removes all copies of a partition. As you add brokers, keep the racks balanced. Adding five brokers to one rack and none to the others weakens this protection and makes some racks carry more traffic.

Fetching From the Nearest Replica

Normally consumers read from the leader. In clusters spread across zones, that means traffic often crosses zone boundaries, which costs money and adds delay. A feature called follower fetching lets consumers read from a nearby follower instead. It reduces cross-zone traffic and spreads read load, at the price of very slightly older data, because followers trail the leader by a small amount.

A cluster can be perfectly healthy while consumers still fall hopelessly behind.

Consumer lag is the gap between the newest event in a partition and the position the consumer group has reached. When lag grows steadily, the readers are slower than the writers. The fix depends on why. It might need more consumer instances, more partitions, faster processing code, or a fix to a slow downstream system that consumers call.

Within a consumer group, each partition is assigned to at most one member. If a topic has twelve partitions, then twelve members can work in parallel, and a thirteenth member sits idle. Adding consumers beyond the partition count gives no speedup. This is the reason partition count planning is tied so closely to consumer scaling.

What Happens When the Group Changes

When a member joins or leaves, the group must redistribute partitions. This is called a rebalance. Older rebalancing behavior paused every member while the work was reshuffled, which caused visible stalls in large groups. Newer approaches improve this. Cooperative rebalancing moves only the partitions that need to move, and static membership lets a restarting member reclaim its old assignment without a full reshuffle. Recent Kafka versions also introduce a redesigned group protocol that moves much of the coordination to the brokers and reduces stop-the-world pauses.

Well-behaved producers can double the useful capacity of a cluster without buying anything.

Producers decide how efficiently data arrives. A producer that sends every event alone creates a flood of tiny requests, and each request has overhead. A producer that groups events into batches sends far fewer requests and uses network and disk much more efficiently. Because this cost is paid by every broker on every request, small changes in producer behavior have large effects on the cluster.

Protecting the Cluster With Quotas

One misbehaving application can consume all the bandwidth of a shared cluster. Kafka supports quotas that limit the bytes per second a client or user may send or receive, and the share of request processing time they may use. When a client goes over its quota, the broker slows its responses, which pushes back on the client. Quotas are a scaling tool because they let a cluster serve many tenants fairly without buying capacity for the worst case of each one.

Moving thousands of partitions by hand is slow and error-prone. Tools exist for a reason.

In a cluster with a handful of topics, an operator can decide by hand which partitions go where. In a cluster with thousands of partitions and changing traffic, that quickly becomes impractical. Balancing tools examine the load on every broker, then propose a set of moves that improves the distribution while respecting rules such as rack placement and disk limits.

What Balancing Goals Usually Cover

The data plane moves events, but the control plane decides who does what. It can become a limit too.

The control plane keeps the cluster’s metadata and coordinates leader elections. In the classic design, a separate ZooKeeper ensemble stored this information, and one broker acted as the controller that read it. When a broker failed, the controller had to load a large amount of metadata and update many partitions, so recovery time grew with the number of partitions.

The newer KRaft design moves metadata into a Kafka-native log managed by a small quorum of controller nodes. Metadata changes become ordinary log events that brokers read and apply, which makes failover much faster and raises the practical limit on partition counts. Recent Kafka releases have removed ZooKeeper mode entirely, so new clusters use KRaft.

Sizing the Controller Quorum

The controller quorum is usually three or five nodes. More nodes do not make the control plane faster, because every metadata change must be agreed by a majority. They only allow more failures to be tolerated. Three nodes survive one failure, and five survive two. The load on controllers grows with the number of partitions and brokers, and with the rate of metadata changes, so very large clusters give the controllers well-provisioned machines and fast disks.

Shrinking a cluster is harder than growing it, because you must empty a broker before you switch it off.

Sometimes traffic falls, a project ends, or costs must be cut. Removing a broker is not as simple as shutting it down. If you stop a broker that still holds the only in-sync copy of some partitions, those partitions become unavailable or lose data. The safe order is to move every replica off the broker first, wait until all partitions are fully in sync elsewhere, and only then stop it.

A bigger cluster is not always the better answer. Sometimes the right move is another cluster.

Kafka clusters can grow large, but there are good reasons to run several. A single huge cluster has a huge blast radius. A software bug, a bad configuration, or a runaway client can affect every team that shares it. Separate clusters give stronger isolation, simpler capacity planning per group of workloads, and independent upgrade schedules.

The trade-off is real. Once data must cross clusters, you take on the challenges of mirroring, such as extra delay, replay after failover, and offset differences between copies. Splitting is a strong tool, but it should be chosen for clear reasons rather than as a reflex.

Web servers scale up and down with traffic in minutes. Kafka brokers cannot, and the reason is data.

Stateless services can be started and stopped freely because they hold nothing important. Kafka brokers are stateful. Each one holds partitions of data on its disks. Adding a broker only helps after data has been moved to it, and removing one requires draining it first. Both operations take time proportional to the amount of data, which can be hours, not seconds.

Running Kafka on Kubernetes

Kubernetes can manage Kafka, but brokers need stable identities, persistent disks that follow them, and careful restart ordering. Operators automate these steps. A pod restart must not remove too many replicas at once, and a moved pod must reattach to the same disk. Storage performance and network topology in the cluster matter greatly, since a broker sharing a node with noisy neighbors will suffer unpredictable latency.

What Can Be Elastic

Numbers make the ideas concrete. Here is a small, invented example that you can adapt.

Imagine a platform that expects to receive one hundred megabytes per second of data on average, with peaks at twice that. Topics use a replication factor of three, and three separate consumer groups read most of the data in real time. Data must be kept for seven days. This is an invented scenario for practice, not a recommendation for any real system.

Step One: Storage

One hundred megabytes per second is about 8.6 terabytes per day. Over seven days, that is roughly sixty terabytes of unique data. With three copies, the cluster must hold about 180 terabytes. Leaving forty percent free for safety, growth, and maintenance operations, the disk needed is about 300 terabytes across the cluster.

Step Two: Network

Each byte received is copied to two followers, so replication adds two hundred megabytes per second of traffic between brokers. Three consumer groups read the data, adding three hundred megabytes per second going out to clients. That is five hundred megabytes per second of outgoing traffic in total on average, and about a thousand at peak. Spread across brokers, each broker must stay well below the capacity of its network card.

Step Three: Choosing the Broker Count

Suppose each candidate broker has thirty terabytes of usable disk and a ten gigabit network card, which is about twelve hundred megabytes per second in theory and about half that as a comfortable working target. Storage points to about ten brokers. Network at the peak of a thousand megabytes per second, with each broker targeting six hundred, points to two brokers at minimum, but the brokers must also receive their share of incoming and replicated traffic, and a failure of one broker must not overload the rest. Considering these together, twelve brokers is a reasonable starting design, giving room to survive the loss of a broker or a rack.

Scaling operations are risky moments, so you need sharper eyes and tighter controls.

What to Watch During a Scaling Operation

Logging and Records

Keep a written log of every scaling action: what changed, when, who approved it, the throttle values, and the observed effects. These logs help the next person plan better, and they are invaluable when reviewing an incident after the fact.

Security Considerations

New brokers must meet the same security standards as old ones. That includes encrypted connections with valid certificates, correct authentication settings, and access rules that allow the brokers to talk to each other. Encryption also has a scaling cost, because it uses processor time, so capacity plans for secure clusters must include it. Operators who run reassignments need strong permissions, and those permissions should be limited, audited, and removed when no longer needed, since a mistaken move can harm the whole cluster.

The same successes and mistakes repeat across organizations.

A checklist can prevent most painful surprises.

The same scaling questions appear in very different businesses.

Kafka began at LinkedIn, where it was built to move enormous streams of activity data, and much of today’s balancing and capacity practice comes from operating clusters at that scale. Since then, organizations in banking, retail, telecommunications, logistics, and gaming have adopted it, each with different pressures. Some care most about storage over long periods, others about millisecond latency, and others about serving many internal teams safely on shared infrastructure.

No. A new broker starts empty and shares no load until partitions are reassigned to it. New topics may use it right away, but existing topics stay where they are.

No. Kafka only allows increasing partitions. To reduce them, create a new topic with fewer partitions and migrate producers and consumers to it.

It is possible but changes which partition new events for a key land in. Anything that depends on strict ordering per key may be affected during and after the change, so plan carefully or use a new topic.

Measure how much a single partition can handle on the producing and consuming sides, divide the target throughput by each figure, take the larger result, and add headroom for growth.

A group cannot use more consumers than there are partitions. Extra members sit idle. Add partitions or use fewer members, and consider whether the busy partitions are unevenly loaded because of skewed keys.

Both extremes have risks. Very large brokers take long to recover and enlarge the failure impact. Very many tiny brokers add operational overhead. Many teams choose a middle path of medium brokers, sized so that losing one is manageable.

Use a replication throttle, move in small batches, watch latency and under-replicated partitions between batches, and do it during quieter periods when possible.

It lets brokers keep only recent data locally and place older data on cheaper remote storage, so retention can grow without adding broker disks, and broker moves and recoveries deal with less local data.

Not usually. Three or five controller nodes are typical. More nodes only add fault tolerance and do not speed up decisions. Very large clusters instead give the controllers strong machines and fast disks.

Clients can, brokers cannot easily. Brokers hold data, so adding or removing them requires moving data, which takes time and load. Plan broker capacity on slower, deliberate cycles.

Drain it by reassigning all its replicas to other brokers, wait until every affected partition is fully in sync, verify it holds nothing, and then shut it down. Remove one broker at a time.

When you need stronger isolation, different tuning for different workloads, regional placement, independent upgrade schedules, or when a single cluster’s metadata load is nearing comfortable limits.

Treating scaling as adding machines without diagnosing the real bottleneck. If consumers, partitions, network, or a client bug is the limit, extra brokers alone will not help.

A widely used habit is to plan expansion when any main resource reaches about sixty percent, leaving room for peaks, failures, and the extra load of data movement itself.

Scaling a Kafka cluster is a chain of small, well-understood decisions. First, find which dimension is under pressure: throughput, storage, partition count, connections, consumer parallelism, or shared use by many teams. Then choose the lever that matches, whether that is bigger brokers, more brokers, better partition design, tiered storage, better client behavior, quotas, or a separate cluster. Finally, carry out the change slowly and with measurement, because moving data is itself a source of load and risk.

The most reliable teams keep generous headroom, size partitions carefully at creation, automate with limits and reviews, and rehearse their procedures. They know that new brokers start empty, that partitions can grow but never shrink, and that consumers cannot outnumber partitions usefully. With that knowledge, scaling becomes a routine part of running Kafka instead of a crisis.