A partition looks free when you create it. The bill arrives later, and it arrives in the form of files, memory, network traffic and recovery time.

A quick picture of Kafka

Apache Kafka stores streams of events. An event is a small record of something that happened, such as “order 5521 was placed.” Events are written into named containers called topics. To spread the work, every topic is cut into pieces called partitions. Each partition is an ordered, append-only list of events. The servers that hold the partitions are called brokers, and a group of brokers working together is a cluster.

Programs that write events are producers, and programs that read events are consumers. Consumers that share the work of one topic form a consumer group, and inside one group each partition is read by only one consumer at a time. That last rule matters a lot: the number of partitions sets the maximum number of consumers that can work in parallel on one topic.

Why more partitions feel like a good idea

There are three real benefits. First, more partitions let more producers write at once, because different partitions can be written in parallel on different disks and brokers. Second, more partitions allow more consumers to read in parallel, which raises the top speed at which a group can process data. Third, a topic with many partitions spreads across many brokers, so no single machine carries all the traffic. Because of these benefits, many engineers assume that more is always better.

Why more partitions are not free

Every partition, no matter how quiet, brings a fixed set of costs to the broker that hosts it. Each partition owns files on disk. Each partition has bookkeeping objects in memory. Each partition with replicas needs copies kept up to date through constant network fetching. Each partition has an entry in the cluster’s metadata, which the controller must load, change and send to every broker. When partitions are few, these costs are invisible. When partitions reach the thousands per broker, the costs start to shape everything the cluster does.

A short history of the guidance

In the early years, Kafka clusters ran on ZooKeeper, a separate coordination service that stored all cluster metadata. Every partition added data to ZooKeeper, and the elected controller broker had to read all of it whenever it took over. Because of that, the Kafka community offered a rule of thumb that became famous: keep each broker to about 4,000 partition replicas and each cluster to about 200,000 partitions. Since then, Kafka replaced ZooKeeper with its own built-in metadata system called KRaft, and Kafka 4.0 removed ZooKeeper entirely. The new design was built so that clusters can hold far more partitions, but the per-broker costs of files, memory and replication did not disappear. That is why per-broker limits still matter.

Four myths that cause trouble

Keep these four myths in mind as you read on. Each chapter that follows will show, with numbers and pictures, why they do not hold.

The number that matters for a broker is not the number of partitions in your topics. It is the number of partition copies the broker actually holds.

Replicas: the copies that keep data safe

To survive machine failures, Kafka keeps several copies of every partition. Each copy is called a replica. The number of copies is the replication factor, and production clusters usually use three. One replica is elected leader. All producers write to the leader, and consumers normally read from it. The other replicas are followers. They continuously fetch new events from the leader so that they can take over if the leader dies. The set of followers that are fully caught up is called the in-sync replicas, often shortened to ISR.

The counting rule

Here is the formula that every Kafka architect should remember. The number of replicas that live on an average broker equals the total number of partitions in the cluster, multiplied by the replication factor, divided by the number of brokers. A cluster of nine brokers holding 3,600 partitions with a replication factor of three carries 10,800 replicas in total, which is 1,200 replicas per broker. Many people forget the replication factor when they compare their numbers with published guidance, and they end up three times too optimistic.

Why leaders and followers cost different things

A leader replica does the most visible work: it accepts writes, serves reads to consumers and answers requests from followers. A follower replica does quieter work: it repeatedly fetches from the leader and writes the data to its own disk. Both kinds own files and memory objects on their broker, so both count toward the broker’s limits. Leaders add request handling load on top, so an uneven spread of leaders can overload one broker even when the total replica counts look equal.

Do not forget Kafka’s own topics

Kafka creates internal topics that also use partitions. The topic that stores consumer group positions, called __consumer_offsets, has 50 partitions by default. If transactions are used, another internal topic called __transaction_state also has 50 by default. Tools such as Kafka Connect and Kafka Streams add more topics of their own. These are usually small next to your business topics, but they belong in your count.

Rack awareness and how replicas are placed

When Kafka creates a topic, it decides which brokers will hold each replica. If brokers are tagged with a location label, such as the Availability Zone they run in, Kafka can spread the copies of one partition across different locations. This feature is called rack awareness. With three zones and a replication factor of three, each partition then has one replica in each zone, so the loss of a whole zone still leaves two copies alive.

Rack awareness affects counting in a subtle way. The brokers in each zone should hold similar numbers of replicas. If one zone has more brokers than the others, its brokers carry fewer replicas each, and the smaller zones carry more. Plan broker counts in multiples of the number of zones, such as 6, 9 or 12 brokers for three zones, so the load divides evenly and no zone becomes the weak link.

To understand the cost of a partition, open the broker’s data folder and look at what is really there.

One partition, one folder

Kafka stores each partition replica as a folder on the broker’s disk. The folder name contains the topic name and the partition number. Inside the folder, the data is not one big file. It is cut into pieces called segments. A segment is simply a chunk of the log that covers a range of offsets. An offset is the sequential number Kafka gives to each event in a partition, like a page number in a book.

Files that belong to every segment

Each segment has a group of files with the same base name. The main file holds the actual events. Next to it sit two small index files. One index helps Kafka jump quickly from an offset to a position in the main file. The other helps it jump from a timestamp to an offset. When transactions are used, a further index tracks aborted transactions. There are also small snapshot files that remember the state of idempotent and transactional producers. The important point is that one segment means several files, and one partition usually means several segments.

flowchart TB
    T["Topic orders"] --> P0["Partition 0 folder"]
    T --> P1["Partition 1 folder"]
    P0 --> S1["Segment A closed"]
    P0 --> S2["Segment B closed"]
    P0 --> S3["Segment C active"]
    S3 --> F1["Data file"]
    S3 --> F2["Offset index"]
    S3 --> F3["Time index"]
    S3 --> F4["Producer state snapshot"]
    P1 --> S4["Segment D active"]

When does Kafka start a new segment?

A segment is closed and a new one started when it reaches a size limit, which is one gigabyte by default, or when it reaches an age limit, which is seven days by default. This age rule surprises many people. A quiet partition that receives only a few events a day will still roll to a new segment every week. Across thousands of quiet partitions, this creates many small segments and therefore many small files.

Retention deletes whole segments

Kafka does not delete individual events. When data becomes older than the retention setting, or when a partition grows past its size limit, Kafka removes complete closed segments. This makes deletion cheap and keeps the disk pattern simple. It also means that the number of segments alive at any moment depends on both retention and segment size. A topic that keeps data for thirty days with daily segments has around thirty segments per partition, each with its own set of files.

These numbers are only illustrations, but they teach a clear lesson. The number of files grows with partitions multiplied by segments, and long retention makes the count much larger. The file count then decides how many things the operating system must track, which is the subject of Chapter 5.

Writes, the page cache and why sequential matters

Kafka is fast partly because it works with the operating system rather than against it. When a broker receives events, it appends them to the end of the active segment. The operating system first keeps that data in a fast area of memory called the page cache and writes it to the disk later in the background. Kafka does not force every message to the disk immediately, because safety comes from having copies on other brokers. Readers who are close to the end of the log are served straight from the page cache, which is why consumers that keep up are so cheap to serve.

This design works best when writes are sequential, which means each write goes right after the previous one on the disk. Appending to a file is sequential. But a broker with thousands of active partitions is appending to thousands of different files at the same time. On a modern solid state disk, this is fine. On a spinning disk, the head has to jump between files, and speed can fall sharply. Even on solid state disks, having thousands of different files receiving writes reduces the benefit of the page cache because the recently written tail of each file competes for the same memory.

A partition is cheap in isolation and costly in bulk. Here is the full price list.

flowchart LR
    PART["One more partition replica"] --> FD["Open file handles"]
    PART --> MM["Memory-mapped index files"]
    PART --> HEAP["Broker heap objects"]
    PART --> REP["Replication fetch work"]
    PART --> CTL["Controller and metadata entries"]
    PART --> IO["Disk write spreading"]
    PART --> REC["Longer restart and recovery"]
    PART --> CLI["Client buffers and rebalances"]

Cost is not equal for all partitions

A partition that receives constant heavy traffic costs far more than an idle one, because it uses network, disk and processor time as well as the fixed costs. A quiet partition still pays the fixed costs: files, memory, fetch bookkeeping and recovery. This is the reason why “our data volume is small, so we can create as many partitions as we like” is a dangerous belief. The fixed costs do not care how much data you have.

Which limit hits first?

In practice, one of several limits gets reached first, depending on the cluster. On small machines, the memory maps or file handles often run out first, and the broker fails with errors that look unrelated to partitions. On mid-sized machines, replication latency and recovery time become the complaints. On large machines with plenty of memory, the controller and metadata size are more likely to matter. That is why a single published number cannot replace testing on your own hardware.

A rough cost model you can use on a whiteboard

Nobody can give one exact price for a partition, because it depends on segments, settings and traffic. But a simple model helps you think. Imagine each replica costs a few units of fixed overhead for its files and memory, a few more for replication bookkeeping, and a share of recovery time. Multiply the fixed overhead by the number of replicas on the broker. Then add the variable cost of traffic, which depends on the bytes per second flowing through the busy partitions. The first part grows with the number of partitions. The second part grows with the number of bytes. A cluster runs into trouble when the fixed part alone uses most of the broker, leaving little room for the traffic part.

This is the reason why two clusters with the same traffic can need very different machines. One cluster serves a few hundred heavy partitions. The other serves twenty thousand light ones. The first is limited by bytes, the second by counts. Knowing which kind you run tells you what to buy: faster disks and networks for the first, more memory, processors and careful limits for the second.

The first wall most teams hit is not in Kafka at all. It is in the operating system underneath.

Open files

Every program on Linux may keep only a limited number of files open at once. This limit is set per process and is often only a few thousand by default on a fresh machine. A Kafka broker keeps the files of its segments open, so its need grows with partitions multiplied by segments. If the limit is too low, the broker fails to open new files and starts throwing errors such as “too many open files,” often in the middle of a busy day when a new segment needs to be created.

Kafka’s own installation guides recommend raising this limit to a very high value, commonly in the range of one hundred thousand or more. Do not guess. Estimate your file count by multiplying replicas per broker by segments per replica by files per segment, add generous headroom for connections and temporary files, and set the limit above that.

Memory maps

Kafka uses a feature called memory mapping for its index files. The operating system lets a program treat a file as if it were part of memory, and this makes lookups very fast. But each mapped file uses one entry in a system-wide table with a limit called the maximum map count, which is 65,530 by default on many Linux versions. Each segment maps at least two index files. Therefore a broker with 20,000 segments would ask for about 40,000 mappings, which is already close to the default limit.

A safe tuning routine

Why sparse index files still matter

Kafka reserves space for an active segment’s index files up front so that they never have to grow while the broker is running. That reserved space is a sparse file, so it does not use real disk until it fills, but the address space and the mapping still count. Thousands of active segments therefore reserve a lot of virtual memory even if the disk stays nearly empty. This is another reason why many idle partitions are not as harmless as they look.

Other operating system details worth checking

Open files are not only segments. Every client connection is also a file handle, so a broker that serves thousands of producers and consumers needs additional room. Add the expected number of connections to your estimate. Next, check that the file system suits Kafka. The Kafka documentation recommends modern file systems such as XFS or ext4 and warns against network file systems. A file system that holds hundreds of thousands of tiny files should also have enough inodes, the records the file system keeps for each file, or it may report “no space left” while gigabytes are still free.

Finally, remember the wider machine. The page cache needs memory outside the Java heap, so do not give all memory to the JVM. A common approach is a moderate heap, often a few gigabytes, and leaving the rest for the operating system to use as cache. Thousands of partitions increase the pressure on both, so measure garbage collection pauses as well. Long pauses on a broker can look like a broker failure to the rest of the cluster.

Every follower replica is a small background job that never stops. Thousands of them add up.

How followers stay current

A follower does not wait to be told about new data. It runs a loop: send a fetch request to the leader, receive whatever is new, write it to disk, and ask again. A broker that hosts many follower replicas groups their requests by leader, so one fetch request can ask about many partitions at once. Each broker also runs a limited number of replica fetcher threads to do this work. With few partitions, these threads are mostly idle. With thousands, they can become the busiest threads on the machine.

The commit rule and why it creates delay

Kafka treats a message as safely stored, or committed, only after all in-sync replicas have it. A consumer normally cannot see a message until it is committed. This means the end-to-end time for a message includes the time for followers to copy it. When one broker must replicate hundreds or thousands of partitions to another broker, each round of fetching takes longer, so the message waits longer to become visible. Measurements published by the Kafka community showed this delay staying small for a few hundred partitions and growing to tens of milliseconds and beyond when a single broker pair shared a thousand or more. The exact numbers depend on hardware, but the trend is clear: latency creeps up as partitions per broker go up.

Falling out of sync

If a follower cannot keep up, it may be removed from the in-sync group. In recent Kafka versions the follower must fail to catch up for about 30 seconds by default before it is dropped. A dropped follower means fewer safe copies, and if producers require several copies to acknowledge a write, the producers may start to see errors or slowdowns. A broker overloaded by too many replicas is a common reason for this pattern, often shown as a rise in under-replicated partitions, which is the number of partitions whose followers are behind.

Levers you can pull

Acknowledgments and the price of safety

Producers choose how sure they want to be that data is safe. With the strictest setting, the leader answers “done” only after all in-sync replicas have the data. A related broker setting, the minimum in-sync replicas, says how many copies must exist for a write to be accepted at all. A very common production choice is three replicas with a minimum of two in sync, which survives the loss of one broker without losing data or stopping writes. This is a good place to see why partition counts matter: when a broker is overloaded and a follower falls out of sync, the partition may drop below the minimum, and producers start receiving errors.

So the partition limit is also a safety limit. A broker that carries too many replicas is more likely to slow down in a way that removes copies from the safe set. The system then loses the very protection you paid for by keeping three replicas.

The brain of the cluster keeps a record of every partition. How that brain is built decides how many partitions the whole cluster can hold.

What the controller does

One broker in the cluster acts as the controller. It watches which brokers are alive, chooses the leader for each partition, and tells every other broker about changes. All of that information is metadata: the list of topics, their partitions, which brokers hold each replica, and who leads. Metadata must always be correct, because clients use it to find where to send their requests.

The ZooKeeper design

In older clusters, the metadata lived in ZooKeeper, a separate service. The controller was one of the brokers, and when it failed, a new controller had to be elected and then had to read the state of every partition from ZooKeeper before it could work again. With hundreds of thousands of partitions, this reading took long enough to cause noticeable outages. Every change to a partition, such as a leader move, was also a write to ZooKeeper, so a broker failure with thousands of leaders produced thousands of writes in a short burst. These facts are the source of the old cluster-wide ceiling of about 200,000 partitions.

The KRaft design

In the modern design, called KRaft, Kafka stores metadata in an internal log that is replicated among a small group of controller nodes using a Raft-style agreement protocol. One controller is active and the others are hot standbys that already hold the full log. When the active controller fails, a standby takes over in a very short time without reloading everything from an outside system. Brokers follow the metadata log and keep an up-to-date copy in memory. This makes failovers much faster and removes the ZooKeeper bottleneck, so the design target for cluster-wide partition counts moved from hundreds of thousands to the millions.

flowchart TB
    subgraph OLD["ZooKeeper era"]
        ZK[("ZooKeeper ensemble")]
        CO["Active controller broker"]
        BR1["Brokers"]
        ZK --> CO
        CO --> BR1
    end
    subgraph NEW["KRaft era"]
        Q1["Active controller"]
        Q2["Standby controller"]
        Q3["Standby controller"]
        ML[("Replicated metadata log")]
        BR2["Brokers with metadata copy"]
        Q1 --> ML
        Q2 --> ML
        Q3 --> ML
        ML --> BR2
    end

Does KRaft remove per-broker limits?

No. This is a very common misunderstanding. KRaft raises the ceiling for the whole cluster and speeds up recovery of the cluster’s brain. But files, memory maps, replication fetching, disk behavior and client buffers are broker-level costs, and they are the same as before. Every broker also keeps the full metadata in memory, so a huge cluster-wide total does still cost each broker some heap. A team that reads “KRaft supports millions of partitions” and decides to put ten thousand replicas on one broker will still meet the older walls.

How metadata reaches brokers and clients

Metadata travels in two directions. Inside the cluster, every broker follows the controller’s metadata log and applies each change to its own in-memory view. Outside, clients ask brokers for the current picture of the topics they use: how many partitions each has and which broker leads each one. Clients refresh this information periodically and whenever they hit an error such as “not the leader for this partition.”

Clients normally ask only about the topics they use, so the response is small. But some tools ask for everything: monitoring systems, administrative dashboards, connectors that read topic lists, and some poorly written scripts. A metadata response that covers hundreds of thousands of partitions is large, and a fleet of tools requesting it every few seconds can burn a surprising amount of broker processor time. When partition counts grow, review which tools ask for full metadata and slow them down or narrow their questions.

The number of partitions on a broker decides how long a bad day lasts.

When a broker dies

Suppose a broker holding 3,000 replicas, about 1,000 of them leaders, suddenly loses power. Every partition it led is now without a leader. Producers and consumers of those partitions get errors until a new leader is chosen. The controller must select a new leader from the in-sync followers for each of those 1,000 partitions, record each decision, and tell the other brokers. Clients then must refresh their view of the cluster and reconnect. If the broker held only 300 leaders, all of these steps would finish about three times faster, and fewer applications would notice.

sequenceDiagram
    participant P as Producer App
    participant B1 as Broker 1 failing
    participant C as Active Controller
    participant B2 as Broker 2 follower
    participant B3 as Other Brokers
    P->>B1: Write to partition leader
    B1--xP: No response, broker is down
    C->>C: Detect missing broker
    C->>B2: Promote follower to leader
    C->>B3: Send updated leader metadata
    P->>B3: Refresh metadata
    B3-->>P: New leader is Broker 2
    P->>B2: Write to new leader
    B2-->>P: Acknowledged

Planned restarts are smoother, but still scale with partitions

For planned maintenance, Kafka offers controlled shutdown. Before stopping, the broker asks the controller to move its leaders elsewhere first, one group at a time, so clients see very little disruption. This gentle process takes time proportional to the number of leaders, which is why rolling upgrades of a broker with many partitions take noticeably longer. After the broker comes back, it must catch up on everything it missed, and later Kafka moves leadership back to the original preferred brokers.

Crashes and log recovery

If a broker stops without a clean shutdown, it cannot trust its files. On start-up it must check the latest segments of every partition and rebuild indexes where needed. A setting called the number of recovery threads per data directory controls how many partitions are checked at once. The default is low, so with thousands of partitions on a busy disk the start-up can take many minutes. Many operators raise this setting on brokers with lots of partitions, keeping in mind that too many threads on one slow disk help little.

A recovery budget

Good architects work backward from a goal. Suppose the business says that no data stream may be interrupted for more than one minute after a single broker failure. You then test how long a failure of your busiest broker really takes, with realistic partition counts, and choose a per-broker limit that keeps this time under the goal with room to spare. This turns an abstract number into a promise you can defend.

A rolling-restart checklist for brokers with many partitions

If this routine takes hours in a large cluster, it is a sign that the per-broker partition count is high or that catch-up traffic is being throttled too much. The time taken is a direct measure of how much room for growth you have left.

Partition counts do not only burden brokers. The applications on the other side of the wire feel them too.

Producers: smaller batches, more memory

A producer does not send each event on its own. It collects events into batches, one batch per partition, and sends a batch when it is full or when a short waiting time has passed. Batching is the secret of Kafka’s high throughput. When a topic has many partitions, the same stream of events is spread over more batches, so each batch fills more slowly and contains fewer events. Smaller batches mean more requests, more overhead and lower efficiency. The producer also needs memory for each partition it is writing to, so a producer touching thousands of partitions may need a larger buffer.

Modern producers reduce the pain for events that have no key. A “sticky” strategy, introduced in Kafka 2.4, keeps sending keyless events to one partition until its batch is full and then switches, which makes batches larger. Events with keys must go to the partition chosen by their key, so this help does not apply to them.

Consumers: the parallelism ceiling and the fetch cost

A consumer group can never use more active consumers than there are partitions in the topic. Extra consumers simply sit idle. This is the true reason many teams want lots of partitions: to leave room for growth in consumer count. But each consumer also fetches from every partition assigned to it, and it holds fetch buffers for each. A consumer assigned hundreds of partitions needs plenty of memory and can spend much of its time managing fetches. Kafka improved this in version 1.1 with incremental fetch sessions: instead of repeating the whole list of partitions in each request, the consumer and broker remember the list and send only the changes.

Rebalances: dividing the work again

Whenever a consumer joins or leaves a group, the group must decide again which consumer reads which partition. This process is called a rebalance. With more partitions, computing and communicating the new assignment takes longer. Older rebalances paused every consumer in the group during the change. Newer approaches reduce the pain: cooperative rebalancing lets consumers keep partitions that are not moving, and the newer consumer group protocol that became generally available in Kafka 4.0 moves the assignment work to the broker so that large groups settle faster.

Client quotas and noisy neighbors

Partition counts also interact with fair sharing. In a shared cluster, one application that reads or writes too hard can slow down everyone whose partitions live on the same brokers. Kafka provides client quotas that cap the bytes per second, and the share of request processing time, that a given user or client identifier may consume. A broker that hosts thousands of partitions serves many applications at once, so quotas are the seat belts that stop one busy application from starving others. Set quotas per service account and review them along with partition budgets.

Another subtle effect is connection count. Each producer or consumer opens connections to every broker that leads one of its partitions. A topic spread over many partitions and brokers therefore makes each client connect to many brokers. With thousands of clients this becomes a large number of connections, each using a file handle and some memory. Very large fleets sometimes reduce this by consolidating producers behind a small number of proxy services.

Before asking what a broker can hold, decide what each topic truly needs.

The throughput formula

A simple method comes from the early Kafka designers. First, measure how fast one partition can accept data from a producer, and how fast one consumer can process data from one partition. Call the target throughput of the topic T, the producer speed per partition p and the consumer speed per partition c. The number of partitions you need is at least T divided by p, and at least T divided by c. Take the larger of these two numbers.

Ordering and keys

Kafka guarantees order only inside one partition, not across a whole topic. When events carry a key, such as a customer id, Kafka uses the key to pick the partition, so all events for one customer stay in order in one partition. The choice is made by a formula that divides a number derived from the key by the count of partitions. Therefore changing the partition count changes where each key goes. New events for a customer may land in a different partition from their old events, and order for that customer can break during the change.

How much headroom is reasonable?

A common practice is to plan for growth over one or two years and allow perhaps two to three times today’s need, not ten or a hundred times. Bigger cushions cost real resources every day. If traffic really does outgrow the topic, there are safe paths: add partitions during a quiet time when the loss of strict key order for a moment is acceptable, or create a new topic with more partitions and migrate. For topics where key order is sacred, plan the migration approach in advance instead of hoping to avoid it.

Questions to ask before choosing a number

Special kinds of topics

Not every topic follows the plain rules. Compacted topics keep only the latest value for each key, and they are common for holding current state such as a customer profile. Background threads called log cleaners rewrite their segments from time to time, so many compacted partitions add to the cleaning workload. Kafka’s own offsets topic is compacted for this reason. Dead-letter topics, which collect messages that failed processing, are usually low volume and rarely need more than a few partitions. Change-data-capture topics that mirror database tables often deserve one partition per table when strict order of changes matters, and this can multiply topic counts in a big database.

Stream processing adds internal topics for repartitioning and state backup. Their partition count is usually copied from the input topic, so an oversized input topic makes every internal topic oversized too. This is another quiet way that one decision spreads through the platform.

Now we reach the main question. The honest answer is “it depends,” but there is a good way to work it out.

The starting point

The classic rule of thumb is to keep the number of partition replicas per broker at or below about 4,000, and many production clusters run well under that. This figure was created for ZooKeeper-era clusters running on reasonably large machines, and it is a ceiling for well-provisioned brokers, not a target. Smaller machines need lower numbers. Some cloud providers publish their own guidance by machine size. Amazon MSK, for example, lists recommended maximum partition counts per broker that rise with instance size, from a few hundred on the smallest test-sized brokers up to about 4,000 on the largest, and it may adjust these numbers over time. Always read the current documentation for the service and version you use.

These ranges are teaching examples, not official limits. Use them only to shape your first estimate, then confirm by testing on your own hardware with your own settings.

The factors that move the number

Leaders matter more than followers for load

A useful refinement is to watch leader counts separately. The busy work of taking writes and serving reads lands on leaders. If your cluster has three replicas per partition, a broker with 3,000 replicas normally has about 1,000 leaders. Limiting leaders per broker, in addition to total replicas, prevents a case where a few brokers have become overloaded leaders while others hold mostly quiet followers.

How to prove your own limit with a load test

Kafka ships with performance test tools for producers and consumers. Use them, or your own load generator, to answer the question for your hardware. The exercise below takes a few hours and is worth far more than any number from a blog post.

Apply a safety factor to the result. Many teams keep production at sixty to eighty percent of the proven limit, so that a broker failure or a burst of new topics does not push the cluster over the edge.

Follow this routine whenever you design a new cluster or check an old one.

flowchart TD
    A["List topics with target throughput and consumer counts"] --> B["Compute partitions per topic with the formula"]
    B --> C["Add headroom of two to three times at most"]
    C --> D["Multiply by replication factor"]
    D --> E["Divide by number of brokers"]
    E --> F{"Replicas per broker under the tested limit?"}
    F -->|Yes| G["Approve design and set alerts"]
    F -->|No| H{"Can partitions be reduced?"}
    H -->|Yes| B
    H -->|No| I["Add brokers or split into more clusters"]
    I --> E

A complete worked example

A company plans a cluster for 900 topics. On average each topic gets 12 partitions, so the business topics use 10,800 partitions. Internal and connector topics add about 200, giving 11,000. With a replication factor of three, that is 33,000 replicas. The test on the chosen machine type proved that a broker stays healthy up to 3,000 replicas with recovery under the one-minute goal, so the soft limit is 1,800 replicas per broker and the hard limit is 2,400. Dividing 33,000 by 18 brokers gives about 1,833 per broker, right at the soft limit. The team decides to start with 18 brokers, budget for 24 within a year, and to clean up topic sizes before adding hardware.

Choosing the number of brokers when partitions are the constraint

If the partition count sets your broker count, the calculation is simple. Divide the total number of replicas by the limit you are willing to allow per broker, then round up to a multiple of the number of zones. For the worked example, 33,000 replicas divided by a soft limit of 1,800 gives about 18.3, which rounds up to 21 brokers when using three zones. Comparing this with the traffic-based estimate tells you which factor is really in charge. If traffic needs only 9 brokers but partitions need 21, the extra machines are paying for the partition overhead. That is a strong signal to reduce partitions before buying hardware.

A limit that nobody enforces is only a suggestion. Here is how large platforms keep partition counts under control.

Kafka has no simple built-in cap

Apache Kafka does not ship with one setting that says “refuse partitions beyond this count per broker.” Instead, platform teams put guard rails around the places where partitions are born. There are four main tools.

A topic request workflow

Mature platforms treat a new topic as a small design review, done by machine wherever possible. The requester states the expected throughput, retention, key needs and consumer count. A tool applies the formula from Chapter 10, proposes a partition count, checks it against the team’s quota and against the cluster’s remaining room, and either creates the topic or explains what to change. Teams get self-service speed, and the platform team gets numbers it can trust.

Charge back and clean up

People treat what is free carelessly. Showing each team how many partitions it owns and what share of the cluster that represents changes behavior quickly. Add a monthly report of topics with no producers, no consumers or no traffic for thirty days. Deleting dead topics is often the cheapest way to add capacity, and it also reduces the size of the metadata every broker holds.

Guard rails for automation and connectors

Partitions are not created only by people. Data integration tools and stream applications often create topics on their own when they start. A connector for each database table, or a stream application with many stages, can create dozens of topics without anyone noticing. Review the defaults of every tool that has topic-creating power. Set the partition and replication values in its configuration to sensible small numbers, and give its service account permission to create topics only under its own prefix. This way, a mistaken configuration can create at most a bounded number of partitions.

Finally, keep a list of the small set of people allowed to override the limits. Exceptions are sometimes right, for example a heavy new pipeline that truly needs 200 partitions. What matters is that exceptions are visible, approved and counted in the same budget as everything else.

Adding brokers does not automatically lighten the load. The partitions must be moved, and moving has its own costs.

New brokers start empty

When you add a broker to a cluster, Kafka does not shift existing partitions onto it by itself. The new broker holds only new topics created afterwards. To share the existing load, you must run a partition reassignment, which copies chosen replicas to the new broker and then removes them from the old ones. While the copy runs, the moving partition has an extra replica, and the data crosses the network and reads the disks of the source brokers.

The cost of moving

Moving replicas competes with normal traffic. A large reassignment can flood the network, slow producers and consumers, and push followers out of sync. Good practice is to move in small waves, apply a throttle that caps the bandwidth used for moves, and schedule bulk moves in quiet hours. Kafka lets you set such a throttle on replication traffic. Remember that on AWS the traffic between brokers in different Availability Zones is billed, so a giant rebalance has a direct money cost as well as a performance cost.

Balancing more than counts

An even split of partition counts is not the same as an even split of load. One partition carrying 50 megabytes per second is worth many quiet ones. Balancing tools consider disk use, network in and out, and processor load as well as counts. LinkedIn, the original home of Kafka, built and open-sourced a tool called Cruise Control that watches these numbers and proposes or performs moves. Many managed services now include automatic balancing features as well.

Leader skew after restarts

After a rolling restart, leaders often end up bunched on a few brokers. Kafka can automatically move leadership back to each partition’s preferred replica on a schedule, and this setting is normally left on. If it is off, or if partitions were assigned with poor preferences, one broker can carry far more leaders than others and become the first to fail under load.

Moving a keyed topic to a new partition count safely

When a topic truly must change its partition count and key order matters, the safest route is a migration to a new topic. The steps below keep readers and writers in order.

You cannot manage a limit that you cannot see. These are the signals worth putting on the main dashboard.

Trends beat snapshots

A single reading of “2,700 replicas on this broker” means little. A chart showing that the number has risen every week for six months tells you when you will run out. Plot the growth rate of partitions per week beside the limit and estimate the date on which the soft limit will be reached. Put that date in the team’s planning calendar and order hardware early.

Measure recovery, not just health

Healthy dashboards on a normal day say nothing about a bad day. Record the time each rolling restart takes, the time to elect leaders after a test failure and the time for a replaced broker to catch up. If these times drift upward as partitions grow, you are approaching a limit even if every other graph is calm.

A weekly review routine

Dashboards only help if someone looks at them on a schedule. A short weekly review keeps partition growth honest. The platform team opens one page that shows, for every cluster, the replicas per broker against the soft and hard limits, the number of new topics created this week, the largest topics by partition count, and the slowest restart or recovery time recorded. Each week the team asks three simple questions. Which team added the most partitions, and was that justified? Which topics look oversized compared with their traffic? And how many weeks remain until the soft limit is reached at the current pace? Ten minutes on these questions each week prevents most of the emergencies described in this guide.

Every partition decision balances speed today against flexibility and safety tomorrow.

Patterns that work

Common mistakes checklist

• Counting partitions instead of partition replicas when comparing with guidance.
• Forgetting internal topics and connector topics in the total.
• Believing that KRaft removes per-broker limits.
• Leaving the operating system’s file and map limits at default values.
• Planning for a perfect day instead of a day with one broker down.
• Adding partitions to a keyed topic without thinking about broken ordering.
• Leaving automatic topic creation on in a shared cluster.
• Never deleting topics that nobody uses.
• Rebalancing without throttling and swamping the network.

Best practices in one place

Measure before choosing. Count replicas, not partitions. Keep per-topic partition counts close to real need. Prove your per-broker limit on your own hardware. Set soft and hard limits and enforce them at topic creation. Watch the trend and the recovery time, not just today’s health. When the limit is near, add brokers and rebalance gently, and remove waste before buying more machines.

A simple decision guide

When two options seem equal, prefer the one that keeps the partition count lower. Fewer partitions are easier to move, recover, monitor and understand, and you can always add more when the data proves that you need them.

Different businesses meet the partition limit in different ways, and they solve it with the same handful of ideas.

flowchart TB
    TEAMS["Product teams request topics"] --> POL["Topic policy: sizing, naming, team quota"]
    POL -->|approved| CL1["Cluster A: payments domain"]
    POL -->|approved| CL2["Cluster B: logistics domain"]
    POL -->|approved| CL3["Cluster C: analytics domain"]
    POL -->|rejected| TEAMS
    CL1 --> MON["Dashboards: replicas per broker, recovery time, lag"]
    CL2 --> MON
    CL3 --> MON
    MON --> PLAN["Quarterly review: clean up, rebalance, add brokers"]
    PLAN --> POL

Lessons that repeat

Every successful platform makes the sensible path the easy path: self-service topic creation that quietly applies good sizing. Every struggling platform lets numbers drift until an outage forces a clean-up. The technology is the same in both. The difference is that the good ones treated partition count as a budget from the very first day.

What to do on day one of a new platform

If you are building a new Kafka platform, you can avoid nearly every problem in this guide with a few early decisions. Write down the naming convention and the sizing formula. Turn off automatic topic creation. Install a create-topic policy with a default team quota. Set the operating system limits before the first broker starts. Choose broker counts in multiples of your zone count. Decide your soft and hard limits from a load test. Put the partition dashboard on the wall. None of these steps is expensive at the start, and each one is painful to add after a hundred teams depend on the cluster.