Before we plan the number of partitions, we need a clear picture of what one partition actually is.

Kafka in one paragraph

Apache Kafka is a system for moving and storing streams of events. An event is a small record of something that happened, such as a customer placing an order, a sensor reporting a temperature, or a user clicking a button. Programs that write events are called producers. Programs that read events are called consumers. The machines that store the events and serve them are called brokers, and a group of brokers working together is a cluster.

Topics and partitions

Events in Kafka are organised into named streams called topics. A topic called “orders” holds all order events. But a topic is only a label. The real storage happens in partitions. A partition is an ordered, append-only log of events. New events are always added to the end, and each one receives a sequential position number called an offset. Once written, an event is never changed in place.

A topic is split into one or more partitions, and those partitions are spread across the brokers in the cluster. When a producer sends an event, Kafka picks exactly one partition for it. When a consumer reads, it reads from partitions. So the partition is the smallest unit that Kafka can place on a broker, copy to another broker, and hand to a consumer.

flowchart LR
  P1["Producer A"] --> T
  P2["Producer B"] --> T
  subgraph T["Topic: orders"]
    direction TB
    PA["Partition 0"]
    PB["Partition 1"]
    PC["Partition 2"]
    PD["Partition 3"]
  end
  PA --> B1["Broker 1"]
  PB --> B2["Broker 2"]
  PC --> B3["Broker 3"]
  PD --> B1
        

Key terms at a glance

It is the practice of choosing how many partitions a topic should have, on purpose, using numbers instead of guesses.

Partition count planning means estimating how much data a topic must handle today and in the future, how many consumers will need to read it in parallel, what ordering rules the business needs, and how much load the brokers can safely carry. From those inputs you choose a partition count that is large enough to avoid bottlenecks and small enough to avoid waste and instability.

Why not just pick a big number?

A common beginner reaction is to say “more is always better, so let us create a thousand partitions for every topic.” Partitions are not free. Each one consumes file handles, memory, network traffic for replication, and work for the cluster controller. Too many partitions make broker restarts slower, make leader elections longer, and make clients use more memory. The goal is a balanced number, not a huge one.

Why not just pick a small number?

The opposite mistake is choosing one or two partitions because it feels simple. A topic with a single partition can only be read by one consumer at a time within a group, and it can only live on one broker for writes. When traffic grows, there is no way to spread the load without changing the design.

Almost every reason to plan partition count comes back to one idea: partitions decide how much work can happen at the same time.

Parallelism on the producer side

Producers send batches of events to the leader of each partition. When a topic has more partitions, its leaders are spread over more brokers, so writes are shared across more disks and network cards. A single partition leader can only accept writes as fast as one broker can handle for that partition, so a topic that must absorb very high write rates needs enough partitions to spread the load.

Parallelism on the consumer side

On the reading side, the rule is strict. Inside one consumer group, each partition is assigned to at most one consumer at any moment. This guarantees that events in a partition are processed in order by a single worker. The consequence is important: if a topic has six partitions, a group can have at most six consumers doing useful work. A seventh consumer would simply wait, idle, as a spare.

flowchart TB
  subgraph Topic["Topic with 4 partitions"]
    P0["P0"]
    P1["P1"]
    P2["P2"]
    P3["P3"]
  end
  subgraph Group["Consumer group with 3 consumers"]
    C1["Consumer 1"]
    C2["Consumer 2"]
    C3["Consumer 3"]
  end
  P0 --> C1
  P1 --> C1
  P2 --> C2
  P3 --> C3
        

Independent groups do not compete

Different consumer groups read the same topic independently. A billing service group and an analytics group can each read every event, and each keeps its own progress. The partition-to-consumer rule applies inside each group separately, so the number of partitions limits the parallelism of your largest group, not the number of groups.

Ordering is where partition planning turns from a math problem into a design decision.

What Kafka guarantees about order

Kafka guarantees order only inside a single partition. If event A is written before event B into the same partition, every consumer will see A before B. Across different partitions, there is no ordering guarantee at all. Two events in different partitions may be read in any relative order.

How keys choose partitions

When a producer attaches a key to an event, the default behaviour is to run the key through a hash function and use the result to choose a partition, roughly the hash value divided by the partition count, keeping the remainder. The important consequence is that the same key always goes to the same partition, as long as the partition count stays the same. This is how you keep all events for one customer, one order or one device together and in order.

flowchart LR
  E["Event with key: customer-42"] --> H["Hash the key"]
  H --> M["Take remainder by partition count"]
  M --> P["Always the same partition"]
        

The hash trap

This is the single most important planning warning in the whole topic. If you increase the partition count of a keyed topic later, the mapping changes. A key that used to land in partition 2 may now land in partition 5. New events for that key go to the new partition while old events remain in the old one. During the transition, consumers may see events for one key out of order, and stateful programs that assumed one key lives in one place can break.

Keys with no ordering need

If events do not need to be grouped by any key, producers can leave the key empty. Kafka then spreads events across partitions in a balanced way, using a strategy that fills batches per partition before moving to the next. For such topics, changing partition count later is much safer because no business rule depends on the mapping.

The most widely used starting point is a short calculation that compares what you need with what one partition can do.

The core idea

Suppose you want a topic to handle a target throughput, which is the amount of data per second the whole topic must carry. You measure or estimate two numbers. The first is how much one partition can absorb on the write side. The second is how much one partition can be processed on the read side by a single consumer. Then you divide the target by each number and take the larger result.

A worked example in words

Imagine a payments topic that must carry a peak of one hundred megabytes per second. Testing shows a single partition can accept about twenty megabytes per second on the write side. A single consumer, because it calls a fraud-scoring service for every event, can only process five megabytes per second. Dividing the target by the write capacity gives five. Dividing by the consumer capacity gives twenty. The larger value is twenty, so the topic needs at least twenty partitions to keep up. Notice that the slow consumer, not the broker, decided the answer.

Why peak matters more than average

Averages hide the busiest moments. A shopping platform may run at a modest rate all week and then receive ten times more events during a sale. If partitions are sized for the average, consumers fall behind during the peak and lag grows. Plan for the peak you must survive, and then check how quickly consumers can catch up afterwards.

Turning throughput into events per second

Some teams find it easier to think in events per second instead of megabytes. Multiply events per second by average event size to get data rate. Small events cost more per byte because of overhead, while very large events strain memory and network buffers. Always use realistic event sizes from production samples, including the slowest and largest cases.

A good partition count leaves room to add consumers later without touching the topic.

Start from the maximum consumer count

Ask how many consumer instances your largest group may ever need. This includes peak season, catch-up after downtime, and future features. Your partition count should be at least that number. If you might one day run thirty consumers, thirty partitions is the floor, not the ceiling.

Choose numbers that divide nicely

When partitions are shared among consumers, even splits make life easier. Twelve partitions can be evenly divided among two, three, four or six consumers. Thirteen partitions cannot be evenly split by any of those. Numbers with many divisors, such as twelve, twenty-four or forty-eight, give flexibility when consumer counts change. This is not a hard rule, but it reduces the situation where one consumer carries an extra partition and becomes the slow one.

How much headroom is enough?

There is no universal answer, but many teams plan for twice to three times the current need when ordering makes later changes painful. For unkeyed topics where increasing later is safe, they start closer to the current need and grow when metrics say so. The right multiplier depends on how costly a change would be for your specific system.

Every partition is a small, permanent commitment for the cluster. Here is what each one costs.

Open files and disk structures

Each partition on a broker is stored as a directory of log segment files, and each segment has companion index files. Kafka keeps many of these files open so it can serve reads and writes quickly. More partitions mean more open file handles on each broker, and operating systems have limits. Running out of file handles leads to failures, so brokers with many partitions need carefully raised limits.

Memory on brokers and clients

Producers keep a memory buffer for batches destined to each partition they write to. If a producer talks to thousands of partitions, its memory use grows, and batches get smaller, which lowers efficiency. Consumers, likewise, reserve fetch buffer space per assigned partition. A consumer that owns hundreds of partitions may need much more memory than one that owns ten.

Replication traffic

Every partition has followers that continuously fetch from the leader. With a replication factor of three, each partition creates two follower streams. Thousands of partitions produce thousands of tiny fetch streams between brokers. Even when data volume is low, this bookkeeping consumes network and processing time.

Controller and metadata load

One component in the cluster, the controller, manages partition leadership and metadata. It decides which replica is leader for each partition and informs the other brokers of changes. More partitions mean more metadata to store, transmit and update. Modern Kafka versions, which manage metadata with their own internal quorum instead of an external coordination service, handle far more partitions than older versions, but the cost never becomes zero.

Partition count changes how the cluster behaves when things go wrong, and how fast data feels.

What happens when a broker fails

When a broker stops, every partition whose leader lived on that broker needs a new leader chosen from its in-sync followers. This election must happen for each affected partition. If a broker was leading a thousand partitions, a thousand elections must complete before all those partitions are available again. During that time, producers and consumers of those partitions may see errors or pauses.

sequenceDiagram
  participant B as Broker 1
  participant C as Controller
  participant F as Follower Brokers
  participant K as Clients
  B-->>C: Broker stops responding
  C->>C: Find partitions led by Broker 1
  C->>F: Promote in-sync followers to leaders
  F-->>K: Clients discover new leaders
  K->>F: Resume reads and writes
        

Restart and recovery time

During a planned restart or after an unclean crash, a broker must load its partitions and bring followers back in sync. More partitions, and more segment files inside them, lengthen this process. Rolling upgrades, where brokers restart one after another, take longer as the partition count per broker grows.

Latency

A record is considered safely committed only after in-sync followers have copied it. A follower broker handles many partitions with limited fetch threads. As the number of partitions per broker rises, each partition gets attention less often, and the time for followers to catch up can increase. For most workloads the effect is small, but for latency-sensitive systems it can matter.

Failure blast radius

Interestingly, more partitions can also make failures gentler in one way. If a topic’s load is spread across many partitions on many brokers, losing one broker removes a smaller share of the work. The trade-off is between a smaller share lost per failure and a larger amount of recovery work per failure. Planning means finding the balance that suits your availability goals.

You can grow a topic, but you cannot shrink it, and growing has side effects you must understand in advance.

Increasing is allowed, decreasing is not

Kafka lets administrators add partitions to an existing topic. It does not let anyone remove partitions from a topic. If you overshoot, the only way back is to create a new topic with fewer partitions and move the data and applications over to it. That is why planning matters: adding is easy, undoing is hard.

What adding partitions does and does not do

New partitions start empty. Existing data is not moved into them. Only new events flow into the new partitions according to the current mapping. For unkeyed topics this simply spreads future load. For keyed topics, as discussed earlier, the mapping of keys to partitions changes, which can disturb ordering and state.

flowchart LR
  subgraph Before["Before: 3 partitions"]
    A0["P0: key K1 history"]
    A1["P1"]
    A2["P2"]
  end
  subgraph After["After: 6 partitions"]
    B0["P0: old history"]
    B1["P1"]
    B2["P2"]
    B3["P3 new"]
    B4["P4 new"]
    B5["P5 new"]
  end
  Before --> After
  B3 -. "K1 may now land here" .-> B0
        

The safer migration path

When a keyed topic truly needs more partitions, the safest method is usually to create a new topic with the desired count, run producers so they write to the new topic, let consumers drain the old topic fully, and then move consumers over. This avoids key mixing across old and new partitions. It needs coordination, time and often a brief period of dual running, which is why teams prefer to choose well the first time.

The right number of partitions is useless if the load lands unevenly on the machines.

Partitions per broker

Kafka places partition leaders and replicas across brokers, trying to balance them. For balance to work, the partition count should be comfortably larger than the broker count. A topic with three partitions on a ten-broker cluster uses only three brokers for its leaders. If you want a topic to use the whole cluster, the partition count should be a multiple of the broker count or at least several times larger.

Data skew and hot partitions

Skew means some partitions receive far more data than others. It happens when keys are unevenly distributed. If one customer generates half of all events and the key is the customer ID, one partition becomes a hot spot, and its consumer falls behind while others sit idle. Adding more partitions does not help because that one customer still maps to one partition.

Ways to reduce skew

Partition count interacts with disk space, retention time and the cost of moving data.

How storage is divided

The total storage a topic needs equals its data rate multiplied by its retention time, multiplied again by the replication factor. This total is shared across all partitions. More partitions do not create more total data, but they divide it into more, smaller pieces. Each partition lives entirely on one broker per replica, so a single partition cannot be larger than the disk space available on the broker that hosts it.

Keeping partitions a manageable size

Very large partitions are hard to move. If one partition holds terabytes of data, then rebalancing it to a new broker, or rebuilding a lost replica, takes a long time and consumes lots of network bandwidth. Very small partitions are easy to move but add overhead. Many teams aim for partitions that stay within a size where recovery and movement finish in a reasonable time window that matches their availability targets.

Retention and compaction effects

Topics with long retention keep old segments for a long time, so file counts and disk usage stay high. Topics with log compaction, which keep only the latest value per key, run background cleaning work per partition. Cleaning threads are a shared resource, so having a very large number of compacted partitions can slow the cleaning and delay space reclaim.

Some workloads follow extra rules, and partition planning should take them into account.

Stream processing applications

Stream processing frameworks that read from Kafka often create one task per input partition. The number of partitions in the input topics therefore sets the maximum parallelism of the application. When an application reads from several topics and joins them, those topics must usually have matching partition counts and matching key rules so related events meet in the same task.

Stateful processing and local storage

Applications that keep local state, such as running totals per customer, attach that state to a partition. If the partition count changes, the state may no longer match the new key mapping. This is another strong reason to plan generously when stateful processing is involved.

Transactions and exactly-once processing

Transactions allow a set of writes across partitions to succeed or fail together. They add coordination cost that grows with the number of partitions touched in one transaction. Designs that write to a huge number of partitions in each transaction can slow down, so consider grouping related data on fewer partitions when using transactions heavily.

Connectors and data integration

Data integration tools that copy data between Kafka and databases usually parallelise by partition on the Kafka side and by table or range on the other side. The number of tasks they can run is limited by partitions, so a sink connector reading a topic with two partitions will never use more than two tasks effectively.

Very small or very slow topics

Configuration topics, audit trails or rarely updated reference data may have tiny volumes. A single partition, or a few, is often perfectly fine. Giving them dozens of partitions adds overhead with no benefit. The same rule applies to dead-letter topics that hold rare failed events.

In a shared cluster, partition count is also a governance question, not just a technical one.

Why governance matters

In a company with many teams sharing one Kafka cluster, any team that can create topics can also create partitions. One team choosing very high partition counts for many topics can quietly consume a large share of the cluster’s capacity and hurt other teams. Treat partition count as a shared resource, like disk space or network bandwidth.

Practical controls

Planning gives you a starting point. Monitoring tells you if reality agrees.

Consumer lag

Consumer lag is the difference between the newest event in a partition and the position of the consumer group. Lag that grows steadily during normal traffic means consumers cannot keep up. If adding consumers helps, and you still have free partitions, the plan is fine. If all partitions are already taken and lag persists, your partition count is limiting you.

Per-partition throughput and size

Track bytes in and out per partition, and the size on disk. Compare the busiest partition with the average. A large gap points to skew. Comparing total traffic with the number of partitions shows how much room you still have.

Broker health signals

Watch the number of partitions per broker, the count of leaders per broker, request handler idle time, network thread usage, open file handle usage, and under-replicated partitions. A rising count of under-replicated partitions during normal operation suggests brokers are struggling to keep up with replication, sometimes because of too many partitions.

Controller and recovery timing

Measure how long leader elections and broker restarts take. Record these times after each cluster change. If restarts take significantly longer than they used to, partition growth may be the reason.

Patterns that work well, and the traps that catch even experienced teams.

Patterns worth adopting

Anti-patterns to avoid

Put everything together into a simple procedure any team can follow for a new topic.

flowchart TD
  S["New topic request"] --> Q1{"Is per-key order or state needed?"}
  Q1 -- "Yes" --> K["Plan generously: change later is risky"]
  Q1 -- "No" --> U["Plan closer to need: growth later is safe"]
  K --> M["Measure write and consumer capacity"]
  U --> M
  M --> C["Compute: target divided by capacity, take the larger"]
  C --> H["Add headroom and align to consumer counts"]
  H --> L{"Within cluster limits?"}
  L -- "No" --> R["Review design, add brokers, or reduce"]
  L -- "Yes" --> D["Document reasoning and create topic"]
  R --> H
  D --> W["Monitor lag, skew and broker health"]
        

A quick reference of what to do and what to avoid.

Four common situations show how the same principles lead to different partition decisions.

What large organisations do

Companies that operate Kafka at very large scale usually build internal platforms around these ideas. They offer standard topic templates, automated checks on requested partition counts, dashboards for lag and skew, and regular reviews of clusters with too many partitions per broker. The common thread is that partition count is treated as a managed, reviewed resource and not as a casual default.

No. The right number depends on throughput, consumer speed, ordering needs and cluster capacity. Two topics in the same cluster often deserve very different counts.

You can start them, but within one consumer group the extra consumers will sit idle. They are useful only as standby replacements if an active consumer fails.

No. Kafka only allows increasing partitions. To end up with fewer, you must create a new topic and migrate producers and consumers to it.

Existing data stays where it is. New partitions begin empty and receive only new events. For keyed topics, the mapping of keys to partitions changes for new events.

Only up to the point where other limits take over, such as disk speed, network bandwidth, consumer processing or broker overhead. Beyond that, extra partitions add cost without adding speed.

Each partition is an independent log handled by its own leader. Coordinating order across many independent logs would slow everything down, so Kafka guarantees order only where it is cheap and reliable.

Yes for load purposes. Each replica is stored and served by a broker, so a topic with ten partitions and three replicas creates thirty partition copies across the cluster.

Not necessarily, but a count that is a multiple of the broker count helps spread leaders evenly. A topic with fewer partitions than brokers cannot use every broker for leadership.

Then Kafka spreads events across partitions on its own, and changing the count later is much safer. You can start with a moderate number and grow when metrics show the need.

Signs include persistent consumer lag even after adding consumers to the limit, single brokers saturated on writes, and inability to scale processing because every partition already has its own consumer.

Signs include slow broker restarts, long leader elections, high memory use in clients, many empty or nearly empty partitions, and very high partition counts per broker compared with your tested limits.

Rarely. A large number of tiny topics multiplies partitions and overwhelms the cluster. A shared topic keyed by customer ID usually meets the need with far fewer partitions.

Partition count planning is the art of giving a Kafka topic enough lanes to carry its traffic and its readers, without paying for lanes it will never use. A partition is the basic unit of storage, replication and parallel reading, so this single number shapes throughput, ordering, recovery time and cost. The best plans combine measured numbers, honest attention to slow consumers, generous thinking for keyed and stateful data, and steady monitoring after launch.

If you remember only one sentence, let it be this: adding partitions later is possible but disruptive for keyed data, and removing them is impossible, so think carefully at the start and review with real traffic.