The honest answer is “it depends on which bottleneck you have,” and the rest of this guide shows how to tell.

Two different meanings of fast

People use the word fast to mean two different things, and mixing them causes most of the confusion. Throughput means how many messages the system can handle every second. Latency means how long a single message takes to travel from the moment it is created to the moment it is delivered and processed. A system can have excellent throughput and poor latency, like a freight train that carries enormous loads but takes a long time to arrive. It can also have low latency and modest throughput, like a courier on a motorbike.

Partitions are Kafka’s main tool for scaling throughput, because they allow many producers and many consumers to work in parallel. But partitions also add overhead to the brokers, the clients, and the cluster’s coordination machinery. Whether the net effect on latency is good or bad depends on whether the system is limited by too little parallelism or by too much overhead.

The shape of the curve

If you measure latency while increasing the partition count on a fixed cluster, the result often looks like the letter U. At first, adding partitions reduces latency because work that was queuing behind a bottleneck now proceeds in parallel. Then the curve flattens, because parallelism is no longer the limiting factor. Finally, it starts to climb, because the overhead of managing so many partitions begins to cost more than the parallelism gains. The bottom of the U is the sweet spot, and where it lies depends on your hardware, your message rate, and your settings.

Before asking how partitions change latency, we must know where latency comes from in the first place.

The stages

A message passes through several stages between its creation and its use. Each stage adds a little waiting time, and end-to-end latency is the sum of all of them. Partition count touches almost every stage, which is why the topic is so rich.

flowchart LR
    A["Message created"] --> B["Producer batch wait"]
    B --> C["Network to partition leader"]
    C --> D["Broker request queue and log append"]
    D --> E["Replication to in-sync followers"]
    E --> F["High watermark advances - message visible"]
    F --> G["Consumer fetch"]
    G --> H["Consumer processing queue"]
    H --> I["Message processed"]
        

Where partitions enter the story

Partition count influences the producer batch wait, because batches are formed per partition. It influences the broker’s write path, because every partition is a separate log. It influences replication, because followers must keep every partition in step. It influences consumer fetching and memory, because each partition is tracked separately. It influences the consumer processing queue, because partitions decide how many consumers can share the work. And it influences recovery time when something fails. The following chapters visit each of these in turn.

To understand the cost of a partition, it helps to see that each one is a real, physical thing with real resources attached.

A partition is a log on disk

A topic is divided into partitions, and each partition is an ordered, append-only sequence of messages. On the broker’s disk, a partition is a directory containing a series of segment files. Only the newest segment receives writes, and older segments are kept until they are deleted or compacted. Alongside each segment are index files that let the broker find a message by its position or by its timestamp without reading the whole segment.

Every partition also has a leader and, usually, two followers, so a topic with a hundred partitions and a replication factor of three has three hundred replicas spread across the brokers. Each replica is its own log directory, with its own open files and its own bookkeeping. The number of partitions therefore multiplies directly into the number of files, memory maps, and replication streams the cluster must manage.

The partition as the unit of everything

A partition is the unit of ordering, because order is guaranteed only inside one partition. It is the unit of parallelism, because at most one consumer in a group reads a partition at a time. It is the unit of replication, because leaders and followers are assigned per partition. It is the unit of leadership, because one broker leads each partition. And it is the unit of failure recovery, because leaders are elected per partition. Since so many things attach to the partition, changing the number of partitions changes many things at once.

Partitions reduce latency in one very important case: when messages are waiting in line because there are too few workers.

Queueing delay in plain language

Whenever work arrives faster than it can be handled, a queue forms, and every item in the queue waits. This waiting is called queueing delay, and it can dwarf every other source of latency. In Kafka, the consumers are often where queues form: if one consumer can process a thousand messages per second and a thousand and five hundred arrive, the extra five hundred pile up every second, and the delay of each new message grows without limit.

The number of partitions sets a ceiling on how many consumers in a group can work at the same time. A topic with four partitions can be shared by at most four active consumers of one group, however many you start. If the consumers are the bottleneck, and the partition count is the ceiling, then adding partitions and consumers removes the queue, and latency drops dramatically.

When the benefit is real

Throughput per partition sets the minimum

Each partition can sustain only a limited rate, set by the slower of the producer and the consumer side, and by the disk and network of the broker leading it. If a single partition can absorb, say, a certain number of megabytes per second, then to handle a target rate you need at least that many partitions, or the partitions become saturated and their queues grow. Being under this minimum causes both low throughput and high latency at once. Section on choosing a count later in this guide shows how to compute it.

The first place where too many partitions hurt latency is inside the producer, before the message even leaves the application.

How the producer batches

A Kafka producer collects messages into batches, and it keeps a separate batch for each partition it is writing to. A batch is sent when it reaches a size limit, set by the setting batch.size, or when it has waited for a time limit, set by linger.ms, whichever comes first. Batching is essential for efficiency, because sending one large batch is far cheaper than sending many tiny requests. It also improves compression, since compression works better on larger blocks of similar data.

Why more partitions make batches smaller

Suppose a producer sends a thousand messages per second. If the topic has ten partitions, each partition receives about a hundred messages per second. If the topic has a thousand partitions and messages are spread evenly, each partition receives one message per second. The batch for each partition fills a hundred times more slowly. So either the batch is sent almost empty, wasting effort, or the batch waits for the time limit, adding delay to every message inside it.

When the linger time is zero, the producer sends batches immediately, so latency for each message stays low, but the batches are tiny, and the number of requests grows. This increases load on the brokers and on the network, and can indirectly raise latency through queueing. When the linger time is greater than zero, batches become fuller, but every message may wait up to the linger time before being sent. Either way, spreading a fixed message rate over more partitions makes batching less effective.

flowchart TD
    A["Fixed producer rate"] --> B{"Partition count"}
    B -->|"Few partitions"| C["Each batch fills quickly"]
    B -->|"Many partitions"| D["Each batch fills slowly"]
    C --> E["Large efficient batches with small wait"]
    D --> F{"Linger time setting"}
    F -->|"Zero"| G["Tiny batches, many requests, more broker load"]
    F -->|"Positive"| H["Full batches but messages wait for the timer"]
        

Keys make it stricter

When messages carry keys, the producer must send each key to the partition chosen by hashing it. The producer has no freedom to gather messages into fewer partitions. The batch for a partition contains only the messages whose keys map to it. With a large number of partitions and a moderate rate, batches for many partitions are nearly empty most of the time.

The sticky partitioner for messages without keys

For messages without keys, the producer has a choice of partition, and newer versions use it wisely. Instead of rotating through partitions one message at a time, the sticky partitioner keeps sending to one partition until its batch is full or has been sent, then switches to another. This concentrates messages into fewer batches and greatly reduces the penalty of a large partition count for keyless traffic. A later refinement adjusts the choice according to how quickly each broker is responding, so slower brokers receive fewer messages and are not allowed to hold up the stream.

Partition count also changes how much memory a producer needs and how its requests are shaped.

The buffer and the batch memory

A producer holds unsent messages in a memory buffer whose total size is set by buffer.memory. Each partition with pending messages takes a batch’s worth of memory, up to the batch size. If a producer writes to thousands of partitions at once, the memory needed for open batches can approach or exceed the buffer. When the buffer is full, the send call blocks, waiting for space, and that waiting is pure latency added to the application.

This is a common surprise in systems that grow their partition count without resizing producer memory. Nothing else changes, the message rate stays the same, and yet the producer begins to block during bursts. The cure is either a bigger buffer, a smaller batch size, or fewer partitions.

Requests are grouped by broker

The producer does not send one request per partition. It groups all ready batches destined for the same broker into a single request. This softens the cost of many partitions, because a request to one broker can carry batches for many of its partitions. But the request still contains a separate section for each partition, the broker must process each section, and the response must acknowledge each one. So the per-partition overhead does not vanish, it is only shared inside a request.

In-flight limits and ordering

A producer limits how many requests it may have unanswered for each broker connection. This limit protects ordering and idempotence, but it also means that if a broker is slow, requests wait in line behind each other. With many partitions per broker and small batches, the number of requests can climb, and the in-flight limit can become the bottleneck that adds latency.

Once a batch reaches the broker, the number of partitions that broker hosts changes how smoothly the disk, the memory, and the operating system can serve it.

Appending is cheap, and that is the secret

Kafka is fast because writing to a log is an append at the end of a file. The operating system takes the bytes, stores them in a memory area called the page cache, and flushes them to disk later in large, friendly chunks. A broker does not force every message to disk before answering, because safety comes from replication rather than from flushing each write.

Sequential turns into scattered

With a handful of active partitions per disk, the writes form a few steady streams, and the operating system can group them well. With thousands of active partitions on the same disk, the flushes go to many different files at once. On spinning disks this scattering is painful because the head must move. On modern flash storage it is much less painful, but it still reduces the efficiency of grouping and can make the moment of a flush more bursty.

The page cache is shared

The page cache is one pool of memory shared by every partition on the machine. Each active partition wants its recent data to stay in that pool so that consumers reading fresh messages are served from memory rather than from disk. When there are many partitions, each one gets a thinner slice of the pool. If consumers keep up, they read very recent data and the pool works well. If some consumers fall behind, they read older data that may have left the cache, and those disk reads compete with the writes, which raises latency for everyone on that broker.

File handles and memory maps

Every partition replica is a directory of segment files, and each segment has index files. The broker keeps many of these open and maps the index files into memory. That leads to two operating system limits that matter when the partition count grows.

Segment rolling and retention across many logs

Every log rolls a new segment when the current one is full or old enough, and background threads delete old segments according to retention. With many partitions, these small events happen very often somewhere on the broker. Each roll creates files and indexes, and each deletion removes them. Individually these are quick, but with thousands of logs, the constant background activity uses the same disk and threads that serve client requests.

Log flushing and the recovery cost

After an unclean shutdown, the broker must check the logs that were being written when it stopped. More partitions means more logs to check, so the broker takes longer to become useful again. This is not steady-state latency, but it belongs to the latency story, because a slow restart extends the period during which clients see delays. Chapter 11 returns to this.

Replication is where partition count often has its largest effect on latency, because every partition must be copied, and copying has a fixed cost per partition.

How followers copy data

A follower replica works like a consumer. It sends fetch requests to the leader of each partition it follows and appends what it receives. A follower broker does not open one connection per partition. Instead, a small number of fetcher threads each handle a group of partitions that share the same leader broker, bundling them into one request.

Why partitions add delay here

A fetcher thread must cycle through its assigned partitions on every round. Each round involves preparing the request, waiting for the leader to answer, and applying the data to each partition log. If a fetcher is responsible for a very large number of partitions, one round takes longer, so new data on any single partition may wait for the round to come back to it. That waiting is added to the time before a message is considered committed.

An often-quoted early measurement, from the period when Kafka guidance about partition counts was first written down, showed that replicating around a thousand partitions from one broker to another with a single fetcher added noticeable delay, on the order of tens of milliseconds. Treat that figure as an illustration of the mechanism rather than a promise, because hardware, software versions, and settings have changed a great deal since then.

Fetch sessions reduce the burden

Newer Kafka versions use incremental fetch sessions. Instead of listing every partition in every request, the follower and the leader remember the list, and each request mentions only the partitions that have changed. This greatly reduces the cost of many quiet partitions, because a partition with no new data no longer adds to the request size. Partitions that are actively receiving data still cost work, so the gain is largest when many partitions are idle.

Waiting at the leader: the purgatory

With the strongest acknowledgment setting, the leader cannot answer the producer until enough followers have caught up. The waiting produce request is parked in a structure called the purgatory. Followers also send fetch requests that the leader may hold for a short time when there is no new data. Each partition with active traffic contributes waiting requests and wake-up events. More partitions with active traffic means more of this bookkeeping, and it happens on request-handling threads shared by all partitions.

sequenceDiagram
    participant P as "Producer"
    participant L as "Leader broker"
    participant F as "Follower broker"
    P->>L: "Produce batch for partition"
    L->>L: "Append to log and park request"
    F->>L: "Fetch round covering many partitions"
    L-->>F: "New data for the partition"
    F->>F: "Append to follower log"
    F->>L: "Next fetch reports progress"
    L->>L: "Advance high watermark"
    L-->>P: "Acknowledge the batch"
        

Read the diagram from top to bottom. The producer sends a batch and waits. The leader stores it and parks the request. The follower’s next fetch round retrieves the data. Then the follower’s following fetch carries the progress report that lets the leader move the high watermark and answer. The two fetch rounds in the middle are the part that stretches when a follower is busy with too many partitions.

Acknowledgment settings change who feels this

Levers on the follower side

Two settings matter most. The first is the number of replica fetcher threads on each broker. Raising it spreads partitions over more threads, so each thread covers fewer partitions and each round is shorter. The second is how long a leader holds a follower’s fetch request when no data is available. Shorter holds react faster but create more requests. Each extra thread costs some CPU and connections, so raise the count gradually and observe.

Consumers are affected in a mixed way: more partitions allow more consumers to share the work, but each consumer that reads many partitions pays extra costs.

How a consumer fetches

A consumer sends fetch requests to the brokers that lead the partitions it owns. Each request may include several partitions from the same broker. The broker answers when either a minimum amount of data is available or a maximum waiting time has passed. Two settings control this balance: a minimum byte threshold and a maximum wait time. A large minimum with a long wait means efficient but slow-arriving data, and a tiny minimum means fast but chatty fetching.

Many partitions per consumer

When the number of consumers is smaller than the number of partitions, each consumer owns several partitions. It must visit all of them, and the data it receives per round is spread across many logs. The client has a limit for how much data it accepts per partition in one response, and another limit for the whole response. Because the memory that a consumer may need grows with the number of partitions it owns times the per-partition limit, the settings must be sized with partition count in mind.

The ceiling of consumer parallelism

Within one consumer group, a partition is read by at most one consumer at a time. This means the partition count is the maximum useful number of consumers. If your processing is slow and you need forty consumers, you need at least forty partitions, or the extra consumers will do nothing. This is the strongest argument for having more partitions, and it is a latency argument, since it removes queueing in the processing stage.

Fairness between partitions

A consumer reading many partitions gets data from them in rounds. If one partition is very busy and others are quiet, a poorly tuned consumer may spend its capacity on the busy one. Modern clients try to rotate fairly, but you may still observe uneven delays across partitions. Uneven delays are a warning that some partitions receive far more traffic than others, which chapter 14 discusses.

Processing time dominates surprisingly often

In many real systems, the time to process a message is larger than all the time spent in Kafka. In that case, the useful question is not “how do partitions change Kafka’s latency” but “how many parallel workers does my processing need.” Adding partitions to enable more workers can cut end-to-end latency by far more than any tuning inside Kafka, and the overhead of the extra partitions is small compared with that gain.

Consumer memory and heap pressure

Each fetched batch is held in memory until the application processes it. A consumer with a large number of partitions and generous per-partition limits may hold a lot of data at once. Garbage collection pauses in the consumer show up as end-to-end latency, even though Kafka itself is healthy. Watch the consumer’s memory and pause times when raising partition counts.

Some latency is not about a single message at all. It is about the pauses that happen when the group of consumers changes.

What a rebalance is

When a consumer joins, leaves, or stops responding, the group must decide who reads which partitions. During that decision, some or all consumers may pause. The pause is a period in which no messages are processed for the affected partitions, so any message that arrives waits. For a system that promises low latency, a rebalance is a real event, and its length depends on how much the group has to decide.

How partition count lengthens the pause

flowchart TD
    A["Membership change"] --> B["Group pauses affected partitions"]
    B --> C["Assignment computed for all partitions"]
    C --> D["Assignment sent to every member"]
    D --> E["Members fetch committed offsets"]
    E --> F["State restored if the application is stateful"]
    F --> G["Processing resumes and backlog drains"]
    G --> H["Latency returns to normal"]
        

The last two steps deserve attention. While the group was paused, messages kept arriving. When processing resumes, those messages form a backlog, and even after the pause ends, the latency for new messages stays high until the backlog is drained. So the visible effect of a rebalance is longer than the pause itself.

Stateful applications feel it most

Stream processing applications keep local state for each partition they process, and that state is rebuilt from a changelog if it moves to another instance. More partitions means more state stores to move and restore. Restoring can take minutes for large state, which is a huge latency event compared with a normal message delay. Standby replicas and careful assignment can reduce the pain, but they cost resources.

Reducing rebalance cost

Newer group protocols move more of the work to the broker side and aim to make rebalances lighter, so the details depend on the Kafka version you run. Check your version’s documentation to see which protocol and assignors are available to you.

Systems are judged not only on a calm day but on the day something breaks, and partition count strongly shapes how long that day feels.

What happens when a broker dies

Every partition needs a leader. When a broker fails, every partition it led loses its leader, and each of those partitions must elect a new one from its in-sync followers. Until a new leader is chosen and clients learn about it, producers and consumers of that partition get errors or timeouts and must retry.

The unavailability window

The time during which a partition has no working leader is a hole in availability. Its length is roughly the time to detect the failure, plus the time to elect leaders for all affected partitions, plus the time for the new information to reach clients. The middle part scales with the number of partitions that were led by the failed broker. With a single controlled shutdown, leadership is moved gracefully before the broker stops, so the hole is small. With a crash, it is not graceful, and the cost is visible.

flowchart LR
    A["Broker fails"] --> B["Failure detected after timeout"]
    B --> C["Controller elects new leaders for each affected partition"]
    C --> D["New leadership propagated to brokers"]
    D --> E["Clients refresh metadata"]
    E --> F["Traffic resumes on new leaders"]
        

Follow the arrows from left to right. Detection takes about the same time no matter what. Election and propagation are the stages that stretch when the failed broker led thousands of partitions. Client refresh depends on client settings, but larger metadata takes longer to fetch and process.

The controller as a bottleneck

In older architectures based on ZooKeeper, the controller broker handled failover by reading and writing per-partition information in the coordination service, so its work was strongly proportional to the number of partitions and the recovery could be slow for very large clusters. Newer architectures based on the built-in Raft quorum, often called KRaft, keep metadata in a replicated log and make controller failover and metadata propagation more efficient, which is why they support much larger partition counts. The exact improvement depends on your version and setup, so read the release notes for the version you run before relying on it.

Recovery after restart

When a broker comes back, it must reload its logs, rebuild any indexes that were not cleanly closed, and catch up with the leaders. More partitions means more logs to open, so the restart takes longer. During this time the broker holds fewer leaders or none, and the cluster runs with less redundancy. Configuring several threads for log recovery per data directory can shorten this phase.

Besides moving messages, a Kafka cluster must keep track of itself, and that bookkeeping grows with the number of partitions.

The two planes

Engineers often speak of a data plane and a control plane. The data plane carries messages between clients and brokers. The control plane manages the facts about the cluster: which topics exist, which broker leads each partition, which replicas are in sync, and what configuration applies. Partition count barely changes the size of a single message, but it grows the control plane linearly, because every partition is one more fact to store, update, and share.

Where the metadata goes

Metadata churn and request latency

Whenever something changes, such as a leader moving or a replica dropping out of sync, updates are sent out. In a cluster with a very large number of partitions, changes happen more often simply because there are more things that can change, and each update carries more data. Brokers spend more time applying updates, and that work competes with serving requests. The visible effect is an occasional rise in request latency around events like rolling restarts or partition reassignment.

Client-side effects

A producer that writes to a topic with many partitions must know the leader of each one, and it refreshes this information periodically and after errors. A larger topic means larger refreshes. Consumers that subscribe by pattern, matching many topics, can be especially expensive, because every metadata refresh must be checked against the pattern, and any change in matching topics may trigger a rebalance. Applications with thousands of short-lived clients, such as serverless functions, multiply the effect, because each new client asks for the full metadata.

Operations that stress the control plane

Rolling restarts, mass topic creation, large reassignments, and broker replacements all cause bursts of metadata change. If they are done gently, one step at a time, a cluster with many partitions handles them well. If they are done all at once, latency spikes appear. Throttle reassignment traffic and spread operations over time.

Averages hide pain. The messages that suffer most are the ones that reveal what partition count really does to your system.

Average, median, and tail

If you line up a thousand message delays from shortest to longest, the median is the one in the middle, and the ninety-ninth percentile is the delay that only one message in a hundred exceeds. The far end is called the tail. An average can look wonderful while the tail is terrible, because a few very slow messages barely move the average. Users and downstream systems, however, often experience the tail, since one slow message can hold up an entire order or workflow.

How many partitions stretch the tail

A single message rarely waits on only one partition. A request that carries batches for many partitions is only complete when the slowest partition section is done. The more partitions involved, the higher the chance that at least one of them is having a bad moment, so the request as a whole is slower than the typical partition. This is a general effect of fan-out: the more independent pieces a task waits for, the more its delay follows the slowest piece.

Common tail sources related to partitions

flowchart TD
    A["Producer request with many partition sections"] --> B["Section for partition 1 - fast"]
    A --> C["Section for partition 2 - fast"]
    A --> D["Section for partition 3 - delayed by disk burst"]
    A --> E["Section for partition 4 - fast"]
    B --> F["Request response waits for all sections"]
    C --> F
    D --> F
    E --> F
    F --> G["Slowest section decides the request latency"]
        

In the picture, three of the four partition sections finish quickly. The third is delayed by a short disk burst. Because the response is sent as one piece, all four batches in that request are acknowledged only when the third is done. This is how a small problem in one partition is felt by many messages.

Watching the right numbers

Always look at percentiles, not only averages. Record the median, the ninety-fifth, the ninety-ninth, and the maximum for each stage. Compare them before and after changing partition counts. If the median stays the same but the ninety-ninth percentile climbs, you have found the cost of too many partitions or of too much sharing.

Choosing a partition count is not only a performance decision. It is tied to how messages with the same key are kept together and in order.

Keys decide the partition

When a message has a key, the default partitioner runs the key through a hash function and picks the partition using the result and the partition count. All messages with the same key go to the same partition, which is how Kafka keeps them in order. This is a valuable promise, because consumers can process all events for one customer or one order in sequence.

Adding partitions changes the mapping

Because the partition depends on the count, increasing the number of partitions changes where many keys go. After the change, a key that used to land on partition three may land on partition eight. New messages for that key go to a different partition than the older ones, so per-key ordering across the change is no longer guaranteed. A consumer that keeps state per key may find that the key’s history is on one instance while its new messages arrive at another.

flowchart LR
    K["Message key A"] --> H["Hash of the key"]
    H --> M1["Divide by 4 partitions - result is partition 3"]
    H --> M2["Divide by 8 partitions - result is partition 8"]
    M1 --> P3["Partition 3 holds old messages"]
    M2 --> P8["Partition 8 receives new messages"]
        

The diagram shows one hash result used with two different counts. Before growth, it selects partition three. After growth, it selects a different partition. The old data stays where it was written, because Kafka never moves existing messages between partitions.

Why this matters for latency planning

Since raising the count later is disruptive for keyed data, teams often choose a large count at the start to leave room for growth. That is the reason many topics begin with more partitions than they need today. The trade-off is that the overheads from the previous chapters begin immediately. A sensible approach is to plan the count for your expected growth over a reasonable period, and to design keyed consumers so that they tolerate a one-time reshuffle if it ever becomes necessary.

Hot partitions and skew

More partitions only helps if traffic spreads evenly. If a few keys account for most of the messages, their partitions become hot while the others sit idle. Extra partitions do not help, since a hot key still lands in one partition. The symptoms are one partition with high consumer lag while others are fine, and one broker with much higher load. Options include splitting a hot key into several sub-keys, using a different key, or accepting that hot keys need special handling.

Custom partitioners

If the default hash does not spread your traffic well, a custom partitioner can route by your own logic, for example sending large tenants to dedicated partitions. This gives control but also creates responsibility, because the routing rules must stay consistent between all producers and be handled carefully when the partition count changes.

Some Kafka features do extra work for every partition they touch, so a high partition count makes them relatively more expensive.

Transactions touch every participating partition

A transactional producer can write to several partitions and then commit or abort them all together. To do this, the transaction coordinator must remember every partition in the transaction, and when the transaction ends, it writes a small marker into each of those partitions. Consumers that read only committed data wait for that marker before they can pass the transaction’s messages.

The consequence is that the delay to see committed transactional data has a part that grows with the number of partitions involved in the transaction, plus the time for the slowest leader to write its marker. Short transactions that touch few partitions are cheap. Long transactions that touch hundreds of partitions have visible latency. Interval settings for commits also matter, since a transaction that stays open for a long time holds back consumers reading committed data on all the partitions it touched.

Compaction across many logs

Compacted topics keep only the latest value for each key. A pool of cleaner threads scans logs that have enough obsolete data, rewrites them without the old entries, and swaps them in. When there are many compacted partitions, the cleaner must spread its attention across all of them. Each cleaning pass reads and writes a lot of data, so it competes with client traffic for the disk. Consumers of compacted topics may also see delay if the cleaner falls behind, since old segments grow and restarts take longer to rebuild state.

Internal topics also have partitions

Kafka keeps internal topics for consumer group offsets and for transaction state. These have their own fixed partition counts, and the coordinators for groups and transactions are the leaders of those partitions. The number of internal partitions determines how many brokers can share coordinator work. Changing them after the cluster is in use is difficult, so it is decided early, and it is worth checking that the values suit a cluster of your size.

Now that each cost and benefit has been examined, we can join them into one story that explains why latency first falls and then rises.

Forces pulling in opposite directions

Some forces reduce latency as partitions increase: more parallel consumers, shorter queues, and lower load per partition. Other forces increase it: thinner producer batches, more replication bookkeeping, heavier metadata, longer rebalances, and slower recovery. Total latency is the sum of both. At small counts the first group dominates, so latency falls. At large counts the second group dominates, so latency rises. The minimum is where the two balance.

flowchart LR
    A["Very few partitions"] --> B["Queues form behind a parallelism limit"]
    B --> C["Increasing partitions removes queues"]
    C --> D["Sweet spot: enough parallelism, small overhead"]
    D --> E["Increasing further adds overhead faster than it removes queues"]
    E --> F["Many partitions: latency rises again"]
        

The diagram is a story in six steps. On the left, too few partitions create queues. Moving right, added partitions remove those queues. In the middle is the sweet spot. Further right, each extra partition adds more overhead than it removes queueing, and latency climbs.

Three regimes

Why the sweet spot moves

The curve is not symmetrical

The left side of the U is usually steep, because a real bottleneck can cause enormous queueing. The right side is usually gentle at first and then sudden, because overhead builds slowly and then hits limits. That asymmetry teaches a useful lesson: being somewhat above the sweet spot is usually cheaper than being below it, as long as you stay away from the cliffs at the far right.

A practical method starts with the throughput you need and then checks it against the latency costs.

Start with throughput

Measure, or estimate, how fast one partition can accept data on the producer side and how fast one consumer can process data on the consumer side. Then compute how many partitions you need to reach your target rate for each side. Take the larger of the two. If a single partition handles a modest rate for production and a lower rate for consumption because processing is slow, the consumer side decides the count.

flowchart TD
    A["Target peak throughput"] --> B["Divide by producer capacity per partition"]
    A --> C["Divide by consumer capacity per partition"]
    B --> D["Take the larger result"]
    C --> D
    D --> E["Add headroom for bursts and growth"]
    E --> F["Check partitions per broker including replicas"]
    F --> G{"Within comfortable limits?"}
    G -->|"Yes"| H["Load test and compare latency percentiles"]
    G -->|"No"| I["Add brokers or reduce the count"]
    I --> F
        

The decision node near the end is the guard against overshooting. If the per-broker count is too high, you have two honest options: add brokers so each carries fewer partitions, or reduce the plan. Then the loop repeats until the check passes.

Rules of thumb, carefully used

You may see guidance such as “a few thousand partition replicas per broker” or “a hundred thousand across a cluster” for classic setups, and higher numbers for newer metadata designs. Such numbers are useful as an order of magnitude and as a warning that the range is not unlimited. They are not guarantees, and they change with versions. Use them to know when to be careful, then verify with your own tests.

Different topics, different counts

Not every topic needs the same number. A high-volume clickstream topic may need many partitions, while a small configuration topic needs one or a few. Assigning a large default count to all topics wastes resources and adds to the cluster-wide total. Set counts per topic, according to its own throughput and consumer needs.

You cannot manage what you cannot measure, and measuring latency across several stages takes some care.

Measure at every stage

A single end-to-end number tells you that latency changed but not why. Stage-by-stage measurements tell you where to look. Kafka exposes many metrics on producers, brokers, and consumers, and most of the useful ones are grouped below by where they sit in the journey.

The broker request time breakdown

For each request type, the broker reports where time was spent. The parts are the wait in the request queue, the local processing, the wait for remote replication, the wait in the response queue, and the time to send the response. This breakdown is one of the most useful tools for our topic. A large remote part on produce requests points at replication. A large request-queue part points at not enough handler threads or too many requests, often from small batches. A large local part points at disk or CPU.

flowchart LR
    A["Request arrives"] --> B["Request queue wait"]
    B --> C["Local processing and log append"]
    C --> D["Remote wait for followers"]
    D --> E["Response queue wait"]
    E --> F["Response send time"]
    F --> G["Client receives answer"]
        

Use the picture as a checklist. When produce latency rises after adding partitions, look at each box in turn. If the queue wait rose, requests are arriving faster than threads can handle them. If the remote wait rose, followers are late. Each answer leads to a different fix.

Measuring end-to-end delay

The simplest method is to put a creation timestamp in each test message and compare it with the time the consumer receives it. This requires clocks that are well synchronized between the producer and consumer machines, or, even better, using the same machine for both during tests. Another method is to use consumer lag expressed in time rather than in message counts. Lag in messages is easier to collect but says little about delay unless you know the rate.

Running a fair experiment

If you need many partitions, several settings and design choices reduce the price you pay for them.

Producer levers

The producer’s job is to keep batches useful even when traffic is spread thin. Raising the linger time gives batches longer to fill, at the cost of a small guaranteed wait. Raising the batch size allows fuller batches when traffic is heavy. Compression works better on larger batches, so it rewards the first two levers. For keyless traffic, the sticky partitioner concentrates records on one partition at a time, which fills batches quickly and rotates for balance.

Broker levers

On the broker, the most helpful settings are the number of replica fetcher threads, the number of network threads, the number of request handler threads, and the number of recovery threads per data directory. Each raises capacity at a cost of extra CPU or memory, so change them step by step and watch the thread idle metrics. Spreading data across several disks helps with scattered writes. Keeping leadership balanced avoids overloading one broker.

Design levers

flowchart TD
    A["Latency too high after adding partitions"] --> B{"Which stage grew?"}
    B -->|"Producer queue time"| C["Raise linger and batch size, check buffer memory"]
    B -->|"Broker remote wait"| D["Add replica fetchers, balance leaders, check followers"]
    B -->|"Broker request queue"| E["Reduce request count, add threads or brokers"]
    B -->|"Consumer lag"| F["Add consumers up to the partition count, tune processing"]
    B -->|"Rebalance pauses"| G["Use cooperative assignment and static membership"]
    B -->|"Failover time"| H["Reduce partitions per broker, review controller design"]
        

The diagram is a map from symptoms to remedies. Start at the top with the observation that latency rose. Identify which stage grew using the measurements from the previous chapter, and then follow the matching branch. Trying all remedies at once makes it impossible to know which one worked.

Most partition problems come from a small set of habits, and knowing them makes them easy to avoid.

These illustrative stories are invented for teaching. They show how the ideas of the earlier chapters combine when you face a real symptom.

flowchart TD
    A["Symptom: high end-to-end latency"] --> B{"Is consumer lag large while brokers are idle?"}
    B -->|"Yes"| C["Parallelism shortage: more partitions and consumers"]
    B -->|"No"| D{"Are batches tiny and request rate high?"}
    D -->|"Yes"| E["Too many partitions for the traffic: tune linger or reduce count"]
    D -->|"No"| F{"Is remote wait or follower lag high?"}
    F -->|"Yes"| G["Replication load: more fetchers, balance leaders"]
    F -->|"No"| H{"Do spikes align with restarts or membership changes?"}
    H -->|"Yes"| I["Rebalance or recovery cost: cooperative assignment, fewer partitions per broker"]
    H -->|"No"| J["Look at hot keys, disk, garbage collection, and network"]
        

The decision path asks four questions in order, from the most common cause to the least. It is not a substitute for measurement, but it gives you a sensible order in which to look, and it stops you from adding partitions when the real problem is elsewhere.

No. When work is queuing because of too little parallelism, more partitions reduce latency. Beyond the sweet spot, overhead grows and latency rises. The effect depends on your traffic, hardware, and settings.

Throughput rises with parallelism until another limit, such as disk, network, or CPU, is reached. After that, extra partitions add only overhead.

It depends on your setup, but the most common are thin producer batches for low-traffic topics, replication rounds on busy brokers, and slow failover and rebalances at very high totals.

Guidance in the past was in the range of a few thousand partition replicas per broker for classic setups, with higher numbers possible in newer versions using the built-in metadata quorum. Treat it as an order of magnitude and test on your own hardware and version.

No, Kafka only allows increasing the count of an existing topic. To reduce it, create a new topic and migrate data and clients.

For keyed messages, the key-to-partition mapping changes, so new messages for a key may go to a different partition than older ones, and ordering across the change is not guaranteed.

Because each partition gets fewer messages, batches stay small and wait for the linger time, so requests are more numerous and each message may wait for its batch timer.

In one group, each partition is read by at most one consumer, so the partition count is the ceiling on useful consumers. Extra consumers stay idle.

Yes. Each partition has several replicas, and all replicas count toward broker load. A higher replication factor multiplies the number of logs and the replication traffic.

Yes. Improvements such as incremental fetch sessions and the built-in metadata quorum raise practical limits and speed up failover. Always check the documentation for the version you run.

Settings that do not wait for replication show the smallest effect on the producer’s send time, but they trade away safety, and consumers still wait for replication before they can see the data.

It is a reasonable choice when bulk topics with many partitions would disturb critical ones. Separate brokers or clusters isolate the effects of heavy traffic and large metadata.

Estimate the count from throughput, add headroom, then run load tests at that count and a step above and below while recording percentile latency at each stage. Pick the count at the flat bottom of the curve.

Yes, if processing is the bottleneck. A consumer can hand messages to a pool of workers, but it must handle ordering and offset commits carefully so that messages are not lost or committed too early.

A broker with many partitions has many logs to open and check, especially after an unclean shutdown. More recovery threads and clean shutdowns help.

Partitions are Kafka’s lanes. Too few lanes create traffic jams, because work queues up behind a parallelism limit. Too many lanes create overhead: thin producer batches, longer replication rounds, heavier metadata, slower rebalances, and slower recovery. Latency therefore follows a U-shaped curve, with a sweet spot that depends on message rate, hardware, cluster size, settings, and software version.

The way forward is not to memorize a number but to follow a method: size from throughput, add headroom, check per-broker totals including replicas, measure percentiles at every stage, test failures, and tune one lever at a time. Watch the tail, watch the operating system limits, and remember that changing the count later disturbs keyed data.