Before we can understand a partitioner, we need to know what it is partitioning and why.

What is Apache Kafka?

Apache Kafka is a system for moving streams of messages between applications. One application writes messages into Kafka, and other applications read them out, sometimes seconds later and sometimes days later. Kafka stores the messages safely on disk and can handle millions of them every second. Companies use it to track website clicks, process payments, collect sensor readings, and connect hundreds of internal services without those services needing to talk to each other directly.

In Kafka’s language, a single message is called a record (or an event). A record has a value, which is the actual content, and it can also have a key, which is an optional label attached to the content. It also carries a timestamp and optional headers, but the key and value are the parts that matter most for this topic.

Topics and partitions

Records are not thrown into one giant pile. They are organised into topics, and a topic is like a named mailbox for one kind of record, such as “orders”, “payments” or “page-views”. A busy topic would be too big and too slow for a single machine, so Kafka splits every topic into smaller pieces called partitions.

A partition is an ordered, append-only log. New records are always added to the end, and each record gets a position number called an offset. Different partitions of the same topic can live on different servers (called brokers), which means the work of storing and serving the topic is shared across many machines.

Why partitions matter

Partitions give Kafka two big powers. The first is scalability: more partitions mean more machines can share the load. The second is parallelism: one consumer can read from each partition at the same time, so a topic with twelve partitions can be read by up to twelve consumers in one group working side by side. The price of these powers is that the producer must decide, for every single record, which partition should receive it. That decision is the job of the partitioner.

The partitioner is the small piece of producer logic that answers one question: “Which partition does this record go to?”

Every time your application hands a record to the Kafka producer, the producer must pick a destination partition before it can send anything. It does not ask the broker, and it does not ask you. It uses a component called the partitioner. The partitioner looks at the record, looks at the topic’s current layout, and returns a partition number.

Two situations the partitioner handles

Records come in two flavours, and the partitioner treats them very differently.

Why not always pick the same partition?

If the partitioner sent every keyless record to one partition forever, that partition and its broker would be overloaded while the others sat idle. A good partitioner must spread records across all partitions over time. The interesting design question is how to spread them, and that question is the whole story of this guide.

Before Kafka 2.4, keyless records were spread one by one in a circle.

In older Kafka versions, the default partitioner handled keyless records with a strategy called round-robin. It kept a counter. The first record went to partition 0, the second to partition 1, the third to partition 2, and after the last partition the counter wrapped around to partition 0 again. Every partition received exactly the same number of records, give or take one.

Why round-robin looked like a good idea

On paper, round-robin is perfectly fair. Each partition gets the same amount of data, so no broker is overloaded and consumers reading different partitions all have similar work. Nobody complained about fairness. The trouble was hiding somewhere else: in how the producer sends data over the network.

The hidden cost of being too fair

Kafka producers do not send records one at a time. That would be very slow, because every network request has a fixed cost no matter how small the payload is. Instead, the producer collects records into groups called batches and sends a whole batch in one go. There is one batch waiting for each partition. When round-robin deals records one to each partition, every batch grows very slowly, because each one only receives a record every third or fourth turn. Batches leave the producer small and half empty.

To appreciate why the Sticky Partitioner exists, you need to understand how a producer batches records.

How the producer collects records

When your code calls the send method, the record does not go straight to the network. It is first placed into an in-memory area called the accumulator. Inside the accumulator, the producer keeps one queue per partition. Each queue holds batches of records that are waiting to be sent to that partition. A separate background thread, often called the sender thread, watches these queues and picks up batches that are ready.

When is a batch ready?

A batch is ready to leave for the broker in one of two situations. Either it has become full, or it has waited long enough. Two settings control this behaviour.

Why bigger batches are better

Bigger batches help in several ways at the same time.

Now the puzzle is clear. Batching is powerful, but round-robin scatters records so thinly across many partitions that each batch stays small. The Sticky Partitioner was invented to solve exactly this puzzle without giving up fairness in the long run.

The idea fits in one sentence: for keyless records, stick to one partition until the current batch is done, then move on.

The Sticky Partitioner is a partitioning strategy for records that have no key. Instead of changing the partition on every record, it picks one partition and keeps sending all keyless records there. It stays with that partition until the batch being filled is complete, meaning it is full or its waiting time is over. At that moment it picks a new partition and “sticks” to that one for the next batch, and so on.

Where did it come from?

Kafka changes are discussed in public documents called Kafka Improvement Proposals, or KIPs. The Sticky Partitioner arrived through KIP-480, and it shipped in Apache Kafka 2.4. From that version onward, the standard producer used it automatically for keyless records. Nobody needed to install anything or switch anything on, which is one reason many teams benefited without even noticing.

The name explained

The word “sticky” describes the behaviour exactly. The partitioner is like a piece of gum: it sticks to a partition for a while and only lets go when the batch is finished. The word has nothing to do with the sticky assignor used by consumers, which is a completely different feature that we will clear up later in the FAQ.

A day in the life of a sticky producer

Let us watch one producer for a few seconds. A shopping website sends page-view events to a topic with four partitions. The producer starts with no memory, so it picks partition 2 at random. For the next few hundred milliseconds, every page-view event goes into the partition 2 batch. The batch becomes full, the sender thread ships it in one request, and the partitioner now picks a new partition, say partition 0. The next few hundred events go there. Then partition 3, then partition 1, then partition 2 again, and so on.

From the point of view of the website, nothing looks different. Events go in and are stored. From the point of view of the network, the picture is very different: instead of thousands of tiny messages flying to four partitions at once, there is a steady stream of well-packed batches, one after the other. The brokers do less bookkeeping, and the producer spends less time waiting for acknowledgements. This quiet improvement is why the change was welcomed by so many teams.

Let us follow a keyless record from your application all the way to the broker.

flowchart TD
    A[Keyless record arrives] --> B{Is there a current sticky partition?}
    B -- No --> C[Pick a random available partition]
    B -- Yes --> D[Use the current sticky partition]
    C --> D
    D --> E[Add record to that partition's batch]
    E --> F{Is the batch full or timed out?}
    F -- No --> A
    F -- Yes --> G[Send batch to the broker]
    G --> H[Choose a different partition for the next batch]
    H --> A
        

A small worked example

Suppose a topic has three partitions and your application sends twelve keyless records in a short burst. With the old round-robin approach, each partition receives four records, so three small batches of four records each are created and three separate requests go out. With the sticky approach, all twelve records fit into one batch for a single partition, so one request goes out carrying all twelve. The next burst will stick to a different partition, so over time the three partitions still share the load equally.

Both strategies are fair in the long run. The difference is what happens inside the producer.

flowchart LR
    subgraph RR[Round-Robin]
        direction TB
        R1[Record 1] --> P0[Partition 0 batch]
        R2[Record 2] --> P1[Partition 1 batch]
        R3[Record 3] --> P2[Partition 2 batch]
        R4[Record 4] --> P0
    end
    subgraph ST[Sticky]
        direction TB
        S1[Record 1] --> Q0[Partition 0 batch]
        S2[Record 2] --> Q0
        S3[Record 3] --> Q0
        S4[Record 4] --> Q0
    end
        

The table shows that the sticky approach does not remove fairness; it delays it. Over a minute of steady traffic, each partition still ends up with roughly the same count of records. What changes is the shape of the traffic leaving the producer: fewer, fatter requests instead of many thin ones.

A lot of confusion about partitioners comes from mixing up keyed and keyless records.

Keyed records ignore the sticky logic

If your record has a key, the default partitioner hashes the key and computes the partition from the hash. The result depends only on the key and on the number of partitions. The sticky logic is never consulted. So a customer with ID 4821 will always land in the same partition, batch after batch, exactly as before.

Ordering guarantees

Kafka only promises ordering within one partition. Records with the same key go to the same partition, so they keep their order. Keyless records were never promised any order across partitions, because they were already being spread around. The Sticky Partitioner therefore does not break any ordering promise that existed before. In fact, keyless records that stick to the same partition for a while are now stored in the order they were produced, which is a small bonus but not a guarantee you should build on.

Choosing between keyed and keyless

Fairness has a time dimension. Look at one second and sticky looks uneven; look at one hour and it looks balanced.

Short term versus long term

Because the partitioner sticks to one partition per batch, a single burst of records lands in one place. If you draw a chart of records per partition for one second, you may see one tall bar and two short ones. If you draw the same chart for ten minutes, the bars become almost equal. The partitioner picks each new sticky partition at random, and randomness evens out when you repeat it many times, just like flipping a coin a thousand times gives close to five hundred heads.

Many producers make it even smoother

Real systems rarely have only one producer. A web application may run fifty copies, and each copy has its own producer with its own sticky choice. Each producer picks its partition independently, so at any moment different producers are stuck to different partitions. When you add all of them together, the load on each partition looks smooth and balanced even in short windows. The unevenness is mostly visible when you have very few producers sending very little data.

What about low-traffic producers?

If a producer sends only a trickle of records, a batch may take a long time to fill or to hit its waiting limit. During that time the producer keeps sticking to one partition. Over many hours, however, the partitions still rotate, so the imbalance stays temporary. Teams that notice a partition with a much larger backlog should first check whether their traffic is so light that the switch rarely happens, and whether some brokers are slower than others, before blaming the partitioner.

The Sticky Partitioner decides where records go. A handful of producer settings decide how well the batches perform.

The role of linger.ms

Many beginners ask, “If linger.ms is zero, does sticky still help?” It does. Even with no deliberate waiting, the sender thread can only carry a limited number of requests at once. While it is busy sending, new records keep arriving and pile up in the same sticky batch. So the batch still grows, just less than it would with a small deliberate wait. Adding a few milliseconds of linger usually makes batches much fuller. Newer Kafka releases have started using a small non-zero default for linger.ms, which shows how well this idea has worked in practice, but you should always check the documentation of the exact version you run.

Sizing batches for your traffic

A useful way to think about batch size is to ask how many records arrive per partition in the time you are willing to wait. Suppose your producer sends about ten thousand small records every second and you are happy to wait five milliseconds. In those five milliseconds, roughly fifty records arrive. If each record is one kilobyte, that is about fifty kilobytes, which is more than the common sixteen-kilobyte default. In this case the batch size limit, and not the waiting time, decides when batches leave, so a larger limit would help. If instead only ten records arrive in that window, the batch leaves because of the waiting time and the size limit does not matter at all.

This simple calculation prevents a lot of guesswork. Estimate the record rate, estimate the record size, multiply by the waiting time you can afford, and compare the result with your batch size. Then confirm your estimate by reading the average batch size metric on a running system. Numbers from real traffic always beat guesses.

How the settings work together

A clear mental picture of the producer’s internal parts makes every later idea easier.

The Kafka producer is a small pipeline made of a few cooperating parts. Your application thread hands over a record. The record passes through optional interceptors, then gets serialised into bytes, then the partitioner picks its partition, and finally the record is appended to a batch in the accumulator. The sender thread runs in the background, collects ready batches and delivers them to the brokers over the network.

sequenceDiagram
    participant App as Application Thread
    participant Ser as Serializer
    participant Part as Partitioner
    participant Acc as Accumulator
    participant Snd as Sender Thread
    participant Brk as Broker
    App->>Ser: Hand over record
    Ser->>Part: Serialized key and value
    Part->>Acc: Record plus chosen partition
    Acc-->>Acc: Add to batch for that partition
    Snd->>Acc: Ask for ready batches
    Acc->>Snd: Full or timed-out batches
    Snd->>Brk: One request with batches
    Brk-->>Snd: Acknowledgement
        

What each part does

Notice how tightly the partitioner and the accumulator work together. The sticky logic is aware of batch completion: the trigger for a switch is the accumulator telling the partitioner that a new batch is needed. This connection is what makes the strategy “sticky per batch” rather than “sticky per second”.

The Sticky Partitioner was not the end of the story. Kafka kept improving how keyless records are placed.

UniformStickyPartitioner

Kafka also offered a variant called the UniformStickyPartitioner. The standard default partitioner used sticky behaviour for keyless records but still hashed records that had keys. The uniform variant ignored keys completely and applied sticky behaviour to every record. That is useful only in special cases where you truly do not care about key-based placement but still send keys for some other reason. It was rarely the right choice, because ignoring keys breaks the “same key, same partition” promise that most applications rely on.

The problem with a slow broker

As people ran the sticky approach at large scale, they noticed a subtle weakness. Imagine one broker is slow. Batches destined for its partitions take longer to be confirmed, so records for those partitions pile up in memory and make the batches larger. In some situations this caused the slower broker to receive a bigger share of the keyless records than the healthy ones, which is the opposite of what you want. Also, when the linger time was zero, the switch could happen so often that batches stayed small again.

Improvements in newer versions

A later proposal, KIP-794 (delivered around Kafka 3.3), changed the approach in two important ways. First, the producer switches partitions after a certain number of bytes have been produced to the current partition, so distribution stays uniform even when batches are small. Second, it introduced adaptive partitioning: the producer keeps track of how many records are waiting for each partition and gives partitions with shorter queues a larger chance of being chosen. Partitions on slow or struggling brokers get fewer records, and healthy partitions get more. There is also an option that lets the producer temporarily avoid partitions whose brokers have been unresponsive for a set time.

Why the version you run matters

Because the behaviour changed over several releases, two teams running different Kafka client versions can see slightly different results from the same application code. A team on an older client may still see plain round-robin for keyless records. A team on a client from the sticky era sees per-batch stickiness. A team on a recent client also gets byte-based switching and adaptive weighting. If you compare performance numbers between environments, first compare client versions, since the client library, and not the broker, holds the partitioner logic.

It is also worth remembering that the broker does not know or care which strategy the producer used. The broker only sees batches arriving for particular partitions. This is why upgrading producer clients can improve performance without touching the cluster at all.

Every design choice is a trade. Here is an honest look at both sides.

When the trade-offs matter most

For high-throughput logging, metrics and event streaming, the advantages clearly win. The short-term unevenness is invisible when many producers send together, and the savings in requests are large. For extremely low-volume topics with just one producer, the difference between strategies is small, so there is little to gain and little to lose.

You cannot improve what you do not measure. Kafka producers expose metrics that tell the story clearly.

Watching the broker side

On the broker side, look at bytes-in per partition. Over several minutes the numbers should be close to each other for a keyless topic. A large, lasting gap usually points to keyed traffic with a few hot keys, to a very small number of producers, or to a slow broker, not to a broken partitioner. Also watch consumer lag per partition, because uneven lag can reveal that one partition is receiving more than its fair share.

Logging and dashboards

Build a dashboard that puts average batch size, request rate and queue time side by side. When you change linger or batch size, you will see the three lines move together, and you can judge the effect at a glance. Keep a record of the settings you tried, since tuning is an experiment and good notes save a lot of repeated work.

The partitioner is about placement and performance, but it lives inside a producer that also cares about safety.

Security is unaffected by partition choice

Access control in Kafka works at the topic level. A producer that is allowed to write to a topic can write to any of its partitions, so the sticky choice does not change who can write or read anything. Encryption in transit, authentication and permissions work exactly the same way. If you use compression together with encryption, remember that larger batches also mean larger encrypted requests, but this is a performance detail and not a security weakness.

Reliability settings still matter

Choosing a partition does not make data safe. Safety comes from other settings: the acknowledgement level, the number of replicas, the minimum in-sync replicas and idempotence. Idempotence prevents duplicates when the producer retries a request, and it works batch by batch. Larger sticky batches mean a retry re-sends more records at once, but idempotence makes sure the broker stores them only once.

What happens when a broker fails?

If the leader broker of the current sticky partition becomes unavailable, the producer retries according to its retry settings, and metadata is refreshed to find the new leader. For keyless records, the partitioner prefers partitions that are currently available, so it will move away from a partition that has no leader. Records already waiting in a batch for the failed partition are retried or eventually time out, and your application receives the error through its callback, where it should handle it properly.

Knowing the feature is half the job. Knowing when and how to use it is the other half.

Good habits

Anti-patterns to avoid

Sticky behaviour shines wherever many small, independent records flow in constantly.

Walking through a before-and-after story

Consider a company that collects logs from two hundred application servers into a topic with twelve partitions. Every server has its own producer and sends log lines without a key. Under the old round-robin behaviour, each producer spread its lines across all twelve partitions. Each batch stayed tiny, so the cluster received a huge number of small requests, and the brokers spent much of their effort handling request overhead rather than storing data.

After upgrading the clients to a version with sticky behaviour, each producer began filling one partition’s batch at a time. The number of requests dropped sharply, average batch size grew, and compression worked better because each batch held many similar log lines. Network use fell, broker processor use fell, and delivery became faster on average. The team did not change a single line of application code and did not touch the partition count. They only upgraded a library and left the key empty, which shows how much value hides in a well-chosen default.

A producer-side feature still changes what the reading side sees, so it is worth understanding both halves.

Consumers read partitions, not producers

A consumer group divides the partitions of a topic among its members. If a topic has six partitions and the group has three consumers, each consumer usually reads two partitions. The consumers have no idea which producer wrote a record or which strategy placed it. They only see the records that appear in their partitions. Because the Sticky Partitioner keeps long-term volumes close to equal, each consumer ends up with a similar amount of work, which is exactly what a healthy group needs.

Short bursts can cause short-lived lag differences

Since one partition may receive a burst before others, the consumer reading that partition may show a little more lag for a short time. Once the sticky partition changes, the next burst arrives elsewhere and the lag evens out. If you see lag that stays high on one partition for a long time, the reason is almost never the sticky switching. Look instead for a slow consumer, a heavy key, or an expensive record that takes a long time to process.

Ordering on the reading side

A consumer that reads several partitions merges them in whatever order data arrives, and it never gets a global order across partitions. This is true whether the records were placed by round-robin or by stickiness. So designing consumers to work without cross-partition order was always the right approach, and the Sticky Partitioner does not make this more or less true.

Batches are also a gift to consumers

Bigger producer batches are stored as bigger record batches on disk. When consumers fetch data, they can read large contiguous chunks instead of many small ones, which makes reading faster and cheaper. The broker can also send data to consumers efficiently because it reads sequentially from disk. So the benefits of sticky batching travel through the whole pipeline, from producer to broker to consumer.

The Sticky Partitioner works best when the topic is set up sensibly, and it is easier to blame than to diagnose. Here is a calm way to look at problems.

How many partitions should a topic have?

There is no single correct number, but a few simple ideas guide the choice. Think about the highest read speed you will need, and divide it by the speed one consumer can handle. That gives the minimum partitions for reading in parallel. Think about the highest write speed and divide it by what one partition can absorb. Take the larger of the two numbers and add some room for growth. Avoid going wildly higher than needed, because every partition adds files, memory use and recovery time on the brokers.

For keyless topics, the partition count affects how smoothly sticky switching spreads data. With very few partitions, each switch has few choices, and randomness takes longer to even out. With many partitions and many producers, balance is excellent. Choose a count that matches your real parallelism, not a number that merely sounds impressive.

A calm troubleshooting checklist

No. Since Kafka 2.4, the default behaviour for records without a key is already sticky. You only need to leave the key empty. If you use a custom partitioner or an older client, you may not get it, so check your client version.

No. Keyed records are still hashed, and the same key always maps to the same partition as long as the number of partitions stays the same.

No, they are two unrelated features that share a word. The Sticky Partitioner lives in the producer and decides where new records are written. The sticky assignor lives in the consumer group and tries to keep partition assignments stable when consumers join or leave.

Over a long enough time and with enough producers, yes, they will be close to equal. In very short windows, one partition may receive more because of the batch-by-batch switching.

Because fewer, larger requests put less pressure on the sender and the broker, and records spend less time queued behind many small requests. The saved queueing time usually outweighs the short time spent filling a batch.

Often yes. A small linger of a few milliseconds usually makes batches fuller, which adds to the benefit. Test it against your latency requirements.

No. Partition choice has nothing to do with durability. Message safety depends on acknowledgements, replication and retry settings.

Then there is nothing to choose, and every record goes to that one partition. Batching still works normally.

Keyless records will simply start using the new partitions too. Keyed records may map to different partitions after the count changes, which is a separate concern that affects ordering for those keys.

Only when you have a clear business rule for placement, such as sending records of a premium customer to a dedicated partition. For plain load spreading, the built-in behaviour is well tested and usually better.

The Sticky Partitioner is a simple idea with a big effect. Instead of dealing keyless records one by one to every partition, the producer keeps filling a single partition’s batch until it is done and only then moves to another partition. Over time every partition receives about the same share, but the producer sends fewer, larger, better-compressed requests. It arrived in Kafka 2.4 through KIP-480 and was refined in later versions with byte-based switching and adaptive weighting. Using it correctly mostly means understanding when to leave the key empty and how to tune batching around it.