What Is the Sticky Partitioner in Kafka?
A plain-English guide to the Kafka producer feature that keeps records without a key "stuck" to one partition for a while, so they travel in bigger batches, use fewer network requests and arrive with lower latency.
Imagine you run a small courier shop. Every day, hundreds of parcels arrive at your counter, and you have three delivery vans waiting outside. One way to work is to hand each parcel to a different van in turn: first parcel to van one, second to van two, third to van three, and so on. Every van leaves half empty, drives many short trips, and burns a lot of fuel. Another way is to keep loading van one until it is full, send it off, and only then start loading van two. The parcels still reach their destinations, but you make far fewer trips. Kafka’s Sticky Partitioner is the second way of working. This guide explains it from zero, so you do not need any earlier Kafka knowledge to follow along.
1The Big Picture: How Data Gets Into Kafka
Before we can understand a partitioner, we need to know what it is partitioning and why.
What is Apache Kafka?
Apache Kafka is a system for moving streams of messages between applications. One application writes messages into Kafka, and other applications read them out, sometimes seconds later and sometimes days later. Kafka stores the messages safely on disk and can handle millions of them every second. Companies use it to track website clicks, process payments, collect sensor readings, and connect hundreds of internal services without those services needing to talk to each other directly.
In Kafka’s language, a single message is called a record (or an event). A record has a value, which is the actual content, and it can also have a key, which is an optional label attached to the content. It also carries a timestamp and optional headers, but the key and value are the parts that matter most for this topic.
Think of Kafka as a huge post office. Applications that write records are people dropping letters into the post office. Applications that read records are people collecting their mail. The post office keeps every letter in order and does not throw a letter away just because someone has already read it.
Topics and partitions
Records are not thrown into one giant pile. They are organised into topics, and a topic is like a named mailbox for one kind of record, such as “orders”, “payments” or “page-views”. A busy topic would be too big and too slow for a single machine, so Kafka splits every topic into smaller pieces called partitions.
A partition is an ordered, append-only log. New records are always added to the end, and each record gets a position number called an offset. Different partitions of the same topic can live on different servers (called brokers), which means the work of storing and serving the topic is shared across many machines.
Producer
An application that writes records into Kafka. Your web server sending click events is a producer.
Broker
A Kafka server that stores partitions and answers requests from producers and consumers.
Topic
A named category of records, split into one or more partitions.
Partition
One ordered log inside a topic. Order is guaranteed only inside a single partition.
Offset
The position number of a record inside its partition, like a page number in a notebook.
Consumer
An application that reads records from Kafka, usually as part of a group that shares the work.
Why partitions matter
Partitions give Kafka two big powers. The first is scalability: more partitions mean more machines can share the load. The second is parallelism: one consumer can read from each partition at the same time, so a topic with twelve partitions can be read by up to twelve consumers in one group working side by side. The price of these powers is that the producer must decide, for every single record, which partition should receive it. That decision is the job of the partitioner.
2What Is a Partitioner?
The partitioner is the small piece of producer logic that answers one question: “Which partition does this record go to?”
Every time your application hands a record to the Kafka producer, the producer must pick a destination partition before it can send anything. It does not ask the broker, and it does not ask you. It uses a component called the partitioner. The partitioner looks at the record, looks at the topic’s current layout, and returns a partition number.
A partitioner is like the sorting clerk at the post office. A letter arrives, the clerk glances at it, and drops it into the right bin: north, south or east. The clerk never delivers the letter, they only choose the bin.
Two situations the partitioner handles
Records come in two flavours, and the partitioner treats them very differently.
Records that have a key
If a record carries a key, such as a customer ID, the partitioner runs the key through a hashing function and turns the result into a partition number. The same key always produces the same partition. This is how Kafka keeps all events for one customer in order: they all land in the same partition.
Records that have no key
If a record has no key, there is nothing to hash and no reason to prefer one partition over another. The partitioner is free to choose any partition. This is exactly the situation where the Sticky Partitioner does its work, and it is the situation we will focus on for most of this guide.
The Sticky Partitioner only changes how records without a key are placed. Records with a key still follow the “same key, same partition” rule that Kafka has always used.
Why not always pick the same partition?
If the partitioner sent every keyless record to one partition forever, that partition and its broker would be overloaded while the others sat idle. A good partitioner must spread records across all partitions over time. The interesting design question is how to spread them, and that question is the whole story of this guide.
3Life Before Sticky: The Round-Robin Approach
Before Kafka 2.4, keyless records were spread one by one in a circle.
In older Kafka versions, the default partitioner handled keyless records with a strategy called round-robin. It kept a counter. The first record went to partition 0, the second to partition 1, the third to partition 2, and after the last partition the counter wrapped around to partition 0 again. Every partition received exactly the same number of records, give or take one.
Round-robin is like dealing playing cards to players around a table. One card to each player, in turn, until the deck is finished. Everybody ends up with a fair share, but the dealer makes a lot of small movements.
Why round-robin looked like a good idea
On paper, round-robin is perfectly fair. Each partition gets the same amount of data, so no broker is overloaded and consumers reading different partitions all have similar work. Nobody complained about fairness. The trouble was hiding somewhere else: in how the producer sends data over the network.
The hidden cost of being too fair
Kafka producers do not send records one at a time. That would be very slow, because every network request has a fixed cost no matter how small the payload is. Instead, the producer collects records into groups called batches and sends a whole batch in one go. There is one batch waiting for each partition. When round-robin deals records one to each partition, every batch grows very slowly, because each one only receives a record every third or fourth turn. Batches leave the producer small and half empty.
Many beginners think “perfectly even” always means “best.” Round-robin was perfectly even, yet it made the producer slower, because small batches mean more requests, more overhead and more waiting.
4Understanding Batching: The Key to the Whole Story
To appreciate why the Sticky Partitioner exists, you need to understand how a producer batches records.
How the producer collects records
When your code calls the send method, the record does not go straight to the network. It is first placed into an in-memory area called the accumulator. Inside the accumulator, the producer keeps one queue per partition. Each queue holds batches of records that are waiting to be sent to that partition. A separate background thread, often called the sender thread, watches these queues and picks up batches that are ready.
When is a batch ready?
A batch is ready to leave for the broker in one of two situations. Either it has become full, or it has waited long enough. Two settings control this behaviour.
batch.size
The maximum size, in bytes, of one batch for one partition. The common default is 16 KB. When a batch reaches this size, it is full and can be sent.
linger.ms
How long the producer is willing to wait for more records before sending a batch that is not full yet. A small wait can create bigger batches.
A batch is a shuttle bus. It leaves when every seat is taken (batch.size) or when the driver’s waiting time is over (linger.ms), whichever happens first. A bus that leaves with two passengers wastes fuel. A bus that leaves full is efficient.
Why bigger batches are better
Bigger batches help in several ways at the same time.
Fewer network requests
One request can carry hundreds of records instead of a handful. The fixed cost of each request is shared among many records.
Better compression
Compression works on a whole batch. The more similar records you compress together, the smaller the result, which saves network and disk space.
Less work for the broker
Each request the broker receives needs parsing, validation and a disk write. Fewer, larger requests mean less repeated overhead.
Lower overall latency
It sounds strange, but larger batches can reduce the average time a record waits. Less queueing at the sender and the broker means each record gets processed sooner.
Now the puzzle is clear. Batching is powerful, but round-robin scatters records so thinly across many partitions that each batch stays small. The Sticky Partitioner was invented to solve exactly this puzzle without giving up fairness in the long run.
5So, What Is the Sticky Partitioner?
The idea fits in one sentence: for keyless records, stick to one partition until the current batch is done, then move on.
The Sticky Partitioner is a partitioning strategy for records that have no key. Instead of changing the partition on every record, it picks one partition and keeps sending all keyless records there. It stays with that partition until the batch being filled is complete, meaning it is full or its waiting time is over. At that moment it picks a new partition and “sticks” to that one for the next batch, and so on.
Go back to the courier shop. The sticky approach is loading van one until it is full, sending it away, then loading van two. Over a whole day every van gets used equally, but each trip carries a full load.
Where did it come from?
Kafka changes are discussed in public documents called Kafka Improvement Proposals, or KIPs. The Sticky Partitioner arrived through KIP-480, and it shipped in Apache Kafka 2.4. From that version onward, the standard producer used it automatically for keyless records. Nobody needed to install anything or switch anything on, which is one reason many teams benefited without even noticing.
The name explained
The word “sticky” describes the behaviour exactly. The partitioner is like a piece of gum: it sticks to a partition for a while and only lets go when the batch is finished. The word has nothing to do with the sticky assignor used by consumers, which is a completely different feature that we will clear up later in the FAQ.
A day in the life of a sticky producer
Let us watch one producer for a few seconds. A shopping website sends page-view events to a topic with four partitions. The producer starts with no memory, so it picks partition 2 at random. For the next few hundred milliseconds, every page-view event goes into the partition 2 batch. The batch becomes full, the sender thread ships it in one request, and the partitioner now picks a new partition, say partition 0. The next few hundred events go there. Then partition 3, then partition 1, then partition 2 again, and so on.
From the point of view of the website, nothing looks different. Events go in and are stored. From the point of view of the network, the picture is very different: instead of thousands of tiny messages flying to four partitions at once, there is a steady stream of well-packed batches, one after the other. The brokers do less bookkeeping, and the producer spends less time waiting for acknowledgements. This quiet improvement is why the change was welcomed by so many teams.
The Sticky Partitioner trades short-term evenness for long-term evenness. In a single second, one partition may get more records than the others. Over minutes and hours, all partitions receive nearly the same amount.
6How the Sticky Partitioner Works, Step by Step
Let us follow a keyless record from your application all the way to the broker.
Your application sends a record without a key
The producer receives the record and sees that the key is empty. It asks the partitioner which partition to use.
The partitioner checks its current sticky partition
For each topic, the partitioner remembers one “current” partition. On the very first record, there is none yet, so it chooses one at random from the partitions that are currently available.
The record joins the batch for that partition
The record is placed into the accumulator, in the queue for the sticky partition. Because the next records will also go there, that batch fills up quickly.
More records stick to the same partition
Record two, three, four and many more are assigned to the same partition. The batch grows quickly and steadily toward its size limit.
The batch completes
The batch either reaches its size limit or its waiting time (linger.ms) runs out. The producer marks it ready, and the sender thread ships it to the broker leading that partition.
The partitioner switches to a new partition
When a new batch has to be started, the partitioner picks a new sticky partition at random, and it tries to avoid choosing the same one again. The cycle repeats from step 3.
flowchart TD
A[Keyless record arrives] --> B{Is there a current sticky partition?}
B -- No --> C[Pick a random available partition]
B -- Yes --> D[Use the current sticky partition]
C --> D
D --> E[Add record to that partition's batch]
E --> F{Is the batch full or timed out?}
F -- No --> A
F -- Yes --> G[Send batch to the broker]
G --> H[Choose a different partition for the next batch]
H --> A
A small worked example
Suppose a topic has three partitions and your application sends twelve keyless records in a short burst. With the old round-robin approach, each partition receives four records, so three small batches of four records each are created and three separate requests go out. With the sticky approach, all twelve records fit into one batch for a single partition, so one request goes out carrying all twelve. The next burst will stick to a different partition, so over time the three partitions still share the load equally.
The partitioner also skips partitions that currently have no available leader. For keyless records it prefers partitions that can actually accept data right now.
7Sticky Partitioner vs Round-Robin
Both strategies are fair in the long run. The difference is what happens inside the producer.
flowchart LR
subgraph RR[Round-Robin]
direction TB
R1[Record 1] --> P0[Partition 0 batch]
R2[Record 2] --> P1[Partition 1 batch]
R3[Record 3] --> P2[Partition 2 batch]
R4[Record 4] --> P0
end
subgraph ST[Sticky]
direction TB
S1[Record 1] --> Q0[Partition 0 batch]
S2[Record 2] --> Q0
S3[Record 3] --> Q0
S4[Record 4] --> Q0
end
| Aspect | Round-Robin | Sticky Partitioner |
|---|---|---|
| Partition choice per record | Changes on every record | Stays the same until the batch is done |
| Batch size | Small, fills slowly | Large, fills quickly |
| Requests per second | High | Lower |
| Latency | Higher because of small batches and queueing | Usually lower |
| Short-term balance | Nearly perfect | Uneven in short bursts |
| Long-term balance | Even | Even |
| Compression efficiency | Weaker | Stronger |
| Applies to | Keyless records | Keyless records |
The table shows that the sticky approach does not remove fairness; it delays it. Over a minute of steady traffic, each partition still ends up with roughly the same count of records. What changes is the shape of the traffic leaving the producer: fewer, fatter requests instead of many thin ones.
8Keyed Records and Ordering: What Does Not Change
A lot of confusion about partitioners comes from mixing up keyed and keyless records.
Keyed records ignore the sticky logic
If your record has a key, the default partitioner hashes the key and computes the partition from the hash. The result depends only on the key and on the number of partitions. The sticky logic is never consulted. So a customer with ID 4821 will always land in the same partition, batch after batch, exactly as before.
Ordering guarantees
Kafka only promises ordering within one partition. Records with the same key go to the same partition, so they keep their order. Keyless records were never promised any order across partitions, because they were already being spread around. The Sticky Partitioner therefore does not break any ordering promise that existed before. In fact, keyless records that stick to the same partition for a while are now stored in the order they were produced, which is a small bonus but not a guarantee you should build on.
If your business logic needs all events of one entity in order, do not rely on sticky behaviour. Give those records a proper key, such as an order ID or user ID. Keyless records can end up on any partition.
Choosing between keyed and keyless
Use a key when order or grouping matters
Bank transactions for one account, state changes of one shopping cart, or readings from one device that must be processed in sequence should carry a key.
Go keyless when records are independent
Application logs, page-view events, metrics samples and other independent facts do not need grouping. Leaving the key empty lets the producer use its sticky logic and spread data evenly.
9Is the Distribution Really Fair?
Fairness has a time dimension. Look at one second and sticky looks uneven; look at one hour and it looks balanced.
Short term versus long term
Because the partitioner sticks to one partition per batch, a single burst of records lands in one place. If you draw a chart of records per partition for one second, you may see one tall bar and two short ones. If you draw the same chart for ten minutes, the bars become almost equal. The partitioner picks each new sticky partition at random, and randomness evens out when you repeat it many times, just like flipping a coin a thousand times gives close to five hundred heads.
Imagine three checkout lanes at a supermarket and a manager who sends a whole group of customers to one lane at a time. During a single minute, one lane is busy and the others are empty. Over a whole day, every lane serves about the same number of customers.
Many producers make it even smoother
Real systems rarely have only one producer. A web application may run fifty copies, and each copy has its own producer with its own sticky choice. Each producer picks its partition independently, so at any moment different producers are stuck to different partitions. When you add all of them together, the load on each partition looks smooth and balanced even in short windows. The unevenness is mostly visible when you have very few producers sending very little data.
What about low-traffic producers?
If a producer sends only a trickle of records, a batch may take a long time to fill or to hit its waiting limit. During that time the producer keeps sticking to one partition. Over many hours, however, the partitions still rotate, so the imbalance stays temporary. Teams that notice a partition with a much larger backlog should first check whether their traffic is so light that the switch rarely happens, and whether some brokers are slower than others, before blaming the partitioner.
When you measure balance, look at a window of several minutes at least, and look at the total across all producers. A one-second snapshot from a single producer can be misleading.
10The Tuning Knobs That Work With Sticky Behaviour
The Sticky Partitioner decides where records go. A handful of producer settings decide how well the batches perform.
batch.size
The upper limit of a batch in bytes. A bigger limit lets the sticky partition collect more records before switching.
linger.ms
How long to wait for a batch to fill. A few milliseconds often create much better batches at almost no visible cost.
compression.type
The compression method for a batch. Larger sticky batches compress better, so this setting pairs very well with sticky behaviour.
buffer.memory
Total memory the producer may use to hold unsent batches. Too little memory makes the producer wait or fail when traffic spikes.
acks
How many broker confirmations are needed before a send counts as successful. It affects safety and latency but not the choice of partition.
max.in.flight.requests.per.connection
How many unconfirmed requests may be on one connection. It influences how many batches wait behind each other.
The role of linger.ms
Many beginners ask, “If linger.ms is zero, does sticky still help?” It does. Even with no deliberate waiting, the sender thread can only carry a limited number of requests at once. While it is busy sending, new records keep arriving and pile up in the same sticky batch. So the batch still grows, just less than it would with a small deliberate wait. Adding a few milliseconds of linger usually makes batches much fuller. Newer Kafka releases have started using a small non-zero default for linger.ms, which shows how well this idea has worked in practice, but you should always check the documentation of the exact version you run.
Sizing batches for your traffic
A useful way to think about batch size is to ask how many records arrive per partition in the time you are willing to wait. Suppose your producer sends about ten thousand small records every second and you are happy to wait five milliseconds. In those five milliseconds, roughly fifty records arrive. If each record is one kilobyte, that is about fifty kilobytes, which is more than the common sixteen-kilobyte default. In this case the batch size limit, and not the waiting time, decides when batches leave, so a larger limit would help. If instead only ten records arrive in that window, the batch leaves because of the waiting time and the size limit does not matter at all.
This simple calculation prevents a lot of guesswork. Estimate the record rate, estimate the record size, multiply by the waiting time you can afford, and compare the result with your batch size. Then confirm your estimate by reading the average batch size metric on a running system. Numbers from real traffic always beat guesses.
How the settings work together
Start with the defaults
Run your producer with the standard batch size and observe average batch size and request rate in the metrics.
Add a small linger
Try a few milliseconds. Watch whether the average batch grows and whether end-to-end latency stays acceptable for your use case.
Raise the batch size if batches are always full
If most batches hit the size limit, a larger limit gives even better throughput. If most batches leave nearly empty, the limit is not your bottleneck.
Turn on compression
Once batches are big, compression saves bandwidth and storage. Test the CPU cost as well, because compression uses processor time on the producer.
Setting batch.size to a huge number without checking memory. Each partition can hold its own batch, so a large batch size multiplied by a large partition count can consume a lot of producer memory.
11Inside the Producer: Where the Partitioner Sits
A clear mental picture of the producer’s internal parts makes every later idea easier.
The Kafka producer is a small pipeline made of a few cooperating parts. Your application thread hands over a record. The record passes through optional interceptors, then gets serialised into bytes, then the partitioner picks its partition, and finally the record is appended to a batch in the accumulator. The sender thread runs in the background, collects ready batches and delivers them to the brokers over the network.
sequenceDiagram
participant App as Application Thread
participant Ser as Serializer
participant Part as Partitioner
participant Acc as Accumulator
participant Snd as Sender Thread
participant Brk as Broker
App->>Ser: Hand over record
Ser->>Part: Serialized key and value
Part->>Acc: Record plus chosen partition
Acc-->>Acc: Add to batch for that partition
Snd->>Acc: Ask for ready batches
Acc->>Snd: Full or timed-out batches
Snd->>Brk: One request with batches
Brk-->>Snd: Acknowledgement
What each part does
Serializer
Turns your key and value objects into raw bytes, because Kafka stores and moves only bytes.
Partitioner
Chooses the partition number. For keyless records, this is where the sticky decision is made and where the choice of a new partition happens when a batch finishes.
Accumulator
Holds batches in memory, one queue per partition, until the sender thread is ready to send them.
Sender thread
Groups ready batches by destination broker and sends them, so one request may carry batches for several partitions that live on the same broker.
Notice how tightly the partitioner and the accumulator work together. The sticky logic is aware of batch completion: the trigger for a switch is the accumulator telling the partitioner that a new batch is needed. This connection is what makes the strategy “sticky per batch” rather than “sticky per second”.
12The Evolution: Uniform Sticky and Adaptive Partitioning
The Sticky Partitioner was not the end of the story. Kafka kept improving how keyless records are placed.
UniformStickyPartitioner
Kafka also offered a variant called the UniformStickyPartitioner. The standard default partitioner used sticky behaviour for keyless records but still hashed records that had keys. The uniform variant ignored keys completely and applied sticky behaviour to every record. That is useful only in special cases where you truly do not care about key-based placement but still send keys for some other reason. It was rarely the right choice, because ignoring keys breaks the “same key, same partition” promise that most applications rely on.
The problem with a slow broker
As people ran the sticky approach at large scale, they noticed a subtle weakness. Imagine one broker is slow. Batches destined for its partitions take longer to be confirmed, so records for those partitions pile up in memory and make the batches larger. In some situations this caused the slower broker to receive a bigger share of the keyless records than the healthy ones, which is the opposite of what you want. Also, when the linger time was zero, the switch could happen so often that batches stayed small again.
Picture delivery vans again. If one van is slow to come back from its trip, a fair manager should send fewer parcels its way. A careless manager who only loads whichever van is still standing there might end up overloading the slow van.
Improvements in newer versions
A later proposal, KIP-794 (delivered around Kafka 3.3), changed the approach in two important ways. First, the producer switches partitions after a certain number of bytes have been produced to the current partition, so distribution stays uniform even when batches are small. Second, it introduced adaptive partitioning: the producer keeps track of how many records are waiting for each partition and gives partitions with shorter queues a larger chance of being chosen. Partitions on slow or struggling brokers get fewer records, and healthy partitions get more. There is also an option that lets the producer temporarily avoid partitions whose brokers have been unresponsive for a set time.
| Generation | Behaviour for keyless records | Main benefit |
|---|---|---|
| Before 2.4 | Round-robin per record | Simple and perfectly even |
| 2.4 onward | Sticky per batch (KIP-480) | Bigger batches, fewer requests, lower latency |
| 3.3 onward | Switch by bytes plus adaptive weighting (KIP-794) | Better balance and resilience to slow brokers |
In newer releases the sticky and adaptive logic lives inside the producer itself, and the old built-in partitioner classes were deprecated and later removed in recent major versions. The idea you learned here is still exactly how keyless records behave. Always check the documentation for your own Kafka version to see the current configuration names.
Why the version you run matters
Because the behaviour changed over several releases, two teams running different Kafka client versions can see slightly different results from the same application code. A team on an older client may still see plain round-robin for keyless records. A team on a client from the sticky era sees per-batch stickiness. A team on a recent client also gets byte-based switching and adaptive weighting. If you compare performance numbers between environments, first compare client versions, since the client library, and not the broker, holds the partitioner logic.
It is also worth remembering that the broker does not know or care which strategy the producer used. The broker only sees batches arriving for particular partitions. This is why upgrading producer clients can improve performance without touching the cluster at all.
13Advantages and Disadvantages
Every design choice is a trade. Here is an honest look at both sides.
Advantages
- Larger batches, so each network request carries more records
- Fewer requests per second, which reduces load on both producer and brokers
- Lower average latency in many workloads, especially at moderate traffic
- Better compression ratios because batches contain more similar data
- Long-term even distribution across all partitions
- Works automatically with no code changes in your application
Disadvantages / Trade-offs
- Short-term unevenness, because one partition gets a burst while others wait
- No effect at all on records that already carry a key
- Hard to reason about when traffic is very light and switches are rare
- Early versions could favour slow brokers, which later versions improved
- Can hide a poor partition count choice, because balance depends on many switches over time
When the trade-offs matter most
For high-throughput logging, metrics and event streaming, the advantages clearly win. The short-term unevenness is invisible when many producers send together, and the savings in requests are large. For extremely low-volume topics with just one producer, the difference between strategies is small, so there is little to gain and little to lose.
14Monitoring: How to See Whether Sticky Is Working
You cannot improve what you do not measure. Kafka producers expose metrics that tell the story clearly.
batch-size-avg
Average bytes per batch. If sticky is helping, this number rises compared with round-robin behaviour.
records-per-request-avg
Average number of records in each request. Higher is generally better for efficiency.
request-rate
How many requests the producer sends each second. It should fall when batches grow.
record-queue-time-avg
How long records wait in the accumulator before being sent. It shows the cost of lingering.
request-latency-avg
Round-trip time for requests to the broker. It helps you spot slow brokers.
compression-rate-avg
How well batches compress. Bigger batches usually give better values.
Watching the broker side
On the broker side, look at bytes-in per partition. Over several minutes the numbers should be close to each other for a keyless topic. A large, lasting gap usually points to keyed traffic with a few hot keys, to a very small number of producers, or to a slow broker, not to a broken partitioner. Also watch consumer lag per partition, because uneven lag can reveal that one partition is receiving more than its fair share.
Logging and dashboards
Build a dashboard that puts average batch size, request rate and queue time side by side. When you change linger or batch size, you will see the three lines move together, and you can judge the effect at a glance. Keep a record of the settings you tried, since tuning is an experiment and good notes save a lot of repeated work.
15Security, Reliability and Data Safety
The partitioner is about placement and performance, but it lives inside a producer that also cares about safety.
Security is unaffected by partition choice
Access control in Kafka works at the topic level. A producer that is allowed to write to a topic can write to any of its partitions, so the sticky choice does not change who can write or read anything. Encryption in transit, authentication and permissions work exactly the same way. If you use compression together with encryption, remember that larger batches also mean larger encrypted requests, but this is a performance detail and not a security weakness.
Reliability settings still matter
Choosing a partition does not make data safe. Safety comes from other settings: the acknowledgement level, the number of replicas, the minimum in-sync replicas and idempotence. Idempotence prevents duplicates when the producer retries a request, and it works batch by batch. Larger sticky batches mean a retry re-sends more records at once, but idempotence makes sure the broker stores them only once.
Assuming that sticky behaviour improves durability. It only improves efficiency. To protect your data, use the safety settings and keep enough healthy replicas.
What happens when a broker fails?
If the leader broker of the current sticky partition becomes unavailable, the producer retries according to its retry settings, and metadata is refreshed to find the new leader. For keyless records, the partitioner prefers partitions that are currently available, so it will move away from a partition that has no leader. Records already waiting in a batch for the failed partition are retried or eventually time out, and your application receives the error through its callback, where it should handle it properly.
16Design Patterns, Anti-Patterns and Best Practices
Knowing the feature is half the job. Knowing when and how to use it is the other half.
Good habits
Leave the key empty for independent records
Logs, metrics and clickstream events do not need ordering per entity. An empty key lets the producer batch efficiently and balance data across partitions.
Use a real business key when order matters
Pick a key that groups related records, such as an account or device identifier, and accept that a very popular key becomes a busy partition.
Pair sticky batching with a small linger and compression
These three together give the largest improvement in throughput and network use for most workloads.
Choose the partition count with growth in mind
More partitions allow more parallel consumers, but each extra partition adds memory use and metadata. Plan for the next year, not only for today.
Measure before and after every change
Record batch size, request rate and latency so you can prove that a tuning change helped.
Anti-patterns to avoid
Problem
A team relies on keyless records staying in order across the whole topic, because they noticed that records seemed to arrive in sequence during testing.
Why It’s Harmful
Keyless records can land on any partition, and the sticky partition changes after every batch. Order across partitions is never guaranteed, so the application breaks the moment traffic grows or partitions change.
Correct Approach
Give records that need order a key that groups them, so they always land on the same partition.
Problem
A team sets a random key for every record “to get better spreading”, not knowing that keyless records already spread well.
Why It’s Harmful
Random keys switch off the sticky behaviour. Records scatter across partitions again, batches shrink, request counts rise and latency gets worse.
Correct Approach
Leave the key empty when you have no ordering need. Let the partitioner do the spreading.
Problem
A team blames the partitioner for uneven partitions and writes a custom partitioner without looking at the real cause.
Why It’s Harmful
Uneven partitions are more often caused by a few hot keys, by too few producers or by a slow broker. A custom partitioner adds maintenance work and can lose the batching benefits.
Correct Approach
Check key distribution, producer count, broker health and metrics first. Write custom logic only when there is a clear business reason.
17Real-World Use Cases
Sticky behaviour shines wherever many small, independent records flow in constantly.
Application and server logs
Thousands of servers write log lines to a central topic. No log line depends on another, so the records are keyless. Sticky batching packs many lines together, reducing network chatter across the data centre.
Website and mobile clickstream
Every tap, scroll and page view becomes a small event. Analytics teams care about totals, not the order of individual clicks across users, so keyless events with sticky batching are a natural fit.
Metrics and telemetry
Monitoring agents send frequent measurements such as CPU use or request counts. The stream is huge and each sample is tiny, which is the perfect setting for large compressed batches.
Internet of Things sensor feeds
Fleets of devices send readings every few seconds. When the readings are only used for aggregate dashboards, keyless sending with sticky batching cuts costs. When per-device order matters, the device ID becomes the key instead.
Data pipeline ingestion
Tools that copy data from databases or files into Kafka often send large volumes of independent rows. Sticky batching helps these bulk loads finish faster with fewer requests.
Walking through a before-and-after story
Consider a company that collects logs from two hundred application servers into a topic with twelve partitions. Every server has its own producer and sends log lines without a key. Under the old round-robin behaviour, each producer spread its lines across all twelve partitions. Each batch stayed tiny, so the cluster received a huge number of small requests, and the brokers spent much of their effort handling request overhead rather than storing data.
After upgrading the clients to a version with sticky behaviour, each producer began filling one partition’s batch at a time. The number of requests dropped sharply, average batch size grew, and compression worked better because each batch held many similar log lines. Network use fell, broker processor use fell, and delivery became faster on average. The team did not change a single line of application code and did not touch the partition count. They only upgraded a library and left the key empty, which shows how much value hides in a well-chosen default.
18The Downstream Effect: What Consumers Experience
A producer-side feature still changes what the reading side sees, so it is worth understanding both halves.
Consumers read partitions, not producers
A consumer group divides the partitions of a topic among its members. If a topic has six partitions and the group has three consumers, each consumer usually reads two partitions. The consumers have no idea which producer wrote a record or which strategy placed it. They only see the records that appear in their partitions. Because the Sticky Partitioner keeps long-term volumes close to equal, each consumer ends up with a similar amount of work, which is exactly what a healthy group needs.
Short bursts can cause short-lived lag differences
Since one partition may receive a burst before others, the consumer reading that partition may show a little more lag for a short time. Once the sticky partition changes, the next burst arrives elsewhere and the lag evens out. If you see lag that stays high on one partition for a long time, the reason is almost never the sticky switching. Look instead for a slow consumer, a heavy key, or an expensive record that takes a long time to process.
Think of three cashiers who each serve one lane. If the manager sends groups of shoppers to one lane at a time, each cashier is briefly busier than the others, but all three finish the day with the same count of shoppers served.
Ordering on the reading side
A consumer that reads several partitions merges them in whatever order data arrives, and it never gets a global order across partitions. This is true whether the records were placed by round-robin or by stickiness. So designing consumers to work without cross-partition order was always the right approach, and the Sticky Partitioner does not make this more or less true.
If a consumer must process related records in order, the producer must give those records the same key. The reading side cannot repair an order that was never protected on the writing side.
Batches are also a gift to consumers
Bigger producer batches are stored as bigger record batches on disk. When consumers fetch data, they can read large contiguous chunks instead of many small ones, which makes reading faster and cheaper. The broker can also send data to consumers efficiently because it reads sequentially from disk. So the benefits of sticky batching travel through the whole pipeline, from producer to broker to consumer.
19Planning Partition Counts and Troubleshooting
The Sticky Partitioner works best when the topic is set up sensibly, and it is easier to blame than to diagnose. Here is a calm way to look at problems.
How many partitions should a topic have?
There is no single correct number, but a few simple ideas guide the choice. Think about the highest read speed you will need, and divide it by the speed one consumer can handle. That gives the minimum partitions for reading in parallel. Think about the highest write speed and divide it by what one partition can absorb. Take the larger of the two numbers and add some room for growth. Avoid going wildly higher than needed, because every partition adds files, memory use and recovery time on the brokers.
For keyless topics, the partition count affects how smoothly sticky switching spreads data. With very few partitions, each switch has few choices, and randomness takes longer to even out. With many partitions and many producers, balance is excellent. Choose a count that matches your real parallelism, not a number that merely sounds impressive.
A calm troubleshooting checklist
Confirm the records really have no key
Some frameworks add a key without telling you. If keys exist, the sticky logic is not being used, and hot keys may explain the uneven partitions.
Check the client version
Find out which client library and version the producer uses. Older clients may still behave like round-robin, and newer ones have the improved logic.
Look for a custom partitioner
If someone configured a custom partitioner class in the past, it replaces the built-in behaviour completely. Many teams forget that such a setting exists in an old configuration file.
Study the metrics over a long window
Compare bytes-in per partition across several minutes, and compare average batch size and request rate before and after any change.
Check broker health and network
A slow disk, a busy network card or an overloaded broker can create latency and lag that look like partitioner problems but are not.
Change one thing at a time
Adjust linger, batch size or compression separately, and measure each change. Changing many settings at once hides which change made the difference.
| Symptom | Likely cause | What to check |
|---|---|---|
| One partition much busier for hours | Hot key or very few producers | Key distribution and producer count |
| Small average batch size | Low traffic, zero linger or random keys | Linger setting and key usage |
| High request rate with low throughput | Records scattered across partitions | Whether keys are being set unintentionally |
| Rising latency on one broker | Slow or overloaded broker | Broker resources and request latency metric |
| Producer memory exhausted | Large batch size times many partitions | Buffer memory and batch size settings |
Jumping straight to a custom partitioner. Most uneven-load stories end with a hot key, a tiny number of producers or a slow broker, and all three are easier to fix than a hand-written partitioning rule.
20Frequently Asked Questions
No. Since Kafka 2.4, the default behaviour for records without a key is already sticky. You only need to leave the key empty. If you use a custom partitioner or an older client, you may not get it, so check your client version.
No. Keyed records are still hashed, and the same key always maps to the same partition as long as the number of partitions stays the same.
No, they are two unrelated features that share a word. The Sticky Partitioner lives in the producer and decides where new records are written. The sticky assignor lives in the consumer group and tries to keep partition assignments stable when consumers join or leave.
Over a long enough time and with enough producers, yes, they will be close to equal. In very short windows, one partition may receive more because of the batch-by-batch switching.
Because fewer, larger requests put less pressure on the sender and the broker, and records spend less time queued behind many small requests. The saved queueing time usually outweighs the short time spent filling a batch.
Often yes. A small linger of a few milliseconds usually makes batches fuller, which adds to the benefit. Test it against your latency requirements.
No. Partition choice has nothing to do with durability. Message safety depends on acknowledgements, replication and retry settings.
Then there is nothing to choose, and every record goes to that one partition. Batching still works normally.
Keyless records will simply start using the new partitions too. Keyed records may map to different partitions after the count changes, which is a separate concern that affects ordering for those keys.
Only when you have a clear business rule for placement, such as sending records of a premium customer to a dedicated partition. For plain load spreading, the built-in behaviour is well tested and usually better.
21Summary and Key Takeaways
The Sticky Partitioner is a simple idea with a big effect. Instead of dealing keyless records one by one to every partition, the producer keeps filling a single partition’s batch until it is done and only then moves to another partition. Over time every partition receives about the same share, but the producer sends fewer, larger, better-compressed requests. It arrived in Kafka 2.4 through KIP-480 and was refined in later versions with byte-based switching and adaptive weighting. Using it correctly mostly means understanding when to leave the key empty and how to tune batching around it.
Key Takeaways
- It only affects keyless records — records with a key still follow the “same key, same partition” rule.
- It sticks per batch — the partitioner keeps one partition until the batch is full or its waiting time ends, then picks another.
- Bigger batches are the goal — fewer requests, better compression and lower average latency come from fuller batches.
- Balance is a long-term property — a single second may look uneven, but minutes of traffic from many producers look even.
- Tune the partner settings — a small linger, a suitable batch size and compression multiply the benefit.
- It is not an ordering or durability tool — use keys for order and replication settings for safety.
- Do not confuse it with the consumer’s sticky assignor — different feature, different side of Kafka.