Custom Partitioners in Apache Kafka
A friendly, from-the-ground-up guide to how Kafka decides which partition a message lands in, and how you can take control of that decision with a custom partitioner.
Imagine a large hospital with many reception desks. Every patient who walks in must be sent to exactly one desk, and the hospital wants two things at once: patients should be spread out fairly so no desk becomes a bottleneck, and patients with the same file number should always go to the same desk so their records stay together. The person standing at the door making that decision, patient by patient, is doing exactly what a Kafka partitioner does for every single message a producer sends.
This post explains what a partitioner is, why Kafka needs one at all, how the built-in partitioner behaves, and how to write your own custom partitioner when the built-in behavior does not fit your needs. Every idea is explained from scratch with plain language and everyday comparisons, so that by the end you can recognize, in your own systems, the exact moment when the default behavior stops being enough.
1Setting the Stage: Topics, Partitions, and Producers
Before partitioning makes sense, we need a shared picture of what a topic and a partition actually are.
Apache Kafka is a system for moving streams of messages between applications. Applications that write messages into Kafka are called producers, and applications that read messages out of Kafka are called consumers. Messages travel through named channels called topics. Behind the scenes, Kafka does not store a topic as one giant file. Instead, it slices a topic into smaller, independently stored pieces called partitions, and spreads those partitions across different machines called brokers.
Think of a topic as a large library and each partition as one shelf in that library. Splitting the collection across many shelves lets many librarians work at once, one per shelf, instead of everyone crowding around a single shelf. A partitioner is the rule that decides which shelf a new book goes on.
Key terms for this post
Topic
A named stream of messages, such as orders or clickstream-events, that producers write to and consumers read from.
Partition
One ordered, append-only slice of a topic. Messages inside one partition always keep a strict order.
Producer record
The object a producer builds before sending: it carries a topic name, an optional key, a value, and optional headers.
Key
An optional piece of data attached to a message, often an identifier such as a customer id or a device id.
Partitioner
The piece of logic inside the producer that decides which partition number a given record should go to.
Consumer group
A team of consumers sharing the work of reading a topic. Each partition is read by exactly one member of the group at a time.
Why does the partition choice matter so much? Because a partition is the smallest unit Kafka can assign to a reader. If all messages went to one partition, only one consumer in a group could ever read them, no matter how many consumers you add. Spreading messages across partitions is what makes Kafka scale. But spreading them randomly is not always correct either, which is exactly the tension this post explores.
It helps to notice that this tension is not unique to Kafka. Any system that splits work across many workers eventually has to answer the same question: how do we divide the work so that it is both fast and correct? Kafka’s answer is the partitioner, a single, well-defined point in the producer where that division is decided, record by record, for every message the application ever sends.
2What Is a Partitioner?
A partitioner is a small, focused decision-maker that runs once for every message a producer sends.
A partitioner is a piece of logic, living inside the Kafka producer client, whose only job is to answer one question for every outgoing record: which partition number should this message go to? It looks at the record, which may include a topic name, a key, a value, and headers, and returns a single integer, the partition number, before the record is placed into a batch and sent to a broker.
sequenceDiagram
participant App as Your Application
participant P as Producer Client
participant PT as Partitioner
participant B as Kafka Broker
App->>P: send record with topic, key, value
P->>PT: Which partition should this go to?
PT-->>P: Partition number
P->>P: Add record to that partition's batch
P->>B: Send batch to the broker that leads the partition
A partitioner is like the sorting clerk at a mail depot. Every parcel that arrives gets glanced at once, and the clerk decides which delivery van it goes onto. The clerk never opens the parcel to read the full letter inside; a quick look at the address label, which is like the key, is usually enough.
Every official Kafka producer client, whatever programming language it is written in, includes a default partitioner and allows that partitioner to be replaced. Replacing it is called writing a custom partitioner, and that is the main subject of this post.
The partitioner only runs on the producer side. Consumers never choose partitions; they simply read whatever partitions they have been assigned by the consumer group.
It is also worth noting that the partitioner decision happens entirely in memory, before any network call is made. This is intentional: by the time a batch of records leaves the producer for the network, the destination partition, and therefore the destination broker, is already fixed. Nothing about network conditions or broker health changes this choice after the fact, which is why the correctness of your partitioning rule matters so much before you ever go live.
3Why Partitioning Exists at All
Partitioning solves two problems at once: spreading out work, and keeping related messages together.
Kafka splits topics into partitions for scalability. A topic with one partition can only ever be processed as fast as one machine, one disk, and one consumer can manage. A topic with twelve partitions can be spread across up to twelve brokers for storage and up to twelve consumers in a group for reading, multiplying the throughput.
But scaling out creates a new problem: order. Kafka guarantees strict ordering only within a single partition, never across an entire topic. If two messages about the same customer land in two different partitions, a consumer reading partition one has no promise about when it will see the message that landed in partition two. For many use cases, this does not matter. For others, it matters enormously.
When only speed matters
Anonymous page-view logging, general telemetry, and other message types where any single message is independent of the others can be spread across partitions purely for balance.
When order matters
Bank account transactions, order status updates, or a device’s sequence of sensor readings must be processed in the order they happened, so they must always land in the same partition.
This is precisely the job a partitioner does: it decides, for every message, whether it should be spread out for balance or grouped together for order, based on the rules you give it.
4The Default Partitioner, Step by Step
Before building something custom, it helps to know exactly what you get for free.
Kafka’s default partitioner follows a small set of rules, and understanding them precisely is the foundation for knowing when you actually need something different.
Explicit partition given
If your code directly specifies a partition number on the record, the partitioner is skipped entirely and that exact partition is used.
Key is present
If a key is given and no partition is given, the default partitioner runs the key’s bytes through a hashing function and maps the result onto one of the available partitions. The same key always maps to the same partition, as long as the partition count does not change.
No key is present
If there is no key, modern Kafka producer clients use a strategy called sticky partitioning, described in the next chapter, which batches records efficiently while still spreading load roughly evenly over time.
flowchart TD
A["New record to send"] --> B{"Partition explicitly set?"}
B -- "Yes" --> C["Use that exact partition"]
B -- "No" --> D{"Key present?"}
D -- "Yes" --> E["Hash the key and map to a partition"]
D -- "No" --> F["Use sticky partitioning to fill one batch at a time"]
Hashing a key is like using the first letter of a visitor’s last name to send them to one of several queues at a government office. Every time “Sharma” walks in, the same rule sends them to the same queue, because the letter never changes. Nobody actually reads the whole name; the rule only needs a small, repeatable signal.
The mapping from key to partition depends on the current number of partitions. If you add partitions to a topic later, the same key can start mapping to a different partition number, which can break assumptions about strict ordering for that key across the change.
A closer look at the hashing step
The default hashing function most Kafka clients use is a fast, well-tested algorithm designed for exactly this purpose: turning an arbitrary sequence of bytes into a number that looks evenly spread out, without needing any cryptographic strength. The producer takes the key’s raw bytes, runs them through this function to get a large number, and then reduces that number down to a valid partition index, typically by taking the remainder after dividing by the number of partitions.
Two properties of this step matter a great deal in practice. First, the function is deterministic: the same input bytes always produce the same output number, on any machine, at any time, which is exactly why the same key reliably lands on the same partition. Second, the function has no notion of meaning; it does not know that “customer-1042” and “customer-1043” are related ids, so two very similar keys can easily land on completely different partitions. This is a feature, not a bug, for spreading load, but it is also precisely why business-aware grouping needs a different tool, which is where a custom partitioner comes in.
5Sticky Partitioning for Keyless Records
When there is no key to hash, modern Kafka clients use a smarter trick than plain round robin.
Older versions of the default partitioner used simple round robin for keyless records, sending message one to partition zero, message two to partition one, and so on. This spread load evenly but created many small, half-full batches, because each message often went to a different partition than the one before it, and Kafka groups messages heading to the same partition into a single network batch for efficiency.
Sticky partitioning fixes this. Instead of switching partitions on every single record, the producer sticks with one partition for keyless records until the current batch either fills up or a short time passes, and only then switches to a different partition for the next batch. This keeps batches full and efficient while still spreading records across partitions fairly evenly over time.
Sticky partitioning is like a delivery van that keeps loading parcels for one neighborhood until it is full, rather than driving to a different neighborhood after every single parcel. The van still visits every neighborhood fairly over the course of a day, but each trip is far more efficient.
This behavior only applies when there is no key. As soon as a key is present, hashing takes over and sticky partitioning does not apply, because the key already decides the partition.
6Where the Default Partitioner Falls Short
The default rules are good general-purpose choices, but several real situations push past what they can do.
Hashing a key is a purely mechanical operation. It has no idea what your business considers fair, important, or related. This mismatch is exactly where a custom partitioner earns its place.
Hot keys
A handful of extremely active keys, such as one very large customer, can flood a single partition while others stay quiet, because hashing does not know one key is busier than another.
No business-aware grouping
You may want messages grouped by region, by priority level, or by a compound rule that combines several fields, none of which plain key hashing understands.
Reserved or priority partitions
Some systems want to set aside certain partitions for urgent messages so a dedicated, faster consumer can serve them first.
Migration and compatibility
Moving from another messaging system that partitioned differently, or keeping compatibility with an existing downstream system, may require replicating its exact partitioning rule.
Partition count changes
Because hashing depends on the current partition count, growing a topic can silently redirect a key’s future messages to a new partition, breaking the assumption that a key always lives on one partition.
Many beginners believe that hashing a key spreads load perfectly evenly. It only spreads evenly if keys themselves occur with roughly equal frequency. A small number of very frequent keys can still create a very uneven, or “hot,” partition even with a mathematically sound hash function.
This is the moment to reach for a custom partitioner: when the shape of your traffic or your business rules do not match what mechanical hashing can express.
7What Exactly Is a Custom Partitioner?
A custom partitioner is simply your own answer to the same one question the default partitioner already answers.
A custom partitioner is a small piece of logic you write and plug into the producer, which replaces the built-in partitioner. It receives the same information the default one receives, such as the topic name, the key, the value, and the current list of partitions, and it must return one partition number. Kafka’s producer clients are intentionally built so that this piece is swappable, through configuration, without changing anything else about how the producer works.
flowchart LR
A["Record: topic, key, value, headers"] --> B["Custom Partitioner"]
C["Current partition list"] --> B
D["Cluster metadata"] --> B
B --> E["One chosen partition number"]
Think of the default partitioner as the standard sorting rule printed on every parcel machine at a shipping company. A custom partitioner is like a special rule sheet a particular warehouse manager writes and slots into the machine, because their warehouse has needs the standard rule was never designed to cover.
What information is typically available
| Input | What it tells your logic |
|---|---|
| Topic name | Useful if the same partitioner class is reused across several topics with different rules. |
| Key and key bytes | The main signal most custom partitioners key their decision on, such as a customer id or region code. |
| Value and value bytes | Occasionally used when the routing rule depends on the message content itself, such as a priority field inside the payload. |
| Cluster metadata | The current list of partitions and which brokers lead them, useful for excluding unavailable partitions or counting the total. |
The output is always the same: a single whole number identifying the destination partition, which must be a valid partition number for that topic at the moment the record is sent.
It is worth stressing how small this contract actually is. Writing a custom partitioner does not require touching how records are serialized, how batches are sent over the network, how retries work, or how brokers replicate data. All of that machinery stays exactly the same. You are only ever replacing one narrow decision, which is what makes custom partitioners approachable even for teams that are otherwise unfamiliar with the deeper internals of the producer client.
8The Internal Working of a Custom Partitioner
Understanding the moment-by-moment mechanics removes the mystery from how a custom rule actually takes effect.
When you configure a producer to use a custom partitioner, the client library loads your logic once, when the producer starts up, and keeps that single instance alive for the producer’s whole lifetime. From then on, every single call to send a record consults that same instance.
Producer starts
The producer reads its configuration, sees that a custom partitioner class is named, and creates one instance of it. Any setup logic in your partitioner runs once here.
Application calls send
Your application code builds a record with a topic, an optional key, and a value, and calls the producer’s send function.
Partitioner is consulted
The producer hands the record and the current cluster metadata to your partitioner instance and asks for a partition number.
Record joins a batch
The chosen partition number decides which in-memory batch the record is appended to, alongside other records heading to the same partition.
Batch is sent
When a batch is full or a short linger time passes, it is sent over the network to the broker that currently leads that partition.
Producer shuts down
When the application closes the producer, the client gives your partitioner a chance to release any resources it opened during setup.
Because one partitioner instance is shared across every send call from that producer, and producers are often used from multiple threads at once, your partitioning logic must be safe to call concurrently. Avoid changing shared internal state without proper safeguards.
This lifecycle is the same shape across Kafka’s different official client libraries: a one-time setup, a repeated decision function, and a one-time teardown. Only the exact method names and language syntax differ.
9Common Custom Partitioning Strategies
Most real-world custom partitioners fall into a handful of recognizable patterns.
Field-based partitioning
Instead of hashing the whole key, the partitioner looks only at part of the message, such as a region code inside the value, and hashes just that part. This groups messages by a business concept even if the full key varies.
Priority or reserved partitions
A small set of partitions is set aside for high-priority messages. The partitioner checks a priority flag and routes urgent messages to those reserved partitions, which a dedicated, faster consumer watches closely.
Hot key splitting
For a small number of known extremely busy keys, the partitioner spreads that one key’s messages across several partitions instead of one, using a secondary counter, while every other key keeps the normal single-partition behavior.
Geographic or tenant-based routing
In a multi-region or multi-tenant system, the partitioner maps each tenant or region to a fixed band of partitions, keeping one tenant’s data physically grouped for easier operational management.
Compatibility partitioning
When replacing an older messaging system, the partitioner reproduces that system’s exact partition-assignment formula so that downstream consumers built for the old ordering continue to see the same grouping.
Time-window or shard-aware partitioning
Some systems combine a natural key with a coarse time bucket, such as the current hour, so that a downstream batch job can process one time window’s data from a small, predictable set of partitions rather than scanning every partition of the topic.
A closer look: hot key splitting
Suppose nine hundred customers each send a modest number of events, but one enormous customer sends as many events as all the others combined. Hashing the customer id sends that one customer’s entire stream into a single partition, which becomes hot while eight other partitions stay comparatively idle.
flowchart TD
A["Incoming record"] --> B{"Is this a known hot key?"}
B -- "No" --> C["Hash the key normally across all partitions"]
B -- "Yes" --> D["Add a small rotating suffix to the key"]
D --> E["Hash the suffixed key across a reserved band of partitions"]
Splitting a hot key across several partitions improves balance, but it means messages for that one key are no longer in a single strict order across the whole key. Only use this pattern when the business logic can tolerate that key’s events being processed slightly out of order relative to each other.
10Data Flow With a Custom Partitioner in Place
Seeing the whole journey from application code to a consumer’s screen ties every earlier idea together.
Once a custom partitioner is configured, it becomes an invisible but decisive checkpoint on every message’s journey. Here is the complete path a single record travels, from creation to being read.
Application builds the record
Business logic decides the topic, the key, such as a customer id, and the value, such as an order payload.
Custom partitioner assigns a partition
Your rule inspects whatever fields it needs and returns a partition number, following the business logic you designed rather than plain hashing.
Broker stores the record
The record is appended to the log file for that partition on the broker that currently leads it, and replicated to follower brokers for durability.
Consumer group is assigned the partition
One member of the reading consumer group owns this partition and will read every record in it, in the exact order they were written.
Application logic sees consistent grouping
Because your partitioner sent related messages to the same partition, the consumer naturally sees them in the correct relative order, without any extra coordination.
A custom partitioner is like a court clerk who assigns every new case filing to a specific judge based on the type of case, rather than by pure lottery. Because the same type of case always reaches the same judge, that judge naturally builds a consistent, ordered history of similar cases, without any judges needing to talk to each other.
11Advantages and Disadvantages
A custom partitioner is a powerful tool, and like most powerful tools it comes with real trade-offs.
Advantages
- Lets business rules, not just raw hashing, decide message grouping.
- Can fix uneven load caused by a small number of very active keys.
- Supports priority routing to reserved, closely watched partitions.
- Enables compatibility with an existing system’s partitioning scheme during a migration.
- Keeps related data grouped for simpler operational reasoning, such as per-tenant partitions.
Disadvantages and Trade-offs
- Adds custom code that must be tested, versioned, and kept in sync across every producer that uses the topic.
- A poorly written rule can create worse imbalance than the default hashing it replaced.
- Splitting hot keys sacrifices strict per-key ordering across the split partitions.
- Any logic that reads cluster metadata must handle partition counts changing over time.
- Debugging skewed load is harder, because the routing decision lives in custom code rather than a single well-known formula.
Reach for a custom partitioner only after confirming that the default partitioner’s key-hash behavior genuinely cannot express what you need. Many apparent partitioning problems are actually solved more simply by choosing a better key, rather than by writing custom routing logic.
It also helps to separate short-term pain from a long-term architectural need. A single unusually large customer causing temporary load might be better solved with a short-lived operational workaround, while a genuine, recurring pattern of uneven traffic across your whole business justifies investing in a properly designed and tested custom partitioner that will live in your codebase for years.
12Design Patterns and Anti-Patterns
A few habits separate a partitioner that quietly works for years from one that causes a painful outage.
Patterns worth adopting
Deterministic logic
Given the same inputs and the same partition count, the partitioner always returns the same partition, with no reliance on random values or wall-clock time.
Graceful fallback
When the special routing rule does not apply, such as a missing field, fall back to standard key hashing rather than failing the send.
Bounds checking
Always compute the returned partition number using the current partition count from cluster metadata, never a hardcoded number.
Lightweight decisions
Keep the logic fast, since it runs on every single record and can become a bottleneck if it does expensive work like network calls.
Habits to avoid
Problem
A custom partitioner hardcodes the total number of partitions instead of reading it from cluster metadata at send time.
Why It’s Harmful
The moment the topic’s partition count changes, the hardcoded logic can return an invalid partition number, causing sends to fail, or it silently stops using newly added partitions.
Correct Approach
Always read the current partition count from the metadata the client provides at the moment of sending, and compute the destination relative to that live count.
Problem
The partitioner uses random numbers or the current time to help decide the partition for a keyed record.
Why It’s Harmful
The same logical key now scatters across different partitions on different sends, destroying the per-key ordering guarantee that most keyed traffic depends on.
Correct Approach
Base the decision purely on the record’s own content, such as the key, so the same key reliably lands in the same partition every time.
Problem
Different producers for the same topic run different versions of the custom partitioner, perhaps because a deployment rolled out only halfway.
Why It’s Harmful
The same key can be routed to different partitions depending on which producer instance sent it, quietly breaking ordering guarantees consumers rely on.
Correct Approach
Version and deploy the partitioner class alongside careful rollout practices, and treat a change to partitioning logic as a breaking change requiring coordination.
13Monitoring and Operating a Custom Partitioner
Once a custom partitioner is live, a small set of signals tells you whether it is actually doing its job.
Per-partition byte and record rate
Broker-side metrics showing bytes and records written per partition reveal at a glance whether load is balanced or concentrated on a few partitions.
Per-partition consumer lag
If one partition consistently lags behind the others, its consumer may be struggling with a disproportionate share of work sent there by the partitioner.
Partition size on disk
Comparing log segment sizes across partitions over time is a simple, durable way to catch long-term skew that short-term rate metrics might miss.
Producer send latency and errors
A custom partitioner that does slow or fragile work, such as an external lookup, can show up as increased send latency or a rise in send failures.
Log a sample of key-to-partition mappings when the partitioner first starts, and periodically afterward. A quick sanity check during rollout catches an off-by-one or a reversed comparison long before it becomes a production incident.
A simple health check
Regularly compare the record count across all partitions of a topic. A perfectly even topic will show similar counts on every partition. A large and growing gap between the busiest and quietest partition is the clearest sign that either the key distribution itself is uneven, or the custom partitioner is not behaving as intended.
14Real-World and Industry Examples
Different industries lean on custom partitioning for different, very concrete reasons.
Ride-hailing and delivery platforms
Location update events are often partitioned by a geographic cell or city id rather than by raw driver id, so that all activity in one city stays grouped together for regional dashboards and regional-specific processing.
Financial trading systems
Order events for a single financial instrument are partitioned by instrument identifier, guaranteeing that every update to that instrument’s order book is processed in strict sequence by one consumer.
Multi-tenant software-as-a-service platforms
Events are partitioned by tenant identifier, sometimes with large tenants deliberately spread across a reserved band of partitions, so no single very active customer can starve smaller customers of consumer capacity.
Internet-of-things and sensor networks
Readings are partitioned by device id so that a single device’s time series of measurements is always processed in the order it was generated, which matters for anomaly detection and state tracking.
Retail and e-commerce order pipelines
Order lifecycle events, such as placed, paid, packed, and shipped, are partitioned by order id, so a single order’s state machine always advances in the correct sequence on one consumer.
Online gaming platforms
Player action events within a single game match are partitioned by match id, ensuring the sequence of moves and outcomes for that match is processed in exactly the order the players experienced it, even while thousands of other matches are being processed in parallel across the rest of the topic.
15Common Mistakes and Troubleshooting
A quick symptom table for the moment something about partition balance or ordering looks wrong.
| Symptom | Likely cause | What to try |
|---|---|---|
| One partition grows much faster than the others | A hot key, or a custom rule that unintentionally favors one partition | Check key frequency distribution and review the partitioner’s mapping logic |
| Messages for the same key appear out of order downstream | The topic’s partition count changed, shifting where the key now hashes, or the key itself is inconsistent | Confirm the key is stable and consider whether the partition count should stay fixed for ordering-sensitive topics |
| Sends occasionally fail with an invalid partition error | The custom partitioner computed a partition number outside the valid range | Always derive the range from live cluster metadata rather than a fixed number |
| Producer throughput drops after adding a custom partitioner | The partitioner performs slow work, such as a network or database call, on every send | Move slow lookups to a cache built once at startup, and keep the per-record logic fast |
| Different environments show different partition assignments for the same key | Different producer instances are running different versions of the partitioner class | Standardize the deployed version and coordinate rollout across all producers |
| Consumer lag is uneven despite similar partition sizes | Message complexity, not just count, varies by partition | Check whether certain keys route to records that are simply more expensive to process |
Assuming that because the default partitioner is “hash based,” it must already be perfectly fair. Fairness depends on how evenly the key values themselves occur in your traffic, not just on the hashing function.
16Frequently Asked Questions
Short, direct answers to the questions people ask most often when they first meet custom partitioners.
It is the logic in a Kafka producer that decides which partition of a topic each outgoing message should be sent to.
No. The default partitioner, which hashes the key or uses sticky partitioning without a key, is a good fit for the large majority of use cases.
No. Partitioning only happens on the producer side, when a message is sent. Consumers simply read whatever partitions they are assigned.
Yes, and this is normal. Many keys typically share each partition; the guarantee is only that the same key always lands in the same partition, not that each partition holds a single key.
Any logic based on the current partition count will see the new, larger count and may compute different destinations than before, so keys can move to different partitions going forward. Existing messages already written are never moved.
Only if the underlying keys occur with genuinely equal frequency, or if the partitioner deliberately splits busy keys across several partitions. No partitioner can make naturally uneven traffic perfectly even without changing what “same key, same partition” means for the busiest keys.
The concept is simple: one function that receives a record and returns a partition number. The difficulty usually lies in choosing the right business rule and handling edge cases like missing fields or changing partition counts carefully.
Yes. If a record already carries an explicit partition number, the partitioner, whether default or custom, is never consulted for that record.
Yes, strongly. If different producers for the same topic use different partitioning logic, the same key can inconsistently land in different partitions, undermining ordering guarantees.
They solve the same kind of problem, deciding a destination partition from a record, but they apply in different places. A custom partitioner configures a plain producer, while a StreamPartitioner is used specifically inside Kafka Streams applications when data is repartitioned internally.
Yes, in most client libraries the partitioner has access to headers as well, which is useful when you want to pass a routing hint separately from the key that consumers and compaction otherwise rely on.
Check the frequency distribution of the keys themselves before assuming the partitioner is broken. Uneven key frequency, not faulty hashing or faulty custom logic, is the most common root cause of an unbalanced topic.
17Security and Robustness Considerations
A custom partitioner runs deep inside the producer’s send path, so it deserves the same defensive care you would give any code that touches every outgoing message.
A custom partitioner is trusted code: it runs inside your own producer process, with the same permissions as the rest of your application, once per outgoing record. It does not cross a network boundary and does not, by itself, expose or protect any data. Even so, several practical robustness and security-adjacent habits matter a great deal in production.
Never let it throw uncaught exceptions
An unexpected error inside the partitioning logic, such as a missing field on an unusual message, should be caught and handled with a sensible fallback rather than crashing the send call or the whole producer.
Avoid leaking sensitive data into logs
If the partitioner logs sample keys for debugging, make sure those keys are not sensitive identifiers such as full account numbers, and mask or hash them in log output.
Guard against malformed or hostile input
If any part of the routing decision is derived from data that ultimately originates outside your trust boundary, such as a field from an external upstream system, validate it defensively rather than assuming it always looks the way you expect.
Keep external calls out of the hot path
A partitioner that calls out to another service or database on every message introduces a new failure mode and a new latency dependency directly inside the critical path of every single send.
Treat the partitioner the way you would treat any function that runs on the hottest path of your system: fast, defensive, and boring in the best sense of the word. Excitement in partitioning logic usually means something is about to go wrong at scale.
18Custom Partitioners and Their Neighboring Mechanisms
A custom partitioner is not the only way to influence how messages are grouped and routed. Knowing its neighbors helps you pick the right tool.
Kafka offers several mechanisms that can shape how messages are grouped or routed, and it is easy to reach for a custom partitioner when a simpler neighboring tool would do the job better. Understanding how they differ prevents both over-engineering and under-engineering the solution.
| Mechanism | What it controls | When it fits better than a custom partitioner |
|---|---|---|
| Choosing a better message key | What value gets hashed into a partition | When the only problem is that the current key is not the right grouping identifier, such as using an order id instead of a random request id |
| Explicit partition on the record | Bypasses the partitioner entirely for that one record | When a small number of special messages need a fixed, hardcoded destination, such as a control message always sent to partition zero |
| Message headers | Extra metadata carried alongside a message, separate from the key and value | When you want to pass routing hints to a custom partitioner without disturbing the key used for compaction or logging |
| Kafka Streams StreamPartitioner | Partitioning specifically for records written by a Kafka Streams application | When your routing logic is part of a Streams topology rather than a plain producer application |
| Increasing partition count | The overall degree of parallelism available | When the real problem is simply too few partitions for the desired consumer parallelism, not an unfair distribution rule |
| Custom partitioner | The actual mapping logic from a record to a partition number | When the grouping rule itself needs business-specific logic that no combination of the above can express |
Choosing a better key is like relabeling parcels so the address itself already points to the right neighborhood. An explicit partition is like a special sticker that sends one specific parcel straight to a particular van, no questions asked. A custom partitioner is like hiring a smarter sorting clerk who applies an entirely new set of sorting rules to every parcel that passes through.
Kafka Streams deserves a special mention here, because it is one of the most common places developers actually encounter this decision. A Streams application that repartitions data internally, such as after a grouping or joining operation, allows you to supply a custom partitioning interface of its own, conceptually similar to a producer’s custom partitioner but scoped to how Streams redistributes records between its own internal processing steps. The underlying idea, deciding a destination partition from a record’s content, is the same lesson taught throughout this post, just applied inside a different part of the Kafka ecosystem.
Before writing a custom partitioner, ask whether simply changing what field you use as the key would solve the problem. A surprising number of “we need a custom partitioner” conversations end with “we just needed a better key” instead.
19Testing and Rolling Out a Custom Partitioner Safely
Because a partitioning mistake can be subtle and only show up under real traffic, deliberate testing and a cautious rollout matter more here than for most producer-side code.
Testing strategies
Deterministic unit tests
Feed the same key and partition count into the partitioner many times and confirm it always returns the same partition number.
Distribution simulation
Run a large, realistic sample of historical keys through the logic offline and plot how many keys land on each partition, checking for unexpected concentration.
Edge case coverage
Test missing keys, empty keys, unusually long keys, and the smallest and largest possible partition counts, such as a topic with only one partition.
Partition count change simulation
Run the same set of keys against both the old and new partition count after a hypothetical resize, and review which keys would move to a different partition.
A staged rollout plan
Shadow test in a non-production topic
Send a copy of real traffic to a throwaway topic using the new partitioner and compare its distribution against expectations before touching production data.
Deploy to a small percentage of producer instances
If your deployment tooling allows a gradual rollout, watch per-partition metrics closely as only a fraction of traffic uses the new logic.
Confirm consumer behavior is unaffected
Check that consumer lag, processing correctness, and any per-key ordering assumptions still hold under the new routing rule.
Complete the rollout
Once metrics look healthy, finish deploying the new partitioner to every producer of that topic, since a mixed fleet of old and new logic can scatter a single key across partitions inconsistently.
Rolling a partitioner back after it has already been live is not free. Any key whose destination partition changed under the new logic will change again on rollback, so plan rollbacks the same way you plan the original rollout, with monitoring in place rather than as a quick emergency toggle.
20Summary and Key Takeaways
A partitioner is the small but decisive piece of logic that decides which partition each Kafka message lands in. The default partitioner does a solid job for most traffic by hashing keys and using sticky batching for keyless records, but it cannot express business-specific rules, cannot recognize hot keys on its own, and depends on the current partition count. A custom partitioner steps in exactly where those limits show up, letting you route messages by business meaning, protect against hot keys, prioritize urgent traffic, or stay compatible with another system’s scheme. Built thoughtfully, deterministically, and with attention to changing partition counts, a custom partitioner turns raw message spraying into deliberate, ordered, and well-balanced data flow.
Key Takeaways
- A partitioner answers one question — given a record, which partition number should carry it, decided once per send on the producer side only.
- The default partitioner is hash-plus-sticky — keyed records are hashed to a partition, and keyless records use sticky batching for efficient, roughly even spreading.
- Ordering lives inside one partition — Kafka never guarantees order across an entire topic, only within each individual partition.
- Custom partitioners solve business-shaped problems — hot keys, priority routing, tenant isolation, and compatibility with legacy partitioning schemes.
- Determinism is essential — the same key, value, and partition count should always produce the same partition, with no randomness or reliance on time.
- Watch live metadata, not fixed numbers — always compute against the current partition count so growing a topic never produces an invalid partition.
- Monitor per-partition balance continuously — record counts, byte rates, and consumer lag per partition reveal whether a partitioning rule is truly working as designed.