Before we can talk about uneven partitions, we need a shared picture of what a partition is and why Kafka has them.

Apache Kafka is a system for moving streams of events between applications. An event is just a small record of something that happened: a customer placed an order, a sensor sent a temperature reading, a user clicked a button. Kafka stores these events durably and lets many other applications read them, either right away or much later.

Why does a topic need partitions at all?

A single log living on a single server can only be written to and read from so fast. One disk, one network card, and one CPU set a hard ceiling. Kafka removes that ceiling by cutting a topic into partitions and spreading those partitions across many brokers. Now writes can flow into many logs at once, and reads can be shared among many consumers at once.

This is the heart of Kafka’s scalability: the partition is the unit of parallelism. Everything Kafka can do in parallel, it does per partition. That single sentence also explains why partition imbalance hurts so much, as we will see.

Leaders and followers

Each partition has copies called replicas, stored on different brokers for safety. One replica is the leader, and all the others are followers. Producers write only to the leader, and consumers normally read only from the leader. Followers quietly copy whatever the leader receives. So the broker that holds the leader of a partition does the heavy lifting for that partition. Remember this, because it means a hot partition puts a hot spot on one specific broker.

Skew is born at the moment the producer decides where each message goes.

When a producer sends an event, someone has to decide which partition receives it. That decision is made by a component called the partitioner, which lives inside the producer client. There are three common situations.

Why keys exist

Kafka guarantees that events inside one partition stay in the exact order they were written. It makes no such promise across partitions. If you want all events for customer 42 to be processed in order, you give them the same key, “42”, so they all travel to the same partition. That is a very useful guarantee, and it is also the root of most skew problems. By choosing to keep related events together, you accept that the size of the biggest group decides the size of the hottest partition.

The two names are close cousins. Let us define each one carefully.

Skew is about the shape of the data. A hot partition is about the pressure that shape creates. In practice, people use the words almost interchangeably, because a heavily skewed topic almost always contains at least one hot partition. Still, the difference is worth remembering: it is possible to have a hot partition without much stored data (for example, a partition with short retention that receives a torrent of messages), and it is possible to have a large partition that is not hot (an old, quiet partition that simply accumulated data over years).

Those three “ones” are the reason a single overloaded partition cannot be rescued by adding more machines. You cannot spread one partition across two consumers in the same group, and you cannot spread its leader duty across two brokers. The slice is indivisible, so when it is overloaded, it is simply overloaded.

flowchart LR
  P[Producers] --> T{Topic with 6 partitions}
  T -->|5 percent| P0[Partition 0]
  T -->|4 percent| P1[Partition 1]
  T -->|60 percent| P2[Partition 2 HOT]
  T -->|6 percent| P3[Partition 3]
  T -->|15 percent| P4[Partition 4]
  T -->|10 percent| P5[Partition 5]
  P2 --> C2[Consumer C2 overloaded]
        

In the figure, six partitions exist, but partition 2 receives sixty percent of all events. Its consumer must process six times more work than an average consumer, its broker must write six times more data than an average leader, and its follower replicas must copy all of it. The whole topic can only go as fast as its slowest, busiest partition.

Imbalance is not just an aesthetic problem. It damages speed, stability, and cost.

Wasted parallelism

Suppose you designed a topic with twenty partitions and twenty consumers so that each consumer would handle five percent of the load. If one partition secretly carries forty percent of it, the consumer assigned to it needs eight times the capacity of its neighbours. The other nineteen consumers finish quickly and sit idle. You pay for twenty workers but effectively receive the throughput of two or three.

Growing lag

Consumer lag is the gap between the newest message in a partition and the last message a consumer has processed. On a hot partition, messages arrive faster than the consumer can handle them, so the gap widens. Users see this as stale dashboards, delayed notifications, and late payments, even though the cluster’s average numbers look perfectly fine.

Broker stress and uneven disks

The broker that leads a hot partition uses more network bandwidth, more disk write speed, and more CPU for compression, checks, and replication. Meanwhile, the disks under a heavily skewed partition fill up faster. A single broker running out of disk space can take partitions offline, and that can turn a “performance problem” into an outage.

Cascading effects on replication

Followers must keep up with the leader. If the leader is receiving a flood, followers may fall behind and drop out of the in-sync replica set. When that happens, producers that require acknowledgement from all in-sync replicas may slow down or see errors, and the topic becomes less durable at exactly the moment it is busiest.

Skew almost never happens by accident. It has a small number of well-known causes.

The heavy-hitter problem in detail

Real-world traffic almost never follows a flat, even pattern. It usually follows what statisticians call a long-tail or power-law shape: a small number of items produce a huge share of activity. A handful of large customers generate most orders, a handful of popular videos generate most views, and a handful of busy devices generate most sensor readings. If you use the customer, video, or device as the key, then the partition that receives the top key will be far busier than the rest, no matter how good your hash function is.

The low-cardinality trap

Cardinality means the number of distinct values a field can have. If you key events by country and you serve twenty countries, you can use at most twenty partitions productively, and if two countries happen to hash to the same partition, that one gets double load. Keying by a boolean or a status field like “success” or “failed” is even worse, because there are only two values in total, so at most two partitions will ever receive data.

The hidden effect of partition count changes

Because the default partitioner takes the hash modulo the number of partitions, adding partitions to a topic changes where existing keys go. A key that lived in partition 3 might now map to partition 11. This does not directly cause skew, but it surprises teams who “fixed” a hot spot by adding partitions and then found that ordering guarantees had quietly broken. We will return to this in the mitigation chapters.

Following one hot partition through the producer, broker, and consumer shows where the pain appears.

sequenceDiagram
  participant App as Producer App
  participant L as Leader Broker of Hot Partition
  participant F as Follower Brokers
  participant C as Consumer of Hot Partition
  App->>L: Flood of writes for one key
  L->>L: Append to log on one disk
  L->>F: Replicate every record
  F-->>L: Fetch requests, may fall behind
  L-->>App: Acknowledgement, slower under load
  C->>L: Fetch records
  L-->>C: Data, but arrives faster than it is processed
  C->>C: Processing falls behind, lag grows
        

On the producer side

Producers collect messages into batches for each partition. A hot partition produces many full batches very quickly, so the producer spends a lot of its effort sending to one broker. If that broker slows down, the producer’s buffer for that partition fills up. Once the buffer is full, calls to send can block or fail, which can slow down the whole application, including the parts that were writing to healthy partitions.

On the broker side

The leader broker of a hot partition receives the writes, appends them to the end of a log file on disk, and serves reads to consumers and follower replicas. Kafka is very efficient at sequential disk writes, but one disk and one network interface still have limits. If several hot partitions happen to have their leaders on the same broker, the situation becomes far worse, because a single machine now carries many hot logs at once.

On the consumer side

Inside a consumer group, each partition is assigned to exactly one consumer. That consumer reads the partition’s messages one after another, in order. When the partition is hot, the consumer has more messages than it can process in time. Adding more consumers to the group does nothing for that partition, because a partition cannot be shared inside the same group. The extra consumers will simply take up other, quieter partitions or sit unused.

The rebalancing problem

Kafka occasionally moves partitions between consumers, for example when one joins or leaves. If a slow consumer of a hot partition is restarted, the partition moves to a new consumer that must catch up on a large backlog, and during the pause other partitions may also stall depending on the assignment strategy. A hot partition therefore makes ordinary maintenance operations riskier.

Not every uneven partition problem is the same. Naming the type helps you pick the right fix.

These types often appear together. Data skew creates a traffic hot spot, which creates processing skew, which shows up as lag. But they can also appear separately. If your partitions hold exactly the same amount of data, you can still have a slow consumer, and if your consumers are all healthy, you can still have one broker leading too many partitions. Diagnosing which type you have is the first job of anyone facing a “Kafka is slow” alarm.

You cannot fix what you cannot see. These are the signals that reveal imbalance early.

Tools people commonly use

These situations, drawn from common industry patterns, show how skew appears in practice.

Notice the common thread: in every case, the key was chosen for a good business reason, and the skew was a side effect of how unevenly the real world behaves. This is why partition key design is one of the most important decisions in any Kafka system.

The cheapest and most effective fixes usually happen before the data enters Kafka.

Strategy 1: Choose a higher-cardinality key

If you key by country and suffer from skew, consider keying by something finer such as customer identifier or session identifier. More distinct keys mean the hash has more items to spread. The question to ask is: what is the smallest group of events that truly must stay in order together? Use that as the key, and no coarser.

Strategy 2: Use a composite key

Combine two fields, for example the customer identifier and the order identifier. Events for the same order still stay together and in order, but the huge customer’s traffic is now spread across many keys, so it is spread across many partitions. This works whenever the ordering requirement is narrower than the original key suggested.

Strategy 3: Salting the hot keys

Salting means adding a small random or rotating number to a key, so one busy key becomes several keys. If a giant seller’s key is “seller-77”, you might send events as “seller-77-0”, “seller-77-1”, up to “seller-77-9”, picking the suffix at random or in rotation. The traffic for that one seller now flows into up to ten partitions instead of one.

Strategy 4: A custom partitioner for known hot keys

If you know which keys are heavy, you can write a custom partitioner that treats them specially. For example, it may spread the known heavy keys across a reserved group of partitions while letting all normal keys use the standard hashing. This gives you targeted control, but it also puts the maintenance burden on your team. The list of heavy keys changes over time, so it needs to be updated or detected automatically, and every producer application must use exactly the same rules.

Strategy 5: Do not use a key when order does not matter

Many pipelines set keys out of habit, even though no consumer cares about ordering. If order is irrelevant, drop the key. Then modern producers can distribute messages evenly, and they even try to steer traffic away from slow brokers. Removing an unnecessary key is sometimes the simplest cure for skew.

Strategy 6: Fix the fallback keys

Look for default values like “unknown”, “null”, “default”, or empty text. If many events carry these, replace the fallback with a value that varies, such as a random identifier, or send those events without a key.

flowchart TD
  A[Events arrive with key] --> B{Is the key known to be hot?}
  B -->|No| C[Use normal hash partitioning]
  B -->|Yes| D{Is strict ordering needed?}
  D -->|Yes| E[Use composite key with a narrower ordering unit]
  D -->|No| F[Add a salt suffix to spread across partitions]
  C --> G[Balanced partition]
  E --> G
  F --> G
        

More partitions sound like the obvious answer. The truth is more nuanced.

When a topic has too few partitions for its throughput, adding partitions really does help, because the same traffic is now shared among more logs, more brokers, and more consumers. However, there are important cautions.

Choosing the partition count wisely

A common approach is to estimate the target throughput, divide it by what a single partition and a single consumer can comfortably handle, and then add generous headroom for future growth and for the maximum number of consumers you might want. A slightly over-provisioned count is usually safer than a tight one, but going very high has its own costs, so aim for a sensible middle path and revisit it as traffic changes.

Sometimes the data cannot be rebalanced, so the consumer must become smarter.

Make each message cheaper to process

Before adding complexity, profile the consumer. A slow call to a database or a remote service for each message is a very common bottleneck. Batching those calls, caching repeated lookups, and doing slow work asynchronously can multiply the speed of a single consumer, which directly relieves a hot partition.

Process multiple keys in parallel within one partition

A partition delivers messages in order, but the ordering rule usually matters only per key, not across different keys. A consumer can therefore hand messages to a pool of worker threads, ensuring that all messages with the same key go to the same worker in order, while different keys run in parallel. The consumer commits an offset only after all earlier messages have finished. This gives you much more parallelism from one partition without breaking per-key order, though it needs careful offset tracking and error handling.

Use a fair assignment strategy

Consumer groups use an assignment strategy to decide which consumer receives which partitions. Modern cooperative strategies reduce disruption when consumers join or leave, so a hot partition is not needlessly moved around. Also, make sure that the number of consumers is compatible with the number of partitions, so that no consumer accidentally receives two hot partitions while another receives none.

Give hot workloads dedicated capacity

Some teams place particularly heavy tenants or workloads into their own topic, served by a separate consumer group with more resources. This isolates the heavy traffic so it cannot slow the well-behaved traffic. It is a form of the “bulkhead” pattern, named after the sealed compartments in a ship that stop a leak in one section from sinking the whole vessel.

Watch for slow hosts

Occasionally the consumer is slow simply because its machine is overloaded, its network is poor, or a neighbour on the same host is consuming resources. If lag is high on one partition but the data rate is normal, try moving that consumer to a healthier host before redesigning your keys.

flowchart LR
  P[Hot Partition] --> R[Single Consumer Reader]
  R --> W1[Worker 1 handles keys A and D]
  R --> W2[Worker 2 handles keys B and E]
  R --> W3[Worker 3 handles keys C and F]
  W1 --> O[Offset commit after all earlier work is done]
  W2 --> O
  W3 --> O
        

Even when partitions themselves are balanced, the way they are placed on brokers can create a hot machine.

Balance leaders across brokers

Every partition has a preferred leader, which is normally the first replica in its list. After failures and restarts, leadership can drift so that one broker ends up leading far more partitions than others. Running a preferred leader election moves leadership back to the intended replicas and restores an even spread. Kafka can also do this automatically on a schedule.

Move partitions between brokers

Kafka lets administrators reassign partitions, which copies the data to different brokers. If two hot partitions share a broker, moving one of them to a quieter broker spreads the pressure. Reassignment uses network and disk bandwidth while data is copied, so it should be throttled and scheduled with care, ideally outside peak hours.

Use automated balancing

Managing this by hand becomes tiring in large clusters. Tools such as Cruise Control continuously measure the load on each partition and broker, and then compute a plan that balances disk, network, CPU, and leader counts together. Many managed Kafka services also include automatic balancing features. Automation is especially valuable when the hot spots move around over the day.

Spread replicas across racks or zones

Rack awareness places replicas of the same partition in different racks or availability zones. While its main purpose is durability, it also helps prevent all the load of a group of partitions from landing in one physical location.

Consider tiered storage for large partitions

When a partition is large mainly because it retains a lot of history, tiered storage can move older segments to cheaper remote storage, leaving only recent data on the broker’s local disk. This reduces disk pressure from data skew, although it does nothing about a partition that is hot because of live traffic.

No fix is free. Each one trades something for something else.

The wise approach is to start with the least invasive fix that works. Check the key first, then the consumer speed, then the broker placement, and only then reach for more powerful but more complicated techniques like salting and custom partitioners.

A few habits keep skew away, and a few habits invite it.

Helpful patterns

Anti-patterns to avoid

Keeping partitions balanced is an ongoing habit, not a one-time task.

What to track continuously

Build a dashboard that shows, for every important topic, the data rate per partition, the consumer lag per partition, the size on disk per partition, and the leader count per broker. Add broker-level charts for network, disk, CPU, and request handler idle time. Store the history so that you can see whether a hot spot is a one-off burst, a daily pattern, or a slow trend that will worsen over months.

Alerting that helps rather than annoys

Alerts should point to action. Good alerts compare each partition with its siblings, warn when lag keeps growing for a sustained period rather than for a moment, and warn when a broker’s disk is approaching its limit. Avoid alerts that fire on every short burst, because people quickly learn to ignore them.

Security and multi-tenant fairness

In shared clusters, a single client can accidentally or deliberately overload a broker. Kafka’s quotas let administrators cap how many bytes per second a client or user may produce or fetch. Quotas do not fix key skew, but they stop one noisy application from harming everyone else. Access controls also matter: restrict who can create topics and change partition counts, because careless changes to a critical topic can disturb ordering and balance for many teams.

Capacity planning with skew in mind

When estimating capacity, do not size the system for the average partition. Size it for the hottest partition you expect, plus headroom. If the busiest partition needs five times the average throughput, then the consumer and broker responsible for it must be sized accordingly, or the skew must be reduced.

Test with realistic data

Load tests that use perfectly random keys will always look healthy. Test with data that has the same long-tail shape as production, including the biggest customers and the ugliest fallback values. Hot partition problems that would have surprised you in production often appear immediately in a realistic test.

Let us put everything together in a realistic story, from alarm to fix.

Imagine a company that runs an order processing pipeline on Kafka. One afternoon, customers complain that order confirmation emails are arriving twenty minutes late. The team investigates.

This journey shows a general truth: the winning move was to understand the real ordering requirement, not to throw hardware at the problem. Most hot partition incidents end this way.

Skew describes uneven distribution of data across partitions. A hot partition describes uneven load, meaning one partition is receiving or serving far more traffic than the rest. Skew usually causes hot partitions, but they are not exactly the same thing.

Not inside one consumer group. A partition can be read by only one consumer of a group at a time, so extra consumers cannot share a hot partition’s load. You must make that consumer faster, add per-key parallelism inside it, or spread the traffic over more partitions.

No. A single key always hashes to the same partition as long as the partition count stays the same, and after a change it will still land in exactly one partition. To spread a hot key you must change or salt the key, or otherwise split its traffic.

It is much less likely. Keyless messages are spread across partitions by the producer, and modern clients also try to avoid overloading slow brokers. Short-term unevenness can still appear because of batching, but there is no heavy-hitter key to cause a persistent hot spot.

Compare the growth rate of the newest offset per partition with the consumer’s processing rate per partition. If one partition receives far more data, the data is skewed. If data rates are even but one consumer falls behind, the consumer or its host is the problem.

No. Salting spreads a hot key across several partitions but breaks ordering across those partitions and can complicate log compaction. It suits keys for which strict ordering is unnecessary, and it is best applied only to the known hot keys.

Leadership may be unevenly distributed. Since leaders handle all client traffic for their partitions, a broker leading many more partitions than the others will be busier. A preferred leader election or a partition reassignment can restore balance.

There is no single right number. Base it on the target throughput, what one partition and consumer can handle, the maximum number of consumers you want, and growth headroom. Too few limits parallelism and worsens skew, while too many increases overhead and recovery time.

Kafka lets you increase the partition count of a topic but not decrease it. To reduce the count you would create a new topic with fewer partitions and migrate the data and applications.

Not directly. Quotas limit how much a client can produce or consume, which protects the cluster from noisy clients, but they do not change how keys map to partitions. They complement, rather than replace, good key design.

It can. Topics share brokers, disks, and network. If a hot partition’s leader overloads a broker, other topics led from that broker may see higher latency, too.

Design the key deliberately. Understand the real ordering requirement, measure how traffic is distributed across the possible key values, and choose the finest key that still satisfies the ordering you truly need.

Kafka gets its power from splitting topics into partitions, and each partition is a single ordered lane that one leader broker and one consumer per group must handle. That design is wonderfully scalable when work is evenly spread, and painfully limiting when it is not. Skewed partitions hold uneven amounts of data, hot partitions carry uneven amounts of traffic, and both usually begin with the choice of partition key meeting the uneven reality of real-world data.

The good news is that skew is diagnosable and fixable. You can detect it with per-partition lag, size, and rate metrics, and you can cure it by choosing better keys, spreading hot keys, adding partitions carefully, making consumers faster and more parallel, and balancing leaders across brokers. The best outcomes come from understanding what ordering your business truly needs, and then giving Kafka the freedom to spread everything else.