To understand keys, we first need to remember what they choose between.

Topics are split into partitions

Apache Kafka stores streams of events, such as orders, payments, clicks, or sensor readings. Events are grouped into named categories called topics. To make a topic big and fast, Kafka splits it into several smaller logs called partitions. Each partition is an ordered, append-only list of events, and different partitions can live on different servers, called brokers.

Someone must choose the partition

When an application sends an event, Kafka must decide which partition receives it. If this choice were random every time, related events would be scattered, and their order would be lost. If the choice were always the same partition, the whole system would be limited by a single log. The partitioning key is the tool that lets the sender guide this choice in a smart, predictable way.

A partitioning key is an optional piece of data attached to an event that Kafka uses to decide which partition stores it.

The simple definition

Every Kafka event, often called a record, has a key and a value. The value is the main content, such as the details of an order. The key is a short identity, such as a customer number, an order number, or a device name. When the key is present, the producer feeds it into a fixed calculation, and the result points to one specific partition. The same key always gives the same answer, as long as the number of partitions stays the same.

Parts of a record

The trick is a fixed calculation that turns any key into a partition number.

The three small steps

flowchart LR
  K["Key: customer 4711"] --> SR["Serialize to bytes"]
  SR --> H["Hash into a large number"]
  H --> M["Divide by partition count and take the remainder"]
  M --> P["Partition 3"]
        

Why hashing works well

A good hash function spreads different keys roughly evenly across all partitions. If you have thousands of different customers, each partition receives roughly the same number of them. The calculation is also extremely fast, needs no lookup table, and does not require any coordination between producers. Any producer, anywhere, running the same rule, sends the same customer to the same partition.

The reason keys matter most is that they give you ordering exactly where you need it.

Order lives inside one partition

Kafka stores and delivers events of a single partition in the exact order they were written. Across different partitions, there is no promised order, because partitions are read independently and at different speeds. So if two events must be read in order, they must be in the same partition, and the key is how you make that happen.

An example with a bank account

A customer deposits money, then withdraws money, then closes the account. If these three events used no key, they might land in three different partitions and be read in any order, and the bank might try to withdraw from an account that has not yet been credited. With the account number as key, all three events share a partition and are read in the order they were sent.

sequenceDiagram
  participant P as Producer
  participant P1 as Partition 1
  participant P2 as Partition 2
  P->>P1: Account A - Deposit
  P->>P2: Account B - Deposit
  P->>P1: Account A - Withdraw
  P->>P1: Account A - Close
  P->>P2: Account B - Withdraw
  Note over P1: Account A events stay in order
  Note over P2: Account B events stay in order
        

A key is optional. Sending without one is a perfectly good choice in many situations.

The keyless approach

When an event has no key, also called a null key, the producer has no identity to follow, so it simply spreads events across partitions to keep the load balanced. Older clients used to rotate one event at a time. Modern clients use a smarter approach called sticky partitioning: they fill a batch for one partition, send it, and then switch to another partition. This keeps batches large, which is much more efficient than scattering single events.

When to skip the key

If the order of events does not matter and no consumer needs related events together, a null key gives the most even balance and the simplest scaling. Click streams, log lines, and general metrics are common examples.

Picking the key is one of the most important design decisions in a Kafka system. A few simple questions guide it.

Examples of good keys

Examples of poor keys

Keys keep related events together, but sometimes “together” becomes too much of a good thing.

What is a hot key?

A hot key is a key that receives a far larger share of the traffic than the others. Since all events with the same key go to one partition, a hot key makes that partition much busier than its neighbours. The partition’s leader broker works harder, and the single consumer reading that partition falls behind, even though the other consumers in the group may be idle.

How hot keys appear

flowchart LR
  subgraph Even["Evenly spread keys"]
    E0["P0 - 25 percent"]
    E1["P1 - 25 percent"]
    E2["P2 - 25 percent"]
    E3["P3 - 25 percent"]
  end
  subgraph Hot["One hot key"]
    H0["P0 - 75 percent"]
    H1["P1 - 10 percent"]
    H2["P2 - 8 percent"]
    H3["P3 - 7 percent"]
  end
        

When one key is too popular, there are ways to spread its load, though each has a price.

Idea 1: Add a small extra part to the key

This approach is often called salting. For the giant customer, instead of using only the customer identity as the key, the producer appends a small number, such as one to ten, chosen in rotation. The events of that customer now spread over up to ten partitions. The price is that order is now guaranteed only inside each of the ten sub-groups, not across the whole customer. Consumers that need the full picture must combine results from all the sub-groups.

Idea 2: A custom partitioner

The producer can use its own rule that treats known heavy keys specially, giving them several partitions, while ordinary keys follow the normal rule. This needs care, because every producer must use exactly the same rule.

Idea 3: Choose a finer key

Sometimes a more detailed identity exists. Instead of the shop, use the shop together with the order number. Events for one order stay in sequence, while the shop’s total load spreads across many partitions.

Idea 4: Give the busy topic more partitions

More partitions help spread many keys but do not help a single key, because one key still maps to one partition. This idea fixes crowding among many keys, not the dominance of one.

The connection between a key and its partition depends on how many partitions exist. This fact surprises many teams.

The map depends on the count

Remember that the partition is the remainder after dividing the hash by the partition count. If the count changes from six to eight, the remainder for the same key changes too. A customer whose events always landed in partition 3 may now land in partition 5.

What goes wrong when you add partitions

Kafka lets you increase the partition count of an existing topic, but it does not move old events. Old events for a customer stay in partition 3, while new events now go to partition 5. For a while, the events of one customer are split across two partitions, and a consumer might read the new partition faster than the old one, seeing events out of order. Also, programs that keep running totals per key in memory may now receive the same key on a different consumer, without its history.

For the promise “same key, same partition” to hold, every producer must calculate the partition in exactly the same way.

The hidden danger

Different programming languages and client libraries may use slightly different default rules for hashing keys. Two producers written in different languages, or using different libraries, could send the same customer’s events to different partitions. Similarly, if one producer serializes the customer number as text and another as a binary number, the bytes differ, the hash differs, and the events scatter.

How to stay safe

Keys shape the work of consumers as much as the work of producers.

One partition, one reader per group

Within a consumer group, each partition is read by exactly one member at a time. Because a key always goes to one partition, all events for that key are handled by one consumer of the group, in order. This is powerful: the consumer can keep running totals, a shopping cart, or a recent history for each key in its own memory, knowing no other member of the group will receive that key’s events.

Moves during rebalances

When a consumer joins or leaves, partitions are redistributed among the members, an event called a rebalance. A key’s partition may then move to another consumer. Well-built consumers recover their state, for example by reading it again from Kafka, or from a store, before continuing.

flowchart LR
  K1["Key: customer 1"] --> P0["Partition 0"]
  K2["Key: customer 2"] --> P1["Partition 1"]
  K3["Key: customer 3"] --> P2["Partition 2"]
  K4["Key: customer 4"] --> P0
  P0 --> C1["Consumer A"]
  P1 --> C2["Consumer B"]
  P2 --> C3["Consumer C"]
        

The key has a second important job: it identifies the “thing” in topics that keep only the latest state.

What is compaction?

Normally, Kafka deletes old events after a retention period. With log compaction, Kafka instead keeps at least the latest event for every key and may remove older events with the same key. The topic then behaves like a table of current values: for every key, the most recent value is always available, even after a very long time.

An example

Suppose a topic stores each customer’s current address, keyed by customer number. The customer moves three times, so three events exist with the same key. After compaction, only the latest address remains for that customer. A brand-new application can read the compacted topic from the beginning and quickly rebuild the current address of every customer, without replaying every historical change.

Deleting with a tombstone

To remove a key completely, a producer sends an event with that key and an empty value, called a tombstone. Compaction eventually removes the key entirely, after giving readers some time to see the deletion.

When streams are combined or counted, keys decide which events can meet each other.

Counting and grouping

Stream processing applications often count events per identity, such as orders per customer in the last five minutes. To do this correctly and quickly, all events of one customer must reach the same processor. The key makes this possible. If the events were not keyed by customer, the application would first need to re-key them, which creates an extra topic and extra work.

Joining two streams

Suppose one topic holds orders and another holds payments, and you want to match each payment to its order. For the match to happen inside one processor, both topics must use the same key, such as the order number, and both must have the same number of partitions. This arrangement is called co-partitioning. It guarantees that an order and its payment arrive in the same partition number of their respective topics, and therefore at the same processor.

flowchart TD
  O["Orders topic - keyed by order number"] --> J["Join processor for partition 2"]
  P["Payments topic - keyed by order number"] --> J
  J --> R["Matched order and payment"]
        

A key is stored with every event, in plain sight of anyone allowed to read the topic. Treat it with care.

Keys are not secret

The key travels with the event and is stored on disk, often for days or longer. Anyone with permission to read the topic can see it. Therefore, avoid using sensitive personal data, such as national identity numbers, email addresses, or card numbers, as raw keys. If you need such an identity to group events, use a stable internal identifier, or a protected version of the value, so that the actual sensitive data is not repeated in every event and across every backup.

Access rules

Kafka can require applications to prove their identity and can restrict which identities may read or write each topic. These rules apply to the whole topic, including its keys and values. If some data is more sensitive than the rest, keep it in a separate topic with tighter access.

You cannot see keys directly on a dashboard, but you can see their effects.

What to watch

Watch the volume of data written to each partition. A partition that receives far more than its siblings points to a hot key. Watch consumer lag per partition rather than only in total, since one lagging partition can hide inside a healthy-looking average. Watch the disk size of each partition, because a partition with a hot key grows much faster. If a topic uses compaction, also watch how well compaction keeps up.

Investigating a hot key

When you suspect skew, sample some events from the busy partition and count which keys appear most often. Often a single identity, a missing value, or a placeholder key accounts for most of the traffic. Fixing that one key can restore balance to the whole topic.

Keys are powerful, but they are a commitment.

Most key problems repeat across teams. Here are the most common.

Best practices at a glance

Let us design the keys for an online marketplace step by step.

flowchart LR
  OS["Order Service"] -->|"key: order number"| OT["Orders topic"]
  PS["Payment Service"] -->|"key: order number"| PT["Payments topic"]
  WS["Website"] -->|"no key"| VT["Product views topic"]
  SP["Seller Profile Service"] -->|"key: seller id"| ST["Seller profiles - compacted"]
  OT --> JN["Join processor"]
  PT --> JN
        

It is the optional identity attached to a Kafka event that is used to calculate which partition stores the event, so events with the same key stay together and in order.

No. Without a key, the producer spreads events across partitions for balance. Use a key only when you need ordering, grouping, joining, or compaction by identity.

No. Keys should be shared by all events that belong together. A unique key for every event gives no grouping and no useful order.

Yes, as long as the number of partitions and the partitioning rule stay the same. If you add partitions, the mapping changes for new events.

Yes. A producer can name an exact partition, or plug in a custom partitioning rule. This is rarely needed and requires every producer to follow the same rule.

A key that receives far more traffic than others, overloading the single partition it maps to. It can be fixed by choosing a finer key, adding a small extra part to the key, or using a custom rule for heavy keys.

No. The key is a separate field. Many designs repeat the identity inside the value too, so that readers can see it easily, but that is a design choice, not a requirement.

Events without a key are spread across partitions, so there is no order between them. Order exists only inside each partition.

Common reasons are that some events were sent without a key, that keys differ in format or spelling, that different producers use different hashing rules, or that the partition count was changed.

It is best not to. Keys are stored openly with every event. Use an internal identifier instead, so sensitive data is not spread through logs and backups.

We began with a post office that sorts letters purely by postcode, sending related letters to the same sack and van. The partitioning key is Kafka’s postcode. It is a small identity on each event that is turned into a partition number by a fast, fixed calculation, so all events with the same key travel together and in order. Keys give you ordering per object, allow counting, joining, and local state, and make compaction possible. They also come with responsibilities: choose a key with many distinct values and even traffic, keep the format and rule identical across producers, plan the partition count early, never rely on order across keys, and never hide sensitive data in a key. Choose the key with the same care you would give to designing a database index, because in Kafka it quietly shapes everything that follows.