Before defining a message key precisely, it helps to see the exact problem it solves.

A topic is split for a reason

A Kafka topic is a named stream of events, and to handle large amounts of traffic, a topic is usually split into smaller pieces called partitions, spread across different machines called brokers. This splitting lets many partitions work in parallel, but it immediately raises a question: when a new event is written, which partition should it actually go into?

If events were scattered across partitions completely at random, related events about the same customer, order, or device could easily end up spread across several different partitions. Kafka guarantees strict ordering only within a single partition, so events landing in different partitions have no guaranteed order relative to each other. For many real systems, this randomness would be a serious problem.

The need for deliberate, consistent grouping

What is really needed is a way to say “these events belong together, always send them to the same place, in the order they were created.” This is exactly the role a message key plays inside Kafka.

A precise definition, then the plain-English version.

A short definition

A message key, often just called a key, is an optional piece of data attached to a Kafka event, alongside its main content, which is called the value. While the value carries the actual information an event is describing, the key’s main job is to identify which group an event belongs to, most commonly used to decide which partition of a topic the event is written to.

In everyday terms, if the value is the letter inside an envelope, the key is the address written on the outside. Kafka looks at the address to decide where the envelope should go, without needing to open it and read the contents.

Keys can be almost anything

A key is usually something meaningful to the business, such as a customer ID, an order number, a device identifier, or a product code. Kafka does not require the key to follow any particular format; it simply treats it as a sequence of bytes, leaving the meaning entirely up to the producers and consumers that agree on what it represents.

The mechanical process behind the scenes, explained simply.

Running the key through a formula

When a producer, the program sending an event into Kafka, includes a key, Kafka runs that key through a mathematical formula called a hash function. A hash function takes any input and turns it into a number in a very consistent, repeatable way, meaning the exact same input will always produce the exact same output number. Kafka then uses this number, combined with the total number of partitions in the topic, to decide exactly one partition for that key to always be sent to.

Why this guarantees consistency

Because the hash function always produces the same number for the same key, and the number of partitions does not normally change during this process, every event sharing the same key will reliably be routed to the very same partition every single time, for as long as the topic’s partition count stays the same. This is what allows Kafka to promise that related events, grouped by a shared key, will always land together and stay in the order they were written.

flowchart LR
  K1["Key: customer-101"] --> H
  K2["Key: customer-204"] --> H
  K3["Key: customer-101"] --> H
  H --> P0[Partition 0]
  H --> P1[Partition 1]
        

Not every event needs to be grouped, and Kafka handles that case too.

Spreading events evenly instead

When an event is published with no key at all, Kafka cannot use the hashing approach described above, since there is nothing to hash. Instead, it falls back to a strategy designed to spread events evenly across all the available partitions, commonly batching several keyless events together and rotating which partition receives each new batch, so that no single partition becomes overloaded while others sit idle.

When this is actually the right choice

Leaving out a key is perfectly reasonable, and often preferable, when events genuinely have no natural grouping and simply need to be spread out for balanced, parallel processing. A stream of unrelated, independent metric readings, where order between different readings does not matter, is a good example of data that often does not need a key at all.

This is the single most important idea to understand about message keys.

Order is only guaranteed within a partition

Kafka guarantees that events are stored and delivered in the exact order they were written, but strictly only within a single partition. There is no guarantee about the relative order of events sitting in different partitions of the same topic. This single fact is precisely why keys matter so much: since the same key always maps to the same partition, using a shared key is the only reliable way to guarantee that a related group of events stays in order relative to each other.

A concrete example

Consider a system tracking the status of customer orders, where each order moves through stages such as “placed”, “packed”, and “shipped”. If these three events for order number 4521 used the customer’s order ID as the key, all three would land in the same partition, in the correct order. If they instead had no key and were spread randomly, a consumer might see “shipped” arrive before “packed”, which would be confusing and incorrect.

sequenceDiagram
  participant P as Producer
  participant K as Key: order-4521
  participant Part as Partition 2
  P->>K: order-4521, "placed"
  K->>Part: Written first
  P->>K: order-4521, "packed"
  K->>Part: Written second
  P->>K: order-4521, "shipped"
  K->>Part: Written third
        

Not every field makes a good key, and the choice has real consequences.

What makes a key effective

A good key should represent the natural grouping that actually matters for ordering, such as a customer ID, an account number, or a device identifier. It should also have enough variety across all the events flowing through the topic, so that events are spread reasonably evenly across the available partitions rather than piling up in just one or two.

The danger of a “hot” partition

If one particular key value is used far more often than any other, for example if a single very large customer accounts for a huge share of all order events, every one of those events will consistently land on the exact same partition. That partition can become overloaded compared to the others, a situation often called a hot partition, since it does far more work than its neighbours while they sit relatively idle.

Keys play a second, very different role in a special kind of Kafka topic.

Keeping only the latest event per key

Some topics are configured to use a special retention mode called log compaction, where Kafka does not keep every historical event forever, but instead keeps only the single most recent event for each distinct key, quietly removing older events once a newer one with the same key has been written. This is extremely useful for topics representing the current state of something, rather than a full history of everything that ever happened.

An example: current customer details

A topic storing the latest known address for every customer might use the customer ID as its key, with log compaction enabled. Every time a customer updates their address, a new event is published with that customer’s ID as the key. Kafka eventually removes the older address events for that same customer, keeping only their most recent address on file, without needing a separate database just to track current state.

The default hashing behaviour is not the only option available.

When the default is not quite right

The standard hash-based approach works well for most situations, but some systems have special requirements that call for a different strategy. Kafka’s producer libraries generally allow a custom partitioning strategy to be defined, letting a team decide exactly how keys, or even other information about an event, should map to partitions, rather than relying purely on the built-in hash function.

An example use case

A system might want to send all events for customers in a certain region to a specific set of partitions handled by machines physically closer to that region, for performance reasons. A custom partitioning strategy could implement this kind of rule directly, going beyond what simple key hashing alone would achieve.

A subtle but important consequence of how key-based partitioning works.

The mapping can shift

Because the partition for a given key depends on both the key itself and the total number of partitions in the topic, adding new partitions to an existing topic can change which partition a given key maps to going forward. Events published with a particular key before the change might be on a different partition than events with that same key published after the change.

Why this matters for ordering

If a system depends on all events for a given key staying together and in order, this kind of shift can quietly break that guarantee, since older and newer events for the same key could end up split across two different partitions. This is one of the key reasons experienced teams try to choose a sensible partition count early on, rather than relying on frequent changes later.

Deliberately choosing a key brings real benefits, alongside a few trade-offs worth knowing.

Seeing a few concrete, everyday examples helps the idea click firmly into place.

Practical lessons for using message keys well.

Best practices

Mistakes beginners often make

No. A key is optional. Events without one, called keyless events, are simply spread across partitions in a balanced way rather than being grouped by a shared identifier.

Yes. This is completely normal, since a topic usually has far fewer partitions than there are possible key values. What matters is that any single key always consistently maps to the same partition.

Both. The key is stored as part of the event itself, alongside its value and timestamp, so consumers can also read and use it, not only Kafka’s internal partitioning logic.

Kafka treats a key as raw bytes and does not enforce a particular format. In practice, keys are commonly strings or numbers representing an identifier, such as a customer ID or order number.

Only by using a single partition for the whole topic, which guarantees overall order but sacrifices the ability to process events in parallel. Using a meaningful key on a multi-partition topic is usually the better solution when both parallelism and grouped ordering are needed.

Yes. Keys are scoped to a single topic’s partitioning decisions. Using the same key value in a different topic has no automatic relationship to how it behaves in another topic.

Monitoring tools that show traffic per partition can reveal this clearly. If one partition consistently receives far more events than the others, it is a strong sign the chosen key does not have enough variety.

A message key is a small but powerful piece of an event that tells Kafka how to group related data together. By consistently hashing the same key to the same partition, Kafka guarantees that events sharing a key stay in the exact order they were written, which is essential for anything from order tracking to account balances. Events without a key are simply spread evenly across partitions, which is perfectly fine when no such grouping is needed. Keys also power log compaction, letting a topic represent only the latest state for each distinct entity. Choosing a good key, one with enough variety to avoid overloading any single partition, is one of the most important design decisions when building anything on top of Kafka.