Kafka Compaction, Explained Simply
A beginner-friendly guide to the basic concept of log compaction, the quiet housekeeping process that lets a Kafka topic hold only the latest, most relevant version of every piece of data.
Imagine a whiteboard in an office where each employee has their own labelled spot showing their current task for the day. Whenever someone’s task changes, they simply erase their old note and write the new one in the exact same spot. Nobody keeps a growing stack of every task that person has ever had; the whiteboard only ever shows the latest one. Kafka’s compaction feature brings this same idea to a topic, quietly keeping only the newest version of each labelled piece of data. In this guide we will build up the whole picture slowly, defining every new word the first time it appears.
1Two Very Different Kinds of Data
Before defining compaction precisely, it helps to see the two different needs it exists to serve.
A history of everything that happened
Some data genuinely needs to preserve every single thing that occurred, in order, forever, or at least for a meaningful period. A record of every payment ever made, or every step an order went through, is valuable precisely because the full history matters, not just the latest entry.
Only the current state actually matters
Other kinds of data behave very differently. Consider a customer’s current shipping address. Nobody typically cares about every address that customer has ever had; what matters is simply the one address that is correct right now. Storing every single past version forever, when only the newest one is ever actually useful, wastes space and adds unnecessary complexity for no real benefit.
Think of a sticky note on your refrigerator listing today’s grocery list. Each morning, you replace yesterday’s note with a fresh one. Nobody keeps every past grocery list stuck to the fridge forever; only today’s version has any real use. Compacted data behaves exactly like that sticky note.
Kafka needed a way to support both kinds well
Kafka’s ordinary retention settings, which remove data purely based on age or total size, work well for the first kind of data, a full history. But they are a poor fit for the second kind, current-state data, since an important, still-relevant record could be deleted simply because enough time had passed, even though nothing newer had come along to replace it. Compaction exists specifically to serve this second kind of data well.
2So, What Exactly Is Compaction?
A precise definition, then the plain-English version.
A short definition
Log compaction is a cleanup policy in Kafka that guarantees at least the latest known event for each distinct key, an identifying label attached to certain events, is retained within a topic’s partition, while allowing older events sharing that same key to eventually be removed once a newer one has replaced them. Unlike ordinary time-based or size-based retention, compaction does not remove data simply because it has become old; it removes data specifically because something newer with the same key has taken its place.
In everyday terms, compaction turns a Kafka topic into something closer to a constantly updating summary, where writing a new event for a key effectively replaces whatever that key’s entry used to say, even though, underneath, Kafka is still just appending new entries as always.
Compaction is a setting, not a separate kind of topic
It is worth being clear that compaction is a configuration choice applied to an ordinary Kafka topic, not some fundamentally different kind of topic. Any topic can be configured to use compaction, ordinary retention, or, in some setups, a combination of both, depending entirely on what that topic’s data actually represents.
Compaction relies entirely on the event’s key to decide what counts as “the same thing” being updated. A topic being compacted without meaningful, consistent keys would not behave usefully, since Kafka would have no reliable way to know which events represent updates to the same underlying record.
3How Compaction Actually Works, Step by Step
Following the process from an ordinary write to a fully compacted result.
Events are written as usual
A producer, the program sending data into Kafka, writes events into the topic exactly as it would with any ordinary topic, each one appended to the end of its partition’s log.
Multiple events can share the same key
Over time, several events might be written using the exact same key, each representing a newer update to the same underlying record.
A background process examines the log
Kafka periodically runs a background cleanup process that scans through the partition’s log, identifying older events whose key has since been written again more recently.
Older, superseded events are removed
These older, now-outdated events are removed, leaving behind only the most recent event for each key, while every other key’s own history remains untouched.
The log gradually shrinks and stabilises
As this process repeats over time, the partition’s overall size tends to stabilise, since it now only needs to hold, at most, one entry per distinct key, no matter how many updates actually happened historically.
flowchart LR
Before["Before: key-A v1, key-B v1, key-A v2, key-B v2, key-A v3"]
Before -- compaction --> After["After: key-B v2, key-A v3"]
4A Concrete, Worked Example
Seeing actual events move through compaction makes the whole idea click.
Tracking a customer’s current address
Imagine a topic named “customer-addresses”, compacted, using each customer’s ID as the key. Over several months, customer 4521 moves house twice, so three separate events are written for that same key: one for their original address, one for their first new address, and one for their most recent address after moving again.
| Order Written | Key | Value | Survives Compaction? |
|---|---|---|---|
| 1st | customer-4521 | 123 Elm Street | No, replaced |
| 2nd | customer-4521 | 45 Oak Avenue | No, replaced |
| 3rd | customer-4521 | 9 Birch Lane | Yes, this is the latest |
After compaction runs, only the third event, showing “9 Birch Lane”, remains for customer 4521’s key. Any system reading this topic from the beginning would only ever see this single, current address for that customer, rather than needing to work through the customer’s full moving history to figure out where they live today.
While compaction is running, or before it has had a chance to run, older versions might still briefly be present in the log. Compaction is an ongoing background process, not an instant, immediate action the moment a new event is written.
5Tombstones: Marking Something as Deleted
Compaction also needs a way to represent “this record no longer exists at all”.
An event with no value
Sometimes a piece of data being tracked genuinely needs to be removed entirely, not just updated, such as a customer closing their account. Kafka handles this using a special kind of event called a tombstone, which is simply an event carrying the relevant key but with an empty value.
What a tombstone tells the compaction process
When compaction encounters a tombstone, it treats it as a signal that everything previously known about that key should eventually be removed entirely, including the tombstone itself, once a further, separate grace period has passed. This gives downstream consumers, programs reading the topic, a reliable window of time to notice the deletion before all trace of that key eventually disappears from the log.
Think of writing “CLOSED” on that same whiteboard spot when an employee leaves the company, rather than simply erasing their name immediately. This gives everyone glancing at the board a clear, visible signal that this spot is no longer active, before it is eventually wiped clean entirely.
sequenceDiagram
participant P as Producer
participant T as Compacted Topic
P->>T: key: customer-4521, value: 9 Birch Lane
P->>T: key: customer-4521, value: null (tombstone)
T-->>T: Tombstone visible for a grace period
T-->>T: Key and tombstone eventually removed entirely
Beginners sometimes assume a tombstone deletes its key immediately. In reality, the tombstone itself remains visible in the log for a configured grace period, giving consumers time to react to the deletion, before it is finally cleaned away for good.
6Clean and Dirty Segments
A peek at the internal terminology Kafka uses to describe compaction’s progress.
What “dirty” and “clean” mean here
Within a compacted partition, the portion of the log that has already been processed by compaction, meaning any duplicate, older keys within it have already been removed, is often referred to as the clean portion. The more recently written portion that has not yet been processed by compaction is referred to as the dirty portion, since it may still contain outdated, superseded entries.
Why this distinction exists
Compaction does not need to reprocess the entire log every single time it runs. It only needs to focus its work on the dirty portion, since the clean portion has already had its redundant entries removed. This makes the ongoing compaction process considerably more efficient than repeatedly reprocessing the entire partition from scratch every time.
flowchart LR
Clean["Clean portion (already compacted)"] --> Dirty["Dirty portion (not yet compacted)"]
A configuration setting controls how much of a log is allowed to become dirty before compaction runs again, letting teams balance how quickly space is reclaimed against how much background work compaction performs.
7Compaction Versus Ordinary Retention
A direct comparison clears up the most common confusion between the two.
| Aspect | Ordinary Retention | Compaction |
|---|---|---|
| What triggers removal | An event’s age or the topic’s total size | A newer event with the same key arriving |
| Guarantee provided | Data is kept for at least the configured period or size | The latest event per key is kept, regardless of age |
| Best fit | A full history of everything that happened | The current state of something, per key |
| Behaviour of very old data | Eventually removed once past the limit | Kept indefinitely if never replaced by a newer event |
If a topic’s purpose is “show me everything that has ever happened”, ordinary retention usually fits better. If a topic’s purpose is “show me the current state of each thing”, compaction is almost always the better choice.
8Combining Compaction with Ordinary Retention
These two approaches are not always mutually exclusive.
A hybrid cleanup policy
In some setups, a topic can be configured to use both compaction and ordinary time-based retention together. In this arrangement, compaction continues to guarantee that the latest event per key is kept, while ordinary retention can still remove very old, already-superseded entries once they exceed a certain age, helping the log stay even smaller over time.
When this combination makes sense
This hybrid approach can be useful for topics where teams want the safety of “always keep the latest state per key”, provided by compaction, but also want an extra layer of cleanup for particularly old, clearly outdated data, without waiting indefinitely for compaction alone to eventually process it.
Example: A User Session State Topic
A topic tracking each user’s current session state might combine compaction, so the latest state per user is always available, with a modest additional retention limit, ensuring extremely old, already-replaced entries are cleaned up promptly rather than lingering unnecessarily.
9Reading a Compacted Topic
What a consumer actually experiences when reading data from a compacted topic.
Rebuilding current state from the beginning
A consumer, a program reading data from Kafka, can read a compacted topic from its very earliest available offset to reconstruct a complete, current picture of every key’s latest value, effectively loading the entire current state into its own memory or local storage as it processes each entry in order.
Why this is genuinely useful
This gives a brand-new service an easy, reliable way to build up an accurate, current snapshot of an entire dataset, simply by reading through a compacted topic from the start, without needing a separate database export or a specially built synchronisation process just for that purpose.
Think of walking past that office whiteboard and reading every currently visible note, one spot at a time, to get a complete, accurate picture of everyone’s current task, all at once, without needing to ask each person individually.
10Advantages and Trade-offs of Compaction
Understanding both sides helps in deciding when compaction is the right tool.
Advantages
- Keeps a topic’s size manageable over long periods, regardless of update frequency
- Guarantees the latest state per key is always available, without a time limit
- Lets new consumers easily rebuild a complete current-state snapshot
- Supports clean, explicit deletion of a key entirely using tombstones
Trade-offs to Keep in Mind
- Not suitable for data where the full historical record genuinely matters
- Requires consistent, meaningful keys to behave usefully at all
- Compaction runs as a background process, so cleanup is not instantaneous
- Tombstones need a well-understood grace period to avoid confusing consumers
11Real-World Uses of Compaction
Seeing a few concrete examples brings the whole idea to life.
Customer Profile Data
A compacted topic can hold the current profile details for every customer, keyed by customer ID, always reflecting only the latest known information for each person.
Configuration Settings
A compacted topic can store an application’s current configuration values, keyed by setting name, letting services load the latest configuration simply by reading the topic through.
Backing a Materialised View
A processing application can maintain a compacted topic as a continuously updated summary, such as each product’s current stock level, ready to be queried or rebuilt at any time.
Storing Internal Cluster Metadata
Kafka itself uses compacted internal topics to track certain kinds of its own operational metadata, relying on exactly the same latest-value-per-key guarantee described throughout this guide.
12Best Practices and Common Mistakes
Practical lessons for using compaction well.
Best practices
Choose keys that genuinely represent the “thing” being tracked
Make sure every event sharing a key truly represents updates to the same underlying record, so compaction behaves meaningfully.
Use tombstones deliberately for real deletions
Publish a tombstone whenever a key’s underlying record has genuinely ceased to exist, rather than leaving stale entries behind indefinitely.
Give tombstones a sensible grace period
Ensure consumers have a realistic chance to notice and react to a deletion before the tombstone itself is fully cleaned away.
Reserve compaction for genuinely current-state data
Avoid applying compaction to topics where a full historical record is actually needed for the system to work correctly.
Monitor how much of the log remains dirty over time
Keep an eye on compaction’s progress to ensure it is keeping up with the pace of new updates flowing into the topic.
Mistakes beginners often make
Turning on compaction for a topic where events do not have a consistent, meaningful key produces confusing, unhelpful results, since Kafka has no reliable way to know which events represent updates to the same thing.
Assuming an older, superseded event disappears the instant a newer one is written ignores the fact that compaction is a background process, running periodically, not an immediate action.
Applying compaction to a topic that genuinely needs to preserve every historical event, such as a full audit trail, silently loses valuable information the moment older entries are cleaned away.
13Frequently Asked Questions
Not by itself. Compaction removes older events specifically because a newer event with the same key has replaced them, regardless of how much time has passed. An event with a key that has never been updated can survive indefinitely under pure compaction.
A tombstone is a special event carrying a key but with an empty value, signalling that everything previously known about that key should eventually be removed entirely, after a grace period.
Yes. Some setups combine both, letting compaction guarantee the latest event per key is kept, while ordinary retention additionally removes very old, already-superseded entries after a certain age.
No. Compaction runs as a periodic background process. Older, superseded events may still briefly remain in the log until the compaction process has had a chance to process that portion of the data.
Without a tombstone, compaction will simply continue to keep that key’s latest known value indefinitely, since nothing has signalled that the underlying record should actually be considered deleted.
Yes. From a consumer’s perspective, reading a compacted topic works the same way as reading any other topic; the difference lies entirely in which events remain available to be read over time.
It depends on what the topic needs to represent. If only the current status matters, compaction fits well. If the full sequence of status changes over time needs to be preserved and replayed, ordinary retention, without compaction, is usually the better choice.
14Summary and Key Takeaways
Compaction is Kafka’s answer to a simple but important need: keeping only the latest, current state of something, per key, without needing a time limit or worrying about full historical bloat. By running as a periodic background process, comparing keys and removing older, superseded events, compaction lets a topic’s size stabilise naturally over time, even as updates continue indefinitely. Tombstones extend this idea further, giving teams a clear, explicit way to signal that a key’s underlying record has genuinely been deleted. Understood alongside ordinary retention, and sometimes combined with it, compaction gives Kafka a flexible, purpose-built tool for representing current state, not just historical events, directly within the same reliable, familiar system.
Key Takeaways
- Compaction keeps only the latest event per key — removing older, superseded entries as newer ones arrive.
- It is triggered by updates, not by age — unlike ordinary retention, which removes data purely based on time or size.
- Tombstones signal genuine deletion — an empty-value event marking a key for eventual, complete removal.
- Compaction runs as a background process — so cleanup is gradual, not instantaneous.
- Clean and dirty portions make cleanup efficient — compaction only needs to reprocess the not-yet-cleaned part of the log.
- Compaction and ordinary retention can be combined — giving teams flexibility for specific, hybrid needs.
- It shines for current-state data — profiles, configuration, and materialised views, not full historical records.