Compaction answers one question: what if I only care about the newest value for each thing, not the whole history?

Kafka is a system that stores streams of records. A record is a small package of data. Every record can have a key, a value, and a timestamp. The key is a label that says what the record is about. The value is the actual information. For example, a record might have the key user-42 and a value that holds that user’s email address.

Now imagine your app sends a new record every time user 42 changes their email. After a year, the stream contains hundreds of records with the key user-42. Only the last one describes what is true today. The other hundreds describe the past.

The problem with plain time-based deletion

By default, Kafka cleans up old data using a retention rule. For example: “delete anything older than seven days.” That is perfect for things like click events or sensor readings, where the past matters only for a short while.

But look at what happens to our user profile stream. Suppose user 42 changed their email once, ten days ago, and has not touched it since. The retention rule sees that record as older than seven days and deletes it. Now Kafka knows nothing about user 42, even though that record was the newest truth about them. The rule removed data by age, not by usefulness.

What compaction does instead

Log compaction changes the rule from “delete by age” to “delete by being replaced.” Kafka promises to keep at least the last known value for every key. Older values for the same key can be thrown away, because a newer one has taken their place. A quiet key that has not changed in a year stays safe, because nothing has replaced it.

flowchart TB
  subgraph B["Before compaction"]
    direction LR
    b0["offset 0, K1 = A"] --> b1["offset 1, K2 = B"] --> b2["offset 2, K1 = C"] --> b3["offset 3, K3 = D"] --> b4["offset 4, K2 = E"] --> b5["offset 5, K1 = F"] --> b6["offset 6, K3 = null"]
  end
  subgraph A["After compaction"]
    direction LR
    a4["offset 4, K2 = E"] --> a5["offset 5, K1 = F"] --> a6["offset 6, K3 = null, a tombstone"]
  end
  B -.->|cleaner runs| A

Read the picture from left to right. Key K1 was written three times (A, C, then F), so only F remains. Key K2 was written twice (B, then E), so only E remains. Key K3 was written with a value D and then with null. A null value is special: it means “this key is deleted.” We call it a tombstone, and Chapter 8 explains its whole life story.

You cannot understand the cleaner without knowing what it cleans. Here is the storage layout in plain words.

Kafka stores data in topics. A topic is like a named folder for one kind of record, such as user-profiles or orders. To handle huge amounts of traffic, each topic is split into partitions. A partition is one ordered line of records, and it lives on disk as a log. Compaction works on one partition log at a time, never across partitions.

How segments look on disk

Inside a partition folder you will see files named after the first offset they hold. A segment called 00000000000000368769.log starts at offset 368769. Next to it are .index and .timeindex files. The .log file holds the actual records in batches. Kafka groups records into batches before writing, because moving many records together is much faster than moving them one by one. Compaction has to respect this batch structure when it rewrites data.

Why the key matters so much

When a producer sends a keyed record, Kafka’s default partitioner hashes the key and picks a partition. The same key therefore always lands in the same partition, and it keeps its order there. This is the foundation of compaction. Since all records for user-42 sit in one partition in the exact order they were written, the cleaner can decide “which record is newest” just by looking at offsets in that one log. If the same key were spread over many partitions, no single cleaner could compare its records.

Every topic has a cleanup.policy setting that decides how old data leaves the log.

A topic can use one of three cleanup styles. The setting is called cleanup.policy at the topic level. Choosing the right one is the first design decision you make, so it helps to see them side by side.

The default delete policy in numbers

With the delete policy, the broker checks whether whole segments are too old or the partition is too big. By default, the time limit is seven days (retention.ms of 604,800,000 milliseconds) and there is no size limit (retention.bytes of -1). The broker removes entire closed segments, never single records. This is cheap because deleting a file is quick.

Compaction is a different kind of work

Compaction cannot just delete whole files, because a segment usually contains a mix of records to keep and records to drop. It has to read the segment, filter out the losers, and write a new smaller file. That is real disk and CPU work, and it is why compaction has so many tuning settings while plain deletion has almost none.

The combined policy

You can set cleanup.policy to compact,delete. Then two rules apply at once. The cleaner removes replaced values, and the retention rule also removes anything that is older than the retention time, even if it is the latest value for its key. This is how Kafka Streams stores windowed data: “keep the latest per key, but forget windows older than the window retention.”

Good engineers know the guarantees by heart. Most production bugs come from expecting more than Kafka promised.

Compaction is safe to build on because it makes a clear set of promises. Understanding both the promises and the limits will save you from painful surprises.

The promises

What it does not promise

Gaps in offsets are normal

After compaction, a partition might hold offsets 4, 5, 6, 11, 12, and 20. The missing numbers belonged to replaced records. A consumer asking for offset 7 simply gets the next available record, offset 11. Well-written consumers never assume that offsets are consecutive. If your code counts records by subtracting offsets, it will be wrong on a compacted topic.

Duplicates in the newest part of the log

The cleaner only works on segments that are closed and old enough. So the newest part of the log can still hold many records for the same key. A consumer reading near the head will see all of them, in order. This is a feature, not a bug: it means live consumers get every change, while late starters get a compact snapshot.

Compaction is not a separate service. It is a small team of workers living inside every Kafka broker.

Each Kafka broker runs its own cleaning machinery inside the same program that serves producers and consumers. There is no extra server to install. When a broker starts, and cleaning is enabled (the default), it also starts a pool of cleaner threads. Let us meet each part of the team.

flowchart TB
  P["Producers write keyed records"] --> AL["Active segment of a partition log"]
  AL --> CS["Closed segments on disk"]
  LCM["LogCleanerManager picks the dirtiest log"] --> CT["Cleaner thread"]
  CT --> OM["Offset map: key hash to newest offset"]
  CT --> CS
  CT --> NEW["Replacement segments, first .cleaned then .swap"]
  NEW --> CS
  CT --> CP["cleaner-offset-checkpoint file"]
  LCM --> CP
  C["Consumers read the log"] --> CS
  C --> AL

Where each replica fits in

Every partition is stored on several brokers as replicas: one leader and a few followers. Each broker cleans its own copy independently. There is no central cleaning command from the leader. Because compaction never changes offsets and only removes records that are provably outdated, every replica still ends up with the same latest value for every key, even if the exact timing of cleaning differs.

The three regions of a log

To talk clearly about the cleaner, we split a partition log into three regions, from oldest to newest.

flowchart LR
  S["Start of log"] --> CL["Clean region: already compacted, one record per key"]
  CL --> DR["Dirty region: newer closed segments, keys may repeat"]
  DR --> UN["Uncleanable region: active segment and protected recent segments"]
  UN --> H["Newest offset"]

• Clean region: the older part that a previous run has already compacted. Inside it, each key appears at most once.
• Dirty region: closed segments written after the last cleaning. The same key may appear many times here.
• Uncleanable region: the active segment, plus any segments that are protected by a minimum compaction lag or by open transactions. The cleaner leaves these alone for now.

The Kafka documentation calls the old part the “tail” and the newest part the “head” of the log. In this guide we use clean and dirty because those words tell you what is inside.

How the planner picks a log: the dirty ratio

The planner computes a number for each log: the cleanable ratio, also called the dirty ratio. It is the size of the dirty region divided by the size of the clean plus dirty regions. A ratio of 0.5 means half of the cleanable data is new and unchecked. The setting min.cleanable.dirty.ratio (default 0.5) says how dirty a log must be before it is worth the effort. Among all logs above the limit, the planner picks the one with the highest ratio.

This is the heart of the guide. Follow one thread as it compacts one partition.

A cleaning run has two big phases: first learn who the newest records are, then copy the log while dropping everything that is not the newest. Here are the steps in order.

Why the cleaner re-reads the clean region

This surprises many people. The map is built only from the dirty region, yet the copy phase starts at the very beginning of the log. Why touch data that was already cleaned? Because new dirty records can replace old clean ones. Imagine key K1 has a value in the clean region. Yesterday’s dirty segment holds a newer K1. The old K1 in the clean region is now outdated and must go. The only way to find it is to walk the clean region again and check every key against the map.

Rewriting batches without breaking producers

Kafka records live in batches, and each batch header carries information about the producer that wrote it, such as its ID and sequence numbers. These are used for idempotent writes and transactions. When the cleaner drops every record in a batch, it can still keep the empty batch shell for the latest batch of each producer, so the broker does not lose the producer’s sequence state. The cleaner also removes records from aborted transactions, since nobody should ever read them, and it drops old transaction marker records after they are no longer needed.

If batches were compressed, the cleaner has to decompress, filter, and compress again. This costs CPU, which is one more reason the cleaner runs in the background with its own throttle.

Producers and consumers keep working

Nothing in the cleaning run blocks writes, because writes only touch the active segment, which the cleaner ignores. Consumers reading old offsets keep reading the old files until the swap; after the swap, they continue from the same offsets on the new files. This works because offsets are stable, which is the single most important design choice behind compaction.

A tiny, clever hash table decides how much of your log can be cleaned in a single run.

The offset map is a fixed block of memory that acts like a hash table. It is nicknamed “skimpy” in the source code because it stores as little as possible. It does not keep your keys. It only keeps a fingerprint of each key plus an offset.

What one entry looks like

A key of 5 bytes and a key of 5,000 bytes both cost exactly 24 bytes in the map. This is a big memory saver when keys are long, such as full URLs or JSON strings.

How much can one run cover?

The total memory for the map is set by log.cleaner.dedupe.buffer.size, default 128 MB. It is divided equally among the cleaner threads. The map also refuses to be more than 90 percent full (the load factor from log.cleaner.io.buffer.load.factor), so lookups stay fast. With one thread and the defaults:

So a single run can look at roughly five million unique keys in the dirty region. If your dirty region holds 50 million unique keys, the cleaner will need about ten runs to work through it. Doubling the buffer doubles the reach. Because many records in the dirty region repeat the same key, the number of records covered per run can be much greater than the number of unique keys.

Open addressing and collisions

The map uses a simple technique called open addressing. To store a key, it turns the fingerprint into a slot number. If the slot is taken by a different fingerprint, it tries the next slot, and the next, until it finds a free one. Lookups follow the same path. Since the fingerprints are 128 bits wide, the chance that two different keys share one is astronomically small. The design accepts that tiny risk in exchange for using far less memory than storing full keys.

What happens when one segment is too big for the map

The cleaner adds whole segments to the map as long as they fit. If even a single segment holds more unique keys than the empty map can take, the cleaner cannot make progress on that partition and reports an error. The fixes are to raise the buffer size, lower the segment size so each segment has fewer keys, or add cleaner threads only if you also add memory. This is one of the most common real-world compaction problems, and it is easy to miss until it shows up in monitoring.

You cannot erase a line from an append-only log, so Kafka deletes by writing a special note that means “this key is gone.”

A tombstone is a record that has a key but a null value. It tells the world that the key no longer exists. Note that an empty string is not null; a tombstone must have no value at all.

The life of a tombstone

flowchart LR
  W["1. Producer writes key with null value"] --> L["2. Tombstone appended as the newest record for the key"]
  L --> C1["3. Cleaner removes all older values of the key"]
  C1 --> KEEP["4. Tombstone is kept for the delete retention time"]
  KEEP --> C2["5. A later cleaning run drops the tombstone"]
  C2 --> GONE["6. Key has fully vanished from the log"]

Why not remove the tombstone immediately along with the old values? Because of consumers that keep their own copy of the data. Imagine a service that reads a compacted topic to fill a local cache. If it is offline while the cleaner removes both the old value and the tombstone, it will never learn the key was deleted. Its cache would hold a ghost entry forever. By keeping the tombstone for a grace period, Kafka gives every consumer time to see it.

The grace period: delete retention

The topic setting delete.retention.ms controls the grace period. Its default is 24 hours (86,400,000 milliseconds). A consumer that is away for less than that time is guaranteed to see the tombstone. In newer Kafka versions, the countdown begins when the cleaner first processes the tombstone in a cleaned segment, so the guarantee is a full window. Older versions measured from the last-modified time of the segment, which could shorten the real window.

Tombstones can be deleted too, on purpose

Some teams want deleted data gone quickly, for privacy or storage reasons. They lower delete.retention.ms. This is fine if all consumers are always online or rebuild after downtime. It is risky otherwise. The choice is a trade-off between deletion speed and consumer safety, discussed further in Chapter 13.

Producing a tombstone

Every Kafka client library offers a way to send a record with a key and a null value. Kafka Streams turns a null update of a table row into a tombstone. Debezium, the change-data-capture tool, emits a delete event followed by a tombstone by default, so that compacted topics can forget the row.

Let us follow a single key through three moments in time to see exactly what compaction does.

Suppose a topic called user-profiles has one partition. Our story covers users u1, u2 and u3. The table shows what the log holds at three moments.

Moment 1: before any cleaning

Moment 2: after the first cleaning run

The cleaner builds its map: u1 maps to offset 4, u2 to offset 1, u3 to offset 5. It then copies the log and keeps only records that match the map.

Offsets 0, 2 and 3 are gone. The offsets 1, 4 and 5 kept their numbers. A consumer reading from offset 0 now jumps to offset 1, then 4, then 5.

Moment 3: after the grace period and another run

The tombstone at offset 5 has been dropped, and u3 has completely vanished from the topic. Any consumer that read the log during the grace period knows u3 was deleted. Any new consumer never even hears about u3, which is exactly right.

What different readers experience

Settings come in two families: topic-level settings that shape one topic, and broker-level settings that shape the cleaner machinery.

Topic settings decide when a log becomes eligible and how long special records live. Broker settings decide how much power the cleaner has. Here are the defaults as of recent Kafka releases. Always confirm against the documentation for your exact version.

Topic-level settings

Broker-level settings

Understanding the three timing settings together

Three settings work like a small traffic light system for when a record may be compacted.

• Dirty ratio says: “Do not start unless enough new data has piled up.”
• Minimum compaction lag says: “Do not touch records younger than this.” It protects consumers who need to see every change for a while, and it gives slow consumers time to catch up.
• Maximum compaction lag says: “Even if the ratio is low, start no later than this.” It puts a ceiling on how long duplicates and deleted data may linger, though it is a target and not a strict deadline, because the cleaner threads may be busy.

Choosing segment size for compacted topics

The active segment is never cleaned. If segment.bytes is 1 GB and your topic gets only a few megabytes a day, the active segment could stay open for a week (the segment.ms default), and everything inside it stays uncompacted for that time. For compacted topics that need timely cleanup, use smaller segments such as 100 MB, or a shorter segment.ms such as one hour. Kafka itself uses 100 MB segments for its internal offsets topic for this reason. Do not go too small: thousands of tiny segments mean many open files and many small reads.

Compaction spends disk, memory and CPU in return for smaller logs and faster restarts. Here is how to keep the bill fair.

Where the cost comes from

The dirty ratio trade-off

Tuning steps that usually help

An illustrative capacity example

Suppose a topic keeps the latest profile for 20 million users, each about 1 KB. The compacted size is about 20 GB per replica, no matter how many updates happened in total. With a replication factor of 3, that is 60 GB across the cluster. A delete-policy topic holding a week of every update might need many times more. If each user updates twice a day, seven days means 280 million records, or about 280 GB per replica. Compaction stores the same current state in roughly one-fourteenth of the space, and a new consumer can load it in minutes instead of hours.

Scaling out

Compaction scales the same way Kafka scales: by adding partitions and brokers. More partitions split the keys, which means each partition has a smaller offset map to fill, and more cleaner threads across the cluster can work in parallel. The price is more files and more metadata, so choose partition counts carefully at design time, since changing the count later breaks the key-to-partition mapping that compaction depends on.

Compaction rewrites files in a living system, so Kafka built it to survive crashes at any moment.

Independent cleaning, shared meaning

Leaders and followers each clean their own copy. Their files may look different in detail, for example with different segment boundaries, but the surviving records for each key match. Since offsets never change, a follower can keep fetching from a leader and a consumer can move to a new leader after failover without any confusion.

Crash safety during the swap

The swap step is designed so that a crash at any point leaves the log recoverable. New data is fully written to a .cleaned file first. Only when it is complete is it renamed to .swap. On broker startup, Kafka looks for leftovers. An unfinished .cleaned file is simply deleted because the original segments are still intact. A finished .swap file is used to complete the replacement. This means the log is never left with half of the old data and half of the new.

Common failure scenarios

Transactions and the safe cleaning limit

If a producer has an open transaction, its records are not yet decided as committed or aborted. The cleaner does not clean past the first offset of any open transaction, so it will never drop or keep a record based on an unfinished decision. Long-running transactions can therefore hold back compaction for their partition, which is one more reason to keep transactions short.

The uncleanable partition idea

A bad partition should not stop the whole cleaner. When a serious error happens for one partition, Kafka marks that partition as uncleanable, records it in metrics, and moves on to other partitions. This isolates the damage, but it also means a silent problem can hide unless you watch the metrics.

Compaction can help you forget data, but it does not forget on a schedule you can promise to a regulator.

Who can delete what

In Kafka, permission to write to a topic includes permission to write tombstones. That means anyone who can produce to a compacted topic can effectively delete any key in it. Treat write access to compacted topics like write access to a database table. Use ACLs so only the owning service can write, and separate read-only and read-write roles clearly.

Also protect topic configuration changes. Someone who can alter delete.retention.ms or the cleanup policy can change what gets erased and when. Limit config-altering rights to platform administrators.

The right-to-erasure problem

Privacy laws such as the GDPR include a right to have personal data erased. Teams often say: “We publish a tombstone, and compaction removes the data.” That is only partly true. Consider all the places the old value can still live:

• In the active segment, which the cleaner cannot touch until it rolls.
• In dirty segments waiting for the next run to trigger.
• On every follower replica, each cleaned separately.
• In records protected by the minimum compaction lag.
• In backups, mirrored clusters, downstream topics, and consumer databases.

Making erasure faster

Encryption and network safety

Compaction itself does not add encryption. Protect data in transit with TLS between clients and brokers and between brokers. Protect data at rest with disk-level or volume-level encryption. If a value contains sensitive fields, consider encrypting the field in the application before it is produced, because compaction copies records byte for byte and anyone with disk access sees the stored bytes.

A cleaner that quietly stops is worse than one that fails loudly. Watch these signals.

Key metrics

Logs

Kafka writes cleaner activity to a dedicated file, usually named log-cleaner.log, next to the broker logs. It records which partition was cleaned, how many bytes were read and written, how many records were retained, the compression ratio, and the duration. It is the first place to look when someone asks “why is this topic still huge?”

Simple alerts worth setting

• Alert if uncleanable-partitions-count stays above zero for more than a few minutes.
• Alert if time-since-last-run-ms grows well beyond your backoff time on busy clusters.
• Warn if max-dirty-percent stays high for many hours.
• Watch free disk space on every data directory, since compaction needs working room.
• Track the size of your key compacted topics over time; a slow leak often means keys are never deleted.

Now that the big picture is clear, let us look at the fine print that decides how the cleaner treats each byte.

The record batch, the real unit on disk

Earlier we said Kafka groups records into batches. Each batch carries a small header, and the cleaner reads and rewrites that header with care. The header holds the base offset of the batch, its length, the leader epoch that wrote it, a checksum, a set of attribute bits, the timestamps, the producer identity, the first sequence number, and the record count. Inside the batch, every record stores a small offset delta relative to the base offset. That design is why offsets stay stable: when the cleaner drops a record, it does not renumber anything. It simply omits that record, and the deltas of the remaining records stay exactly as they were.

The delete horizon

Earlier we said newer Kafka versions start the tombstone countdown when the cleaner first sees it. The cleaner remembers this moment by writing a delete horizon into the batch that holds the tombstone. A flag in the batch header says that the timestamp field now carries the horizon, meaning the time after which the tombstone may be dropped. On later runs, the cleaner just compares the horizon to the current time. This is more reliable than using file modification times, because file times can change when segments are copied or rewritten for unrelated reasons.

Control records and transaction markers

Transactions add special control records that mark a transaction as committed or aborted. Readers never see them, but they exist in the log. The cleaner treats them in a careful way:

• Records from aborted transactions are removed, since no committed reader should ever see them.
• Commit and abort markers are kept for a while, following the same delete-retention idea used for tombstones, so consumers that are still reading through the region can interpret earlier batches correctly.
• Cleaning never advances past an offset that belongs to an open transaction. The lowest such offset is called the first unstable offset, and it sits at or above the last stable offset that consumers in read-committed mode are allowed to see.

Other files that survive cleaning

A partition directory holds more than segments. It also has a leader epoch checkpoint that tracks which leader wrote which offset range, and producer state snapshots that remember the latest sequence number of each producer. Compaction does not touch these files. That is why keeping the last batch of each producer, even when empty, is important: it allows the broker to rebuild producer state after a restart by scanning only recent batches.

Why the cleaner works per batch and not per record

Kafka compresses at the batch level, and the checksum covers the whole batch. Editing a record inside a batch means rebuilding the batch. So the cleaner reads a batch, decides which of its records survive, and writes a new batch containing only those records. If all records survive, it can copy the batch bytes as they are. This shortcut is one reason cleaning a log full of unique keys is cheaper per byte than cleaning a log full of duplicates.

Large messages and the cleaner buffers

The read and write buffers of the cleaner start at the size set by log.cleaner.io.buffer.size. If a single batch is bigger than the buffer, the cleaner grows the buffer up to the largest allowed message size so it can still process that batch. This means a few unusually large records can temporarily increase memory use. Keep an eye on maximum message size settings for compacted topics, and avoid stuffing large blobs into values when a reference to external storage would do.

Compaction is one of several tools. Knowing its neighbors helps you choose well.

Compaction is not a database

A compacted topic answers exactly one question well: “What is the latest record for each key, and what changed since offset X?” It cannot answer “Which users live in Pune?” without reading everything. If your reads look like queries, put a query engine or a database next to the topic and feed it from the topic. Many good architectures use the compacted topic as the source of truth for change and the database as the read-optimized view.

Compaction and tiered storage

Kafka’s tiered storage feature moves older closed segments to cheaper remote storage such as an object store. Support for compacted topics in that feature has been limited or unavailable in releases so far. Check the documentation for your version before planning to combine them, and remember that a compacted topic already stays small when the number of live keys is modest.

Compaction and Kafka Streams tables

A table in Kafka Streams is stored twice: once in a fast local store for lookups, and once in a compacted changelog topic for safety. Compaction is the safety net, not the working copy. This split shows a general lesson: use the compacted topic for durability and replay, and keep a local or external structure for fast reads.

The reader side of compaction has its own patterns. Get them right and your services start fast and stay correct.

The bootstrap-then-follow pattern

Most consumers of compacted topics do two jobs in order. First they bootstrap: read from the beginning until they reach the end of the log as it was when they started. This gives them the current state. Then they follow: keep reading new records and apply each as an update or a delete.

flowchart LR
  A["Service starts with empty local state"] --> B["Ask broker for end offsets of each partition"]
  B --> C["Read from offset zero, applying each record"]
  C --> D{"Reached the recorded end offsets?"}
  D -->|"No"| C
  D -->|"Yes"| E["Mark service as ready"]
  E --> F["Keep reading, apply updates and tombstones live"]

The end offsets are noted at the start so the consumer has a clear finish line. Without it, a busy topic could keep the consumer “bootstrapping” forever. Because compaction leaves gaps, “reached the end” must be tested by comparing the consumer’s position to the end offset, not by counting records.

Applying records correctly

• Upsert on a normal record. Put the value into the local map under the key, replacing any older value.
• Remove on a tombstone. A null value means remove the key. Do not store null as a value.
• Be idempotent. Reading the same record twice, for example after a restart, must leave the state unchanged. Upserts and removes are naturally idempotent, which is one of the beauties of this pattern.
• Use partition-aware state. Keys of one partition never overlap with another, so each partition can be loaded by a different worker in parallel.

Kafka Streams restoration

When a Kafka Streams instance starts on a fresh machine, it restores each local store by reading the matching changelog topic from the beginning. The time this takes depends on the compacted size of the topic, not on its lifetime write volume, and that is compaction at work. To cut restart delays even more, teams enable standby replicas. A standby is another instance that continuously applies the changelog to a warm copy, so failover only needs to catch up on the last few records.

Global tables and small reference data

Some data, such as country codes or product categories, is small and needed by every instance. A global table loads every partition of a compacted topic into each instance. Because compaction keeps the topic as small as the number of live keys, this stays cheap even after years of updates.

Deciding when a consumer is caught up

For read-committed consumers, the end of the readable log is the last stable offset. For plain consumers, it is the high watermark, which is the highest offset copied to all in-sync replicas. Both are reported by the broker. A service that says “ready” too early would answer requests using incomplete state, which is a classic subtle bug in cache-loading services.

Numbers make ideas real. Let us size a compacted topic step by step for an imaginary but realistic service.

Imagine an online store that keeps a compacted topic called product-catalog. It expects 50 million products, each with a value of about 2 KB, spread across 12 partitions, and a replication factor of 3. Updates arrive at a steady pace, and the team uses the default dirty ratio of 0.5.

Step 1: the live data

Step 2: worst-case log size before cleaning

At a dirty ratio of 0.5, the cleaner may wait until the dirty region equals the clean region. In the worst case, every dirty record replaces a clean one, so the log can reach twice its compacted size. For our example that is 200 GB per replica, or 600 GB for all three copies. Add the active segment and some slack for growth, plus 20 percent free space for temporary cleaning files, and you should plan roughly 720 GB of raw capacity for this topic alone.

Step 3: the offset map

In the worst case, one partition’s dirty region contains every one of its 4.17 million unique keys. At 24 bytes per key, that is about 100 MB. Dividing by the 0.9 load factor gives about 111 MB per thread. The default 128 MB buffer with a single thread would pass barely. If the team adds a second thread, each thread gets only 64 MB, and cleaning would need two runs per partition. A sensible choice is to raise the dedupe buffer to 512 MB with two threads, giving each 256 MB with a comfortable margin.

Step 4: the cleaning workload

Each run re-reads the whole partition up to the map’s limit. A partition holds about 8.3 GB of compacted data (100 GB ÷ 12) and up to about 16.7 GB when dirty. A run may therefore read around 17 GB and write about 8 GB. With a throttle of 50 MB per second, that takes several minutes. Twelve partitions spread across brokers and threads finish in parallel. This is a manageable load for modern disks, but it should be measured, not guessed.

Step 5: consumer startup time

A new consumer reads about 100 GB to bootstrap. At a sustained read rate of 200 MB per second, that takes roughly eight to nine minutes. With twelve partitions read by twelve parallel workers, each reads only around 8 GB, so bootstrap drops to about one minute. Lowering the dirty ratio would reduce it a bit further by trimming duplicates in the log.

When something looks wrong, walk through these checks in order. Most compaction problems fall into a handful of buckets.

Symptom: the topic is not shrinking

Other symptoms and likely causes

Compaction behaves differently from ordinary retention. Test it on purpose, and plan for change from day one.

How to test compaction in a staging cluster

Waiting days for default settings is not a practical test. Instead, create a test topic with aggressive settings: a very small segment size, a tiny dirty ratio, a zero compaction lag, and a short tombstone retention. Then follow this routine:

Schema evolution with long-lived records

In a delete-policy topic, records age out and old formats disappear on their own. In a compacted topic, a quiet key can keep a record for years. So every version of your value schema that has ever been produced may still exist in the log. Adopt compatibility rules that let a new consumer read every past version, and a new producer avoid breaking old readers. In schema registry terms, the transitive backward-compatible modes suit compacted topics well.

Keys are part of the contract

The cleaner compares keys as raw bytes. Changing how you serialize the key, such as adding a field to a structured key or switching from text to binary, produces different bytes for the same logical entity. The old and new records then look like different keys, and old copies remain forever. Treat the key format as frozen once a compacted topic is in production.

Migrating to a new topic

When you must change partitions, keys, or settings that cannot be changed in place, migrate. Create the new topic, copy the latest records from the old one with a small program or a replication tool, switch producers, let consumers catch up, then switch consumers and retire the old topic. Because the old topic is compacted, the copy only needs to transfer the live data, not the whole history, which makes migration much cheaper than with a delete-policy topic.

A compacted topic is a tool with a shape. Use it where the shape fits, and avoid it where it does not.

Patterns that work well

Anti-patterns to avoid

You have probably used compaction today without knowing it, because Kafka’s own brain depends on it.

Inside Kafka itself

Kafka ecosystem tools

Industry background

Kafka was created at LinkedIn, and its designers described compaction as a way to treat a log as a replicated, ordered copy of database state. Jay Kreps, one of the original authors, wrote an influential essay called “The Log” which explains why a compacted change log is a natural way to keep systems in sync. Apache Samza, another LinkedIn project, uses compacted changelog topics to make stream processing state recoverable. Today, the same idea appears anywhere that companies mirror database tables into streams: every consumer can bootstrap from the compacted topic, then follow live changes.

Pros and cons at a glance

A short field guide from the lessons teams learn the hard way.

Best practices

Common mistakes

• Forgetting keys. Producers that omit keys are rejected by compacted topics, and code that worked on a normal topic suddenly fails.
• Using empty strings as deletes. Only a true null value is a tombstone. An empty value is just another update.
• Expecting instant cleanup. Cleaning waits for the dirty ratio, the compaction lag, and the active segment to roll.
• Tuning the wrong knob. Adding threads does not help if the buffer is too small for a single segment.
• Ignoring the recopy cost. Very large logs with frequent cleaning can spend most of their effort recopying unchanged data.
• Counting records by offset. Gaps make offset arithmetic meaningless.
• Combining compact with a short retention by accident. Latest values then start disappearing.
• Restarting consumers with stale local state. After long downtime, replay from the beginning.

Quick answers to the questions people ask most often.

Everything above, in a form you can remember.

Log compaction turns a Kafka topic from a history book into a living table. A small group of cleaner threads inside each broker scan the older, closed part of every compacted log. They build a compact memory of the newest offset per key, then copy the log while dropping everything that has been replaced. Offsets never move, order never changes, and consumers keep working as usual. Deleted keys travel as tombstones that stay for a grace period so slow readers can learn of them, then quietly vanish.