To understand retention, we first need to see what makes Kafka different from many older messaging tools.

The old way: read it and it disappears

In many traditional message queues, a message is removed as soon as one consumer reads it. The queue is like a to-do tray: a note is picked up, done, and thrown away. The system never needs to decide how long to keep messages, because a message lives only until someone takes it.

The Kafka way: keep it, and let everyone read it

Apache Kafka stores streams of events, such as orders, payments, clicks, or sensor readings, in topics. Each topic is split into partitions, and each partition is an ordered log on disk. When a consumer reads an event, the event is not removed. It stays in the partition, so that other consumers can read it too, and so that the same consumer can read it again later. This is what allows replay, recovery, and many independent readers.

The question this raises

If reading does not delete anything, then something else must. Disks are finite, and events arrive endlessly. Kafka needs a rule for when old events may be removed. That rule is retention.

A retention period is the amount of time Kafka promises to keep events before it becomes allowed to delete them.

The simple definition

The retention period is a setting on a topic that says how long its events should be kept. By default, Kafka keeps events for seven days. After that, events become eligible for deletion. You can shorten this to hours, or lengthen it to weeks, months, years, or even keep events forever.

Retention is a minimum promise, not an exact deadline

The retention period tells you the earliest time when Kafka may delete an event, not the exact moment it will. Deletion happens in chunks, checked periodically, so an event may live a little longer than the period. What you can rely on is that an event will not be deleted before its retention period has passed, unless another limit, such as a size limit, forces earlier deletion.

Three related ideas

To understand how deletion really works, we must look at how a partition is stored on disk.

A partition is made of segment files

A partition is not one gigantic file. It is split into smaller files called segments. At any moment, only the newest segment, called the active segment, receives new events. When it grows large enough or becomes old enough, Kafka closes it and starts a new active segment. Closed segments are never changed again.

Deletion works on whole segments

Kafka does not remove individual events one by one. It deletes entire segment files. A closed segment becomes eligible for deletion only when every event in it has passed the retention limit. In practice, Kafka looks at the newest timestamp in the segment: if even the newest event is older than the retention period, then all the events in that segment are old enough, and the whole file can go.

flowchart LR
  S1["Segment 1 - closed - all events older than retention"] --> S2["Segment 2 - closed - some events still young"]
  S2 --> S3["Segment 3 - closed - young"]
  S3 --> S4["Segment 4 - active - receiving new events"]
  S1 -.->|"Deleted at next check"| X["Removed from disk"]
        

A surprising fact confuses many beginners: old events can outlive the retention period because of the segment they sit in.

The active segment stays

Kafka never deletes the active segment, because it is still being written. Even if every event in it is older than the retention period, it remains until it rolls and becomes a closed segment. Then it can be deleted at a later check.

A slow topic keeps old events longer

Consider a topic that receives very few events. Its active segment may take weeks or months to fill up. Until it rolls, everything in it stays, even events far older than the retention period. To prevent this, Kafka also rolls a segment when it becomes older than a chosen segment age, by default seven days. Setting this age smaller than the retention period makes deletion more predictable for slow topics.

A busy topic rolls quickly

On a busy topic, segments fill quickly and roll often, so old data leaves close to the retention period. On a quiet topic, data may linger much longer, which surprises teams who expected exact deadlines.

Kafka offers two limits, and whichever is reached first triggers deletion.

Time-based retention

You choose how long to keep events, such as three days or thirty days. Closed segments whose newest event is older than that are deleted. This is the most common form and is easy to reason about: “we can replay the last seven days.”

Size-based retention

You can also limit how large each partition may grow. When a partition exceeds the limit, Kafka deletes the oldest closed segments until it is back under the limit. Note that the limit applies to each partition separately, not to the whole topic, so a topic with twelve partitions may hold twelve times the limit in total. The default is no size limit at all.

Both together

If you set both, Kafka deletes as soon as either limit is exceeded. Time-based limits protect against keeping data too long. Size-based limits protect the disk against sudden bursts of traffic that would fill it long before the time limit arrives.

flowchart TD
  C["Broker checks a closed segment"] --> Q1{"Is the newest event older than the retention time?"}
  Q1 -->|"Yes"| D["Delete the segment"]
  Q1 -->|"No"| Q2{"Is the partition over its size limit?"}
  Q2 -->|"Yes"| D
  Q2 -->|"No"| K["Keep the segment"]
        

Age needs a clock. Kafka has to decide which time to use.

The record’s timestamp

Every event carries a timestamp. Kafka looks at the timestamps inside a segment to decide whether it is old enough to delete. There are two ways a timestamp can be set. In create time, the producer sets the timestamp when it creates the event. In log append time, the broker sets the timestamp at the moment it stores the event.

Why the choice matters

Sometimes the age of an event matters less than whether it is the latest one for its key.

The idea

With log compaction, Kafka keeps at least the latest event for every key, however old it is, and may remove older events with the same key. The topic then behaves like a table of current values. A new application can read the topic from the beginning and rebuild the current state of everything, without replaying every historical change.

An example

Suppose a topic stores each customer’s current address, keyed by customer number. A customer moves three times, producing three events with the same key. After compaction, only the newest address remains for that customer. Customers who never move keep their single event indefinitely.

Deleting with a tombstone

To remove a key completely, a producer sends an event with that key and an empty value, called a tombstone. Compaction eventually removes earlier events for that key, and later removes the tombstone itself after a grace period, by default one day, so readers have time to see the deletion.

Delete, compact, or both

flowchart LR
  A["Before: key A=1, key B=1, key A=2, key A=3, key B=2"] --> B["Compaction"]
  B --> C["After: key A=3, key B=2"]
        

Retention affects readers directly. A consumer that falls too far behind can lose events forever.

The race against deletion

A consumer reads events at some position, called its offset. If the consumer is slow, stopped for maintenance, or broken for a long weekend, it falls behind the newest events. Meanwhile, Kafka keeps deleting old segments. If the consumer’s position falls behind the earliest event that still exists, the events it wanted to read are gone.

What happens when the position is gone

When the consumer returns and asks for an event that has been deleted, Kafka answers that the requested position is out of range. The consumer must then decide where to start instead, according to its reset setting: from the earliest event that still exists, or from the newest event only. Starting from the earliest means it processes everything still available, but it has permanently missed what was deleted. Starting from the newest means it skips all the events it had not yet read.

flowchart LR
  A["Earliest available offset moves forward as old segments are deleted"] --> B["Consumer position falls behind it"]
  B --> C["Position is out of range"]
  C --> D{"Reset rule"}
  D -->|"Earliest"| E["Restart from oldest remaining event; deleted events are missed"]
  D -->|"Latest"| F["Skip to the newest event; unread events are missed"]
        

The consumer’s own progress is stored in Kafka too, and it also has a retention rule.

Where bookmarks live

The committed offsets of consumer groups are stored in a special internal Kafka topic. Like other data, they are not kept forever. When a group has no active members, its stored bookmarks are kept for a limited time, by default seven days, and then removed.

What this means in practice

If a consumer group is stopped for longer than this period, its bookmarks vanish. When the group returns, Kafka has no record of where it stopped, and the group starts according to its reset rule, either from the earliest available event or from the newest. This can cause a large amount of reprocessing or a large gap, even though the events themselves might still exist on disk.

Every extra day of retention is paid for in disk space, on every copy of every partition.

The simple formula in words

The disk space a topic needs is roughly the volume of data per day, multiplied by the number of days kept, multiplied by the replication factor, which is the number of copies. That total is then spread over all brokers, with extra room for growth and for safety.

An example in words

Suppose a topic receives fifty gigabytes of events per day and keeps them for seven days, with three copies. That is fifty times seven times three, or one thousand and fifty gigabytes, just over a terabyte, for this one topic. Extending the retention to thirty days multiplies the requirement by more than four, to about four and a half terabytes. Adding compression on the producer side can reduce all of these numbers, sometimes by a large margin.

Some businesses want to keep events for months or years. Kafka has ways to make this affordable.

Why keep data for so long?

The cost problem

Keeping everything on the brokers’ local fast disks is expensive, because you pay for fast storage across every replica, and you must add brokers simply to hold old data that is rarely read.

Tiered storage

Tiered storage splits the log into two layers. Recent data stays on the brokers’ local disks for fast access. Older data is moved to cheaper, larger storage, such as cloud object storage, while remaining readable through Kafka. You then set two limits: a shorter one for how long data stays on local disks, and a longer one for how long it is kept overall. Consumers can still read old data through the normal interface, though it is slower to fetch than recent data. This feature is available in current Kafka releases, and its exact status and maturity should be checked for the version and service you use.

flowchart LR
  N["New events"] --> L["Local broker disks: recent data, fast"]
  L -->|"After local retention"| R["Remote storage: older data, cheaper"]
  R -->|"After total retention"| D["Deleted"]
  C["Consumers"] --> L
  C --> R
        

There is no universal right answer. A few questions lead you to a sensible choice.

Retention is not only a technical setting. It is often a legal and ethical one.

Keeping too long is a risk

Many privacy rules say that personal data should not be kept longer than necessary. Events containing customer details that sit in Kafka for years, and in every backup and replica, can become a liability. A short, well-chosen retention period is a simple, automatic way of respecting such limits.

Deleting on request

When a person asks for their data to be erased, Kafka’s append-only nature makes this awkward. For compacted topics, a tombstone for that person’s key eventually removes their earlier events. For ordinary topics, the events stay until retention removes them, so short retention, or keeping personal data out of events, becomes important. Some organisations encrypt personal fields with a per-person key and delete the key when erasure is requested, making the old events unreadable.

Keeping long enough

Other rules require records to be kept for a minimum time, such as financial transactions. In such cases, retention must not be shorter than the required period, or a separate long-term archive must hold the records.

A few signals show whether retention is healthy or heading for trouble.

What to watch

Retention is a bargain between memory and cost.

Most retention troubles come from the same few mistakes.

Best practices at a glance

Let us see how an online shop might design retention for its main topics.

flowchart LR
  CL["Clicks topic - 2 days"] --> AN["Analytics consumers"]
  OR["Orders topic - 14 days"] --> SV["Order and email services"]
  PY["Payments topic - 14 days"] --> SV
  PR["Profiles topic - compacted"] --> NS["New services load latest state"]
  DL["Dead letter topic - 30 days"] --> EN["Engineers investigate"]
  OR --> AR["Connector copies to long-term archive"]
  PY --> AR
        

It is the length of time Kafka keeps the events of a topic before they become eligible for deletion, regardless of whether anyone has read them.

Seven days for ordinary topics, unless the cluster or the topic has been configured differently. Always check the actual setting of your topic.

No. Reading does not delete anything. Events stay until retention removes them, which is why many consumers can read the same data and why replay is possible.

Deletion works on whole closed segments and runs on a schedule. If the segment has not rolled yet, especially on a quiet topic, or its newest event is still young, older events stay with it.

Yes, by turning off time and size limits, or by using compaction or tiered storage. But storage grows continuously, so plan for cost, recovery time, and privacy obligations.

The events it had not read are deleted, and its saved position becomes invalid. It must restart from the earliest available or the newest event according to its reset rule, and the deleted events are lost for it.

No. It applies to each partition separately, so the total size of a topic can be the limit multiplied by the number of partitions.

Deletion removes events older than a time or beyond a size, regardless of key. Compaction keeps at least the latest event for every key and removes older events with the same key.

Yes. The committed offsets of a group with no active members are kept for a limited time, by default seven days, and then removed. A group that stays idle longer starts again according to its reset rule.

Yes. Retention is a topic-level setting, and different topics should have different values according to their purposes.

We began with a library that must decide how long to keep newspapers before recycling them, and whose readers must know the rule. Kafka’s retention period is that rule. Unlike a traditional queue, Kafka does not delete events when they are read. It keeps them in segment files on disk and removes whole closed segments only when their events are older than the time limit or when the partition exceeds its size limit, checking on a schedule, never touching the active segment, and judging age by timestamps. Compaction offers a different rule: keep the latest event per key. Retention shapes everything around it: how long a slow consumer can be away before it loses events, how long consumer bookmarks survive, how much disk and money the cluster needs, how long personal data is exposed, and how deep your replay window is. Choose retention topic by topic, make it longer than your worst realistic outage, calculate the storage honestly, use compaction and tiered storage where they fit, and watch disk, lag, and earliest available offsets. Done well, retention is a quiet promise that keeps every reader safe.