Log retention monitoring is the practice of continuously checking that Kafka keeps data for as long as you promised, and no longer than your disks can afford.

Start with the building blocks. A message is a small record of something that happened, such as “order 42 was shipped”. A topic is a named collection of messages, stored in order. Each topic is divided into partitions, and each partition is stored on disk as a log, a file-based sequence of messages that only grows at the end. Kafka does not delete a message when a consumer reads it. Instead, it keeps messages until a retention policy says they are old enough or the log is big enough to remove them.

Log retention monitoring is everything you do to make sure this system behaves as intended. It includes watching disk space, watching how big each log is, watching how old the oldest message is, checking that retention settings have not been changed by mistake, and making sure consumers finish reading data before it is deleted.

Three questions the monitoring answers

Retention failures are quiet. Nothing crashes at first. The damage shows up later as missing data or a dead broker.

Most Kafka problems announce themselves loudly: a broker goes down, a consumer throws an error, a dashboard turns red. Retention problems are different. When data is deleted early, the deletion is a normal, healthy action. Nobody gets an error. The gap appears only when someone tries to read the missing data, which may be days or weeks later. Similarly, a disk filling up looks perfectly fine right up to the moment it is full.

The four things at stake

To watch retention you must first understand what is being deleted. Kafka deletes files, not individual messages.

Each partition of a topic is stored as a directory on a broker’s disk. Inside that directory, the log is split into segments. A segment is a set of files that holds a slice of the partition’s messages. Only one segment at a time, the newest, accepts new messages. This is called the active segment. When the active segment becomes big enough or old enough, Kafka closes it and opens a new one. Closed segments are never changed again, except when they are deleted or compacted.

flowchart LR
  T["Topic orders"] --> P0["Partition 0"]
  T --> P1["Partition 1"]
  P0 --> S1["Segment offsets 0 to 9999 closed"]
  P0 --> S2["Segment offsets 10000 to 19999 closed"]
  P0 --> S3["Segment offsets 20000 onward active"]
  S1 --> F1["Log file and index files"]
  S2 --> F2["Log file and index files"]
  S3 --> F3["Log file and index files"]

The files in a segment

Offsets you should know by name

Kafka has two cleanup policies. One removes old data by age or size. The other keeps the latest value per key.

Every topic has a setting called the cleanup policy. It decides what Kafka does with old data. There are two basic choices, and they can be combined.

Delete policy in plain words

With the delete policy, Kafka periodically checks each partition and removes closed segments that are too old or when the total size is too large. This is the default and the most common policy. It suits streams of events, where the value lies in the recent past.

Compact policy in plain words

With compaction, Kafka rewrites closed segments so that only the newest message for each key remains. The aim is not to limit age but to keep a full, up-to-date snapshot of the state while shrinking the log. A special message with a key and an empty value, called a tombstone, tells Kafka to remove that key entirely after a delay.

The most common rule: keep messages for a fixed length of time, then let them go.

Time-based retention says, “keep every message for this long”. At the broker level it is controlled by settings named log.retention.hours, log.retention.minutes and log.retention.ms, where the smaller unit wins if several are set. The default is one hundred and sixty-eight hours, which is seven days. Each topic can override the broker default with its own retention.ms setting. Setting it to minus one means keep forever.

How Kafka decides a message is old

Kafka does not look at each message separately. It looks at each closed segment and asks: what is the newest timestamp of any message in this segment? If even the newest message in the segment is older than the retention time, the whole segment is eligible for deletion. This detail is very important. A segment full of old messages stays on disk as long as it contains a single message that is still young enough.

The second rule: keep data only until the log reaches a maximum size, whatever its age.

Size-based retention is controlled by log.retention.bytes at the broker level and retention.bytes for a topic. It sets the maximum size of a single partition’s log. When the log grows above this limit, Kafka deletes the oldest closed segments until the size is back under the limit. The default is minus one, meaning no size limit at all.

Two facts trip people up. First, the limit applies to each partition, not to the whole topic. A topic with twelve partitions and a limit of ten gigabytes per partition can hold up to a hundred and twenty gigabytes. Second, the limit counts one copy of the data. With three replicas, the total disk use across the cluster is three times larger.

When both limits are set

If a topic has both a time limit and a size limit, whichever is reached first triggers deletion. A topic set to seven days and fifty gigabytes per partition keeps data for seven days during quiet periods, but during a traffic spike it may keep data for only a day, because the size limit is hit first. Monitoring must therefore track the actual age of the oldest data, not just the configured time.

Deletion is a background job with several steps, and each step has a delay that monitoring must allow for.

flowchart TD
  A["Retention check runs every few minutes"] --> B{"Is a closed segment past the time limit or is log over size limit"}
  B -->|"No"| C["Keep segment and wait for next check"]
  B -->|"Yes"| D["Advance log start offset"]
  D --> E["Rename segment files with deleted suffix"]
  E --> F["Wait for file delete delay"]
  F --> G["Remove files and free disk space"]

What this means for monitoring

A low-traffic topic with a one-day retention can keep data for a week. The reason is the active segment.

Kafka never deletes the active segment, the one currently receiving writes. It closes a segment and starts a new one when the segment reaches its maximum size, controlled by segment.bytes with a default of one gibibyte, or its maximum age, controlled by segment.ms with a default of seven days. Only after a segment is closed does it become eligible for retention.

Now picture a topic that receives only ten megabytes a day. The active segment will take about a hundred days to reach one gibibyte, so the size trigger never fires. The age trigger of seven days fires instead. Suppose the retention is set to one day. Data written on Monday sits in the active segment. The segment closes the following Monday. From that moment, the retention check finds it eligible, but the newest message in it is from that Monday, so it must wait another day. In the end, Monday’s data survives for about eight days, not one.

Retention is measured with message timestamps. If those are wrong, retention is wrong.

Every Kafka message carries a timestamp. By default the timestamp is set by the producer when it creates the message, a mode called CreateTime. A topic can instead be set to LogAppendTime, where the broker stamps the message when it stores it. Time-based retention uses these timestamps to decide the age of a segment.

Protecting yourself

Use the broker-stamped mode, LogAppendTime, for topics where retention must be predictable. Where producer timestamps must be kept, limit how far a timestamp may differ from the broker’s clock, using the topic-level settings that reject or adjust messages with extreme timestamps. Monitor the gap between the newest timestamp in a topic and the current time; a large positive gap is a warning sign of a producer with a bad clock.

Compacted topics do not expire by age, so their health depends on a background cleaner that must be watched.

On a compacted topic, a background pool of cleaner threads rewrites closed segments. The cleaner picks the partitions with the highest proportion of duplicate keys, called the dirty ratio, once it passes a threshold set by min.cleanable.dirty.ratio, default one half. It copies only the newest message for each key into new segments and discards the rest.

What can go wrong

Kafka keeps track of where each consumer group stopped reading, and that record has its own retention rule.

Consumers save their position in an internal compacted topic named __consumer_offsets. When a consumer group has no active members, Kafka starts a countdown. After the time set by offsets.retention.minutes, which is seven days by default in recent versions, the saved positions of that group are deleted. If the group returns after that, it no longer knows where it stopped and falls back to its reset policy.

The reset policy is a client setting called auto.offset.reset. With “earliest”, the group starts from the oldest available message and may reprocess a great deal. With “latest”, it starts from the newest and skips everything in between. With “none”, it stops with an error. None of these is good if the group merely paused for a long weekend and lost its place.

Tiered storage splits retention into two parts: a short stay on fast local disks and a long stay in cheap remote storage.

In recent versions of Kafka, a feature called tiered storage lets a broker copy closed segments to a remote store, such as object storage, and later remove them from local disk. The total retention still applies to the data as a whole, but a second, shorter local retention says how long data stays on the broker’s own disks. Consumers reading recent data are served from local disk, and consumers reading old data are served from the remote store, more slowly.

A small catalog of signals covers almost every retention risk. Learn these first and add others only when you need them.

Group them into four families

Metrics travel from brokers and hosts to a time-series store, then to dashboards and alerts.

flowchart LR
  B["Kafka Brokers"] --> JX["JMX Exporter Agent"]
  H["Host Metrics Agent for Disk and CPU"] --> TS["Time Series Database"]
  JX --> TS
  CG["Consumer Lag Exporter"] --> TS
  CF["Config Checker Job using Admin API"] --> TS
  TS --> GR["Dashboards"]
  TS --> AM["Alert Manager"]
  AM --> ON["On Call Pager and Chat"]
  GR --> TM["Team Review"]
  LB["Load Balancer and API Gateway for Dashboards"] --> GR

The pieces in plain words

Disk space is the first and most important signal. Watch both the level and the speed of change.

Disk usage on a broker is the sum of all partition logs it hosts, plus indexes, plus a small amount of other data. What matters for alerting is not just the percentage used, but how quickly it is changing. A disk at seventy percent that is growing by one percent an hour has about thirty hours left. A disk at seventy percent that is flat is comfortable.

Why a disk should never be allowed to fill

A full disk is not a graceful failure. When a broker cannot write, it may mark the log directory as offline. If that was its only directory, the broker shuts down, and all its partitions must move to other brokers. Those brokers now receive extra work and extra data, and may themselves run short of space. What began as one full disk can spread through the cluster.

Broker metrics show the size and shape of every log, which lets you find the topics that consume the space.

Kafka brokers publish per-partition measurements, including the size of the log in bytes, the log start offset, the log end offset and the number of segments. Summing size across a topic’s partitions, and multiplying by the replication factor, tells you the true cost of a topic on disk.

Reading the shape

The most important retention question: can every consumer read its data before Kafka deletes it?

Consumer lag is the distance between the newest message and the position of a consumer. It can be measured in messages, but for retention purposes it is much more useful in time: how old is the message the consumer is about to read? If that age approaches the retention time, the consumer is about to lose data.

flowchart LR
  OLD["Oldest stored message: log start offset"] --> RISK["Danger zone: consumer position near here"]
  RISK --> POS["Consumer position"]
  POS --> LAG["Lag: messages not yet read"]
  LAG --> NEW["Newest message: log end offset"]

The number to watch: time to expiry

A helpful measure is the time to expiry: how long until the message at the consumer’s position will be deleted. It is roughly the retention time minus the age of that message. If retention is seven days and the consumer is reading a message that is five days old, only two days remain. If lag is growing, the time to expiry is shrinking, and the alert should fire well before it reaches zero.

Retention settings live in several layers. Without checks, they drift away from what you intended.

A topic’s effective retention comes from layers. The broker file sets a default. A topic-level override, set through the admin tools, replaces the default for one topic. A dynamic broker setting can change the cluster default while brokers are running. Different brokers can even hold different files if they were configured by hand. Because the layers are invisible to most people, the value that actually applies is often not the value that people believe applies.

A simple audit routine

Good alerts are few, clear and actionable. Bad alerts teach people to ignore the pager.

Rules for good alerts

A simple formula turns your traffic and retention promise into the disk space you must buy.

The disk space needed for a topic is the product of four things: how fast data arrives, how long you keep it, how many copies you store, and a safety margin. In words: bytes per second, times seconds of retention, times replication factor, times a margin. Compression changes the number of bytes actually stored, so use the compressed rate measured on the broker, not the size of the raw messages.

A worked example

An online retailer has a topic that receives fifty megabytes per second of compressed data at peak. It promises seven days of retention and uses three replicas. In one day there are 86,400 seconds, so one day of data is 50 times 86,400, which equals 4,320,000 megabytes, or about 4.3 terabytes. Seven days is about 30.2 terabytes for one copy. With three replicas, the cluster stores about 90.7 terabytes. Adding a thirty percent margin gives roughly 118 terabytes across all brokers.

Follow a typical incident from the first sign of trouble to the recovery, and see which signals would have warned earlier.

flowchart TD
  A["Traffic spike doubles bytes in per second"] --> B["Retention deletes at normal pace"]
  B --> C["Disk usage climbs past eighty five percent"]
  C --> D{"Was time to full alert configured"}
  D -->|"Yes"| E["Team acts early: adds size limit or capacity"]
  D -->|"No"| F["Disk reaches one hundred percent"]
  F --> G["Log directory goes offline and broker stops"]
  G --> H["Partitions fail over to other brokers"]
  H --> I["Neighbor brokers take extra load and may also fill"]

What the responders should do

The opposite problem: retention worked perfectly, and that is exactly what hurt.

A nightly job reads yesterday’s data from a topic and loads it into a warehouse. One night it fails because the warehouse is down. The team fixes the warehouse two days later and restarts the job, expecting it to pick up where it left off. But the topic retention is only twenty-four hours, and the job’s saved position now points to deleted data. The job resets to the earliest available message, the oldest of which is only a day old. Two days of events are missing from the warehouse.

Lessons

Retention changes are quick to make and impossible to undo once data is deleted. Follow a routine.

Retention can be changed while brokers are running, using the topic configuration tool. Increasing retention is safe and takes effect immediately: data that would have been deleted simply stays. Decreasing retention is the risky direction. At the next retention check, Kafka may delete a large amount of data in one go, and there is no undo.

A good retention dashboard answers the three questions of chapter one in one glance.

Design guidelines

Monitoring costs effort and money, and saves far more. Still, every choice trades one thing for another.

The main trade-offs

Watching a big cluster generates a lot of data. Keep the monitoring itself light and useful.

A cluster with thousands of partitions can produce hundreds of thousands of metric series. Collecting every one of them every few seconds is costly, both in network and in storage of the metrics system. The aim is to keep the signals that drive decisions and to summarize the rest.

Retention has its own reliability concerns, and so does the system that watches it.

Watch replicas, not just leaders

Retention runs on every replica of a partition, so leaders and followers may hold slightly different amounts of data at any moment. Large, lasting differences point to a problem such as a follower that has fallen out of sync, a disk that is failing or a broker that was configured differently. Compare the log start offsets and sizes across replicas of the same partition and investigate anything that does not converge.

Whoever can change retention can destroy data or fill disks. Control that power.

When the pager rings, a short table of symptoms and first checks saves precious minutes.

A few habits keep retention healthy. A few shortcuts create the incidents in this guide.

Patterns that work

Anti-patterns to avoid

A short checklist gathered from production clusters. Follow it and most retention problems never happen.

Mistakes seen again and again

Retention monitoring appears wherever Kafka carries data that matters. These examples show the pattern.

Kafka began at LinkedIn to move activity data, and today it carries huge volumes at many companies. Public talks and blog posts describe how operators manage disks, retention and lag at scale. Internal details differ and change, so treat the following as illustrations of the technique rather than exact blueprints.

A quick reference for the vocabulary used throughout this guide.

Quick answers to the questions that come up most when teams start watching retention.

Retention is a promise about time and space. Monitoring is how you keep that promise.