What Are Log Segments in Kafka?
A partition's log looks like one continuous, endless file, but underneath it is quietly cut into smaller, manageable pieces. Here is why that small trick matters so much.
Imagine a librarian tasked with keeping a single, giant diary of every event happening in a town, going back decades, with no end in sight. Rather than writing everything into one impossibly huge, unwieldy notebook, a wise librarian would instead use a new, smaller notebook for each year, neatly labeled and shelved in order. Old notebooks could be archived or discarded once no longer needed, without ever touching this year’s notebook. Kafka faces exactly this problem with every partition it stores, and its solution is called a log segment.
1The Big Picture: What Is a Log Segment?
A log segment is simply one smaller file that makes up a portion of a partition’s overall log, rather than the whole thing being stored as one giant file.
Every Kafka partition is conceptually a single, ordered, ever-growing log of events. But storing that entire log as one continuously growing file on disk would eventually become slow and unwieldy to manage. Instead, Kafka quietly splits each partition’s log into a series of smaller files, called segments, stored one after another in order, together representing the complete history of that partition.
At any given time, only the newest segment, called the active segment, is actively being written to. Every other segment for that partition is considered closed, meaning it will never receive new data again, though it can still be read from freely by any consumer that needs it.
Think of a thick, ongoing series of numbered diaries kept by a lighthouse keeper, one volume per few months. Only the current volume is being actively written in day to day. Every earlier volume is finished, sealed, and shelved, but can still be pulled down and read at any time.
2Why Kafka Splits Logs Into Segments at All
A single, endlessly growing file creates real, practical problems that segmentation quietly solves.
Imagine a busy Kafka partition receiving millions of events every day, all stored in one single, ever-growing file. Deleting old data that has passed its retention period would mean rewriting or carefully trimming a massive file, an expensive and risky operation. Searching for a specific point in that file would also become slower and slower as it kept growing without bound.
By splitting the log into smaller segments instead, Kafka can simply delete an entire old segment file once it is no longer needed, a fast, simple, and safe operation, rather than editing a giant file in place. It can also search more efficiently, since it only needs to look within the relevant segment rather than scanning through the partition’s entire history.
Fast, Safe Deletion
An entire expired segment can simply be deleted as a whole file, rather than editing a massive file in place.
Efficient Searching
Finding a specific offset only requires looking within the relevant segment, not scanning the entire partition history.
Manageable File Sizes
Individual files stay a reasonable, predictable size, easier for the operating system to handle efficiently.
Safer Operations Overall
Operations like compaction and cleanup can work on one closed segment at a time, without disrupting the active one.
3Core Vocabulary Around Log Segments
A handful of terms describe how segments are structured and managed. Learning them together makes the rest of this guide easier to follow.
Segment
One individual file holding a contiguous portion of a partition’s ordered log.
Active Segment
The single, most recent segment currently receiving new records for a partition.
Closed Segment
Any earlier segment that is no longer written to, but can still be read from freely.
Segment Roll
The moment Kafka closes the current active segment and starts a brand-new one to receive future writes.
Base Offset
The offset of the very first record stored in a given segment, also commonly used in the segment’s file name.
Index File
A companion file stored alongside each segment, mapping offsets to their exact physical location within the segment for fast lookups.
Retention
The rule deciding how long a segment, and the data within it, is kept before becoming eligible for deletion.
The index file is like the table of contents printed at the front of each diary volume, letting the lighthouse keeper flip almost instantly to a specific date instead of paging through the entire volume one page at a time.
4How Segments Are Structured Inside a Partition
Seen from the outside, a partition looks like one seamless, ordered log. Underneath, it is actually a tidy sequence of segment files.
Each segment is named and ordered using the base offset of the very first record it contains. This means, just by looking at a segment’s name, Kafka, or a curious engineer, can immediately tell roughly which range of offsets that segment covers, without needing to open and scan the file itself.
flowchart LR
S1["Segment 1
offsets 0–4999
closed"] --> S2["Segment 2
offsets 5000–9999
closed"]
S2 --> S3["Segment 3
offsets 10000–
active"]
Alongside each segment’s actual data file, Kafka also maintains a small index file, which maps specific offsets to their exact byte position within the segment. This lets Kafka jump almost directly to a requested offset, rather than reading through a segment from the very beginning every single time.
Because index files are compact and efficient, they let Kafka serve a read request starting from any specific offset extremely quickly, even within a segment holding many thousands of records.
5When Does a New Segment Get Created?
Kafka does not wait for a segment to grow indefinitely; it closes the current one and starts fresh once certain conditions are met.
This transition, called a segment roll, typically happens when one of two configured limits is reached: the active segment grows past a certain maximum size, or a certain amount of time has passed since it was first created. Whichever limit is hit first triggers the roll, closing the current segment permanently and opening a brand-new active segment to receive all future writes.
The active segment keeps growing
New records are continuously appended to it as they arrive for the partition.
A configured limit is reached
Either its maximum size or its maximum age, whichever comes first, is hit.
The segment is closed
It becomes permanently read-only, never to receive new records again.
A new active segment begins
A fresh segment file is created, ready to receive all subsequent writes to the partition.
Assuming segment size and segment age settings do nothing meaningful and leaving them at defaults regardless of actual traffic patterns. A topic with very low traffic may roll segments purely based on age, while a very high-traffic topic may roll based on size well before that time limit is ever reached.
6How Retention Actually Deletes Data
Segments are the actual unit that Kafka’s retention policy works with, not individual records.
When a topic’s retention policy decides that data older than a certain age, or beyond a certain total size, should be removed, Kafka does not go through and delete individual records one at a time. Instead, it simply deletes entire closed segment files whose data has fully aged out, or whose removal is needed to bring the partition back under its configured size limit.
flowchart LR
S1["Segment 1
expired"] -->|deleted whole| X["Removed"]
S2["Segment 2
still within retention"] --> Keep["Kept"]
S3["Segment 3
active"] --> Keep2["Kept"]
This is precisely why segmentation matters so much for performance: deleting one whole file is a fast, simple, and safe operation for the operating system, whereas trying to carefully trim old data out of the front of one gigantic, continuously growing file would be far slower and riskier.
Because retention works on whole segments, the very oldest data in a segment might occasionally be kept slightly longer than the configured retention period, simply because the rest of that same segment has not yet fully expired.
7Log Compaction and Segments
Some topics use a different cleanup approach entirely, called compaction, and segments play a central role there too.
Instead of deleting data purely based on age, a compacted topic keeps only the most recent record for each unique key, discarding older, outdated versions. This is useful for data representing current state, such as “the latest known address for customer 88,” where only the newest value genuinely matters.
Compaction works segment by segment: Kafka examines each closed segment, identifies records whose key has since been superseded by a newer record elsewhere in the log, and rewrites the segment without those now-outdated entries. The active segment is left untouched during this process, since it is still being actively written to.
Why Compaction Needs Segments
Working on individual, closed segment files lets Kafka safely rewrite and clean up historical data without ever disrupting the currently active segment, keeping ongoing writes completely unaffected by the cleanup happening on older data.
A topic can technically use both time-based retention and compaction together, though most topics are designed around one strategy or the other, depending on whether full history or only current state actually matters.
8Segments and Reading Performance
Segmentation is not just about cleanup; it directly shapes how quickly Kafka can serve a read request for any given offset.
When a consumer asks for records starting at a particular offset, Kafka first uses the segment file names to quickly identify which specific segment must contain that offset, without needing to check every segment in the partition. It then consults that segment’s index file to jump almost directly to the right position within the file, rather than reading from the very beginning.
A read request arrives for a specific offset
A consumer asks Kafka for records starting from a particular point in the partition.
The correct segment is identified
Based on the requested offset and each segment’s known base offset, Kafka finds the one segment that must contain it.
The index file narrows the search further
Kafka consults that segment’s index to find approximately where in the file this offset lives.
Records are read and returned
Kafka reads forward from that position, delivering records to the consumer efficiently.
This is very much like finding a specific date in a multi-volume diary series: first you pick the correct year’s volume off the shelf using the labels on the spines, then you use that volume’s own table of contents to flip almost directly to the right page, rather than reading from page one.
9Segments and Disk Usage
Because segments are the true unit of storage cleanup, understanding them helps explain some of the storage behavior teams actually observe in practice.
| Observation | Explanation Through Segments |
|---|---|
| Disk usage does not shrink immediately when retention expires | Kafka waits for an entire segment to fully age out before deleting it as a whole file |
| A low-traffic topic keeps oddly large or few segment files | Segments roll based on age even with light traffic, or size limits are rarely reached |
| Compacted topics still show some duplicate keys temporarily | Compaction runs periodically per segment, not instantly the moment a newer value is written |
10Real-World and Industry Examples
Log segmentation quietly underpins nearly every serious, high-volume Kafka deployment, even though it rarely gets discussed directly.
High-Volume Clickstream Topics
Busy topics roll new segments frequently based on size, letting Kafka clean up yesterday’s data efficiently while today’s stream keeps flowing without interruption.
Compacted “Latest State” Topics
Topics tracking current account balances or user profiles rely on segment-based compaction to keep only the newest value per key over time.
Long-Retention Audit Logs
Topics required to retain data for months or years accumulate many closed segments, expiring and being deleted gradually, one whole file at a time, as retention allows.
Low-Traffic Configuration Topics
Lightly used topics may roll segments primarily based on age rather than size, since their traffic rarely fills a segment before the time limit is reached.
11Advantages, Disadvantages and Trade-offs
Segmentation is a quiet but deliberate engineering trade-off, and understanding it clarifies why Kafka behaves the way it does.
Advantages
- Old data can be deleted quickly and safely, one whole file at a time.
- Reads can jump almost directly to a requested offset using index files.
- Compaction can clean up historical data without disturbing active writes.
- File sizes stay predictable and manageable for the operating system.
Disadvantages / Trade-offs
- Very old data within a not-yet-expired segment can linger slightly past its configured retention.
- Poorly tuned segment size or age settings can lead to too many small files or too few large ones.
- Extra index files add a small amount of additional storage and management overhead.
- Understanding segment behavior adds a layer of internal complexity beginners must eventually learn.
12Security Considerations
Segments are ordinary files on disk, which means normal file-level security principles apply directly to them.
Disk-Level Encryption
Since segments are stored as real files, encrypting the underlying disk protects every segment automatically, including older, closed ones.
File System Permissions
Only the Kafka broker process itself should have direct access to the raw segment files stored on disk.
Retention as a Privacy Tool
A shorter, well-considered retention period naturally limits how long sensitive data actually remains present in any segment on disk.
Simply “deleting” a record from an application’s perspective does not remove it from a segment already written to disk. Genuine removal only happens once the entire containing segment is deleted by retention, or rewritten through compaction.
13Monitoring Segment Behavior
Segment-related metrics reveal a surprising amount about a partition’s real, underlying health.
Segment Count per Partition
An unusually high number of segments can indicate the roll settings need retuning for that topic’s actual traffic pattern.
Disk Usage per Partition
Directly reflects how many segments are currently being retained, and helps forecast future storage needs.
Segment Roll Frequency
How often new segments are being created, useful for understanding a topic’s real write volume over time.
Log Cleaner Activity
For compacted topics, how actively and successfully old, superseded records are being cleaned up from closed segments.
An unexpectedly large number of very small segment files across a topic often points to a segment size or time setting that no longer matches how that topic is actually being used today.
14Design Patterns and Anti-patterns
A few consistent habits separate teams who tune segment behavior thoughtfully from teams who quietly accumulate storage and performance headaches.
Helpful patterns
- Matching segment size to actual traffic — larger segments for very high-volume topics, smaller ones for lighter traffic.
- Setting sensible retention deliberately — balancing genuine business need for history against real storage cost.
- Monitoring segment count over time — catching misconfigured roll settings before they cause real storage strain.
- Understanding compaction’s segment-by-segment nature — expecting cleanup to happen gradually, not instantly.
Problem
Leaving segment size and age settings at generic defaults for every topic, regardless of how wildly different their actual traffic volumes are.
Why It’s Harmful
A very high-traffic topic may roll segments far too frequently, creating unnecessary overhead, while a low-traffic topic may hold onto an oversized active segment for far too long.
Correct Approach
Tune segment size and age settings deliberately per topic, based on its own real, observed traffic pattern.
Problem
Assuming that deleting a record from an external system also instantly removes its corresponding data from a Kafka segment already written to disk.
Why It’s Harmful
Data remains physically present in its segment file until that whole segment ages out through retention or is rewritten through compaction, which can be a genuine surprise during a compliance or privacy review.
Correct Approach
Plan retention and compaction settings deliberately around how quickly sensitive data actually needs to be fully removed from disk.
15Best Practices and Common Mistakes
A short checklist for working confidently with Kafka’s log segments.
Best Practices
- Tune segment size and age settings to match each topic’s real traffic pattern.
- Monitor segment count and disk usage as ongoing, routine health signals.
- Choose retention or compaction deliberately, based on whether full history or only current state matters.
- Remember that retention removes whole segments, not individual records, when planning cleanup timing.
Common Mistakes
- Leaving every topic on identical, generic segment settings regardless of actual usage.
- Assuming deleted or expired data disappears from disk immediately.
- Ignoring a steadily growing segment count as simply “how Kafka behaves.”
- Forgetting that compaction, like retention, works gradually and segment by segment.
16Frequently Asked Questions
It varies over time, depending on traffic volume, segment size and age settings, and how much history retention allows. A partition typically has one active segment and any number of closed ones.
Yes, closed segments remain fully readable for as long as retention keeps them around. Being closed only means they no longer receive new writes.
No, a segment roll is a routine, lightweight internal operation. Producers and consumers continue working normally without any noticeable interruption.
No, a partition is the full, logical ordered stream of data. A segment is just one smaller file that makes up a portion of that partition’s overall storage.
The companion index file is deleted right alongside its segment’s data file, since it has no purpose without the segment it describes.
Yes, segment size and age settings can typically be adjusted at any time, though the change generally applies going forward, affecting new segments rather than retroactively altering ones already written.
17Summary and Key Takeaways
We began with a lighthouse keeper’s shelf of numbered diaries and ended inside one of Kafka’s quieter but genuinely important internal designs. A log segment is simply one file making up part of a partition’s overall log, with only the newest segment ever being actively written to at a time. This small, deliberate structural choice is what lets Kafka delete old data quickly and safely, find any specific offset efficiently, and clean up historical records through compaction without ever disrupting the data currently being written. Understanding segments is one of the clearest ways to see how Kafka turns the abstract idea of “an endless log” into something genuinely practical and fast to operate at real scale.
Key Takeaways
- A partition’s log is really many segment files — smaller, ordered pieces rather than one giant, ever-growing file.
- Only one segment is ever active — every other segment is closed, read-only, but still fully readable.
- Segments roll based on size or age — whichever configured limit is reached first closes the current segment.
- Retention deletes whole segments — never individual records trimmed out of a file in place.
- Compaction works segment by segment — cleaning up outdated keys without ever touching the active segment.
- Index files make reads fast — letting Kafka jump almost directly to any requested offset.
- Segmentation is what makes Kafka’s log practical — turning an abstract, endless stream into something efficient to store, search, and clean up at real scale.