What Is Log-Based Storage in Kafka?
A beginner-friendly guide to the simple but powerful storage idea underneath Kafka, and how writing data in one direction only makes the whole system fast and dependable.
Imagine an old ship’s logbook, the kind a captain fills in by hand, one entry after another. Each entry gets written at the very next empty line, numbered in order, and nothing already written is ever erased or rearranged. If the captain wants to know what happened on a particular day, they simply look up that entry’s line number. This centuries-old idea of an ordered, append-only record turns out to be the exact same idea sitting at the heart of how Kafka stores every piece of data it handles. In this guide we will build up the whole picture slowly, defining every new word the first time it appears.
1Why Kafka Needed a Different Kind of Storage
Before defining log-based storage precisely, it helps to see the problem it was designed to solve.
Kafka’s job is very different from a typical database
A typical database is often built to answer questions like “what is the current value of this specific record”, frequently updating individual pieces of data in place as things change. Kafka’s job is different. It needs to continuously accept a massive, ongoing stream of new events, such as customer orders or sensor readings, and make them available to many independent readers, often very quickly after they arrive, without losing anything.
This is a fundamentally different workload. Kafka rarely needs to update an existing piece of data; it mostly needs to record new things as they happen, in the exact order they happened, and let that ordered record be read, and reread, reliably by many different programs.
Think of the difference between a filing cabinet, where documents are constantly pulled out, updated, and refiled, and a diary, where new entries are simply added to the end, one after another, and old entries are never rewritten. Kafka behaves far more like the diary than the filing cabinet.
An old, simple idea, applied at massive scale
The append-only log, a storage structure where new entries are only ever added at the end, is not a new invention. It has been used for decades in various forms. What makes Kafka notable is how it took this simple idea and built an entire, highly scalable messaging and storage system directly on top of it.
2So, What Exactly Is Log-Based Storage?
A precise definition, then the plain-English version.
A short definition
Log-based storage, in the context of Kafka, refers to storing each partition, a smaller ordered slice of a topic, as an append-only file, meaning new events are always written to the very end of the file, and nothing already written is ever changed or removed in place. Each event’s position in this file is identified by an offset, a number that increases by one with every new event.
In everyday terms, a Kafka partition behaves like that ship’s logbook, growing only by adding new lines at the bottom, with every existing line staying exactly where it was originally written, forever identifiable by its own permanent position.
The word “log” here is not about error messages
It is worth clarifying a common point of confusion early. In everyday computing conversation, the word “log” often refers to diagnostic messages an application prints out, such as error or debug logs. Here, “log” instead refers to this specific, ordered, append-only storage structure, a much older and more general use of the word that predates modern application logging entirely.
This append-only design is precisely what allows Kafka to guarantee that events within a single partition stay in the exact order they were written, since their positions never move once assigned.
3Why Appending Only Is So Much Faster
A surprising truth about how computer disks actually work.
Sequential writes versus scattered writes
Physical storage disks, and even modern solid-state drives to a lesser degree, tend to perform far better when data is written or read sequentially, meaning one position after another in a straight line, compared to when data is scattered randomly across many different positions. Constantly jumping around a disk to update existing records in place is considerably slower than simply writing new data continuously onto the next available space.
Kafka leans directly into this strength
Because Kafka only ever appends new data to the end of a partition’s log, and never modifies existing data in place, nearly all of its disk activity is sequential. This design choice directly takes advantage of what disks are naturally best at, which is a major reason Kafka can handle extremely high volumes of writes efficiently, even though it is durably writing everything to disk rather than only keeping it in memory.
Think of writing continuously along a single unrolling scroll of paper versus constantly flipping back and forth through a stack of loose index cards to update specific ones. Writing along the scroll, always moving forward, is naturally much faster and simpler than the constant searching and shuffling the index cards require.
Beginners sometimes assume that writing everything to disk must be slow, since disks are often thought of as much slower than memory. Sequential disk writes, like the ones Kafka relies on almost exclusively, are dramatically faster than random disk writes, and can rival memory-based systems for raw throughput.
4Segments: Breaking One Giant Log into Manageable Pieces
A single, ever-growing file would eventually become impractical. Kafka has a clean solution.
Why a single file is not enough
If a partition’s entire log lived in one continuously growing file forever, that file would eventually become enormous, and even simple tasks, like deleting old data past its retention period, would require expensive, wasteful work on a massive single file. Kafka avoids this by dividing each partition’s log into smaller pieces called segments.
How segments work
At any given time, a partition has one active segment, the file currently receiving new writes, and any number of older, closed segments containing earlier data. Once an active segment reaches a certain configured size, or a certain age, Kafka closes it and starts a fresh new active segment, continuing the log seamlessly across this new file.
Think of a diary that fills one notebook and then simply starts a brand-new notebook once the current one is full, rather than trying to cram an entire lifetime of entries into a single, ever-expanding book. Old, filled notebooks can be archived, or thrown away entirely, as complete units, without disturbing today’s notebook at all.
flowchart LR
Seg1["Segment 1 (closed)"] --> Seg2["Segment 2 (closed)"] --> Seg3["Segment 3 (active, currently being written to)"]
Because old data can be removed by simply deleting an entire closed segment file, cleaning up expired data becomes a fast, simple operation, rather than needing to search through and remove individual old entries one at a time.
5Indexes: Finding a Specific Offset Quickly
A log alone would still be slow to search without some extra help.
The problem with searching a plain log directly
If a consumer, a program reading data from Kafka, wanted to start reading from a specific offset, simply scanning through an entire segment file from the very beginning, checking every single entry until the right one was found, would be far too slow for a system built to handle huge volumes of data.
A separate index file for each segment
To solve this, Kafka maintains a separate, much smaller index file alongside each segment, mapping certain offsets to their approximate physical position within that segment’s file. When a consumer requests a specific offset, Kafka uses this index to jump very close to the right position almost immediately, needing only a small amount of additional scanning from there, rather than searching through the entire segment from the start.
Think of the index at the back of a large reference book. Rather than flipping through every single page to find a particular topic, you check the index, which points you almost directly to the right page, needing only a small amount of extra searching once you get there.
flowchart LR
Req["Request: offset 8500"] --> Idx[Index File Lookup]
Idx --> Pos["Approximate position found"]
Pos --> Scan["Small scan to exact offset"]
Scan --> Found["Event at offset 8500 returned"]
6Leaning on the Operating System: The Page Cache
One of Kafka’s cleverer performance tricks involves deliberately not reinventing something the operating system already does well.
What the page cache is
Modern operating systems automatically keep a copy of recently accessed disk data in memory, an area often called the page cache, so that reading the same data again shortly afterward can be served quickly from memory rather than needing to go back to the slower physical disk each time.
Why Kafka relies on this instead of managing its own cache
Rather than building and managing its own separate memory caching system, Kafka deliberately writes and reads through the operating system’s normal file mechanisms, letting the page cache do this caching work automatically. Since Kafka’s data is often read shortly after being written, especially by consumers keeping up closely with new events, a large portion of reads end up being served directly from this fast memory cache, without needing to touch the disk again at all.
Think of a librarian keeping recently returned books on a nearby cart instead of immediately reshelving them deep in the stacks. If someone asks for one of those books again soon afterward, the librarian can hand it over instantly from the cart, without a slow trip into the depths of the library.
This is one reason why running Kafka brokers with a generous amount of available memory, even beyond what Kafka’s own internal processes strictly need, often improves performance considerably, since more memory means more room for this operating-system-managed page cache.
7Zero-Copy: Cutting Out Unnecessary Work
A technical optimisation that quietly saves a surprising amount of effort.
The usual, wasteful path
In many ordinary systems, sending data stored on disk out over the network involves several unnecessary steps: reading the data from disk into the application’s own memory, and then copying that same data again into a separate area specifically used for sending it over the network. Each of these extra copies takes real time and computing effort.
Skipping the middle steps
Kafka takes advantage of a technique called zero-copy, which allows the operating system to send data directly from its page cache straight out over the network, without needlessly copying it through the application’s own memory space first. This removes unnecessary work from a very common, high-volume operation, letting Kafka serve consumers extremely efficiently.
Think of a warehouse worker handing a package directly from the shelf straight to a waiting delivery truck, instead of first carrying it to a separate desk, unpacking and repacking it there, and only then walking it out to the truck. Skipping the unnecessary middle steps gets the same package delivered with far less wasted effort.
flowchart LR
Disk["Disk / Page Cache"] -->|Zero-copy| Net["Network, straight to the consumer"]
8Retention: How Log-Based Storage Decides What to Keep
An append-only log cannot grow forever, so something has to decide when old data goes.
Time-based and size-based cleanup
Every topic has a retention setting controlling how long its data is kept, commonly expressed as a length of time, such as seven days, or as a maximum total size. Once a segment’s data falls entirely outside this retention window, or the topic exceeds its allowed size, Kafka removes the relevant old, closed segments entirely, freeing up their disk space.
Why removing whole segments is efficient
Because segments group many individual events into one file, cleaning up expired data simply means deleting entire segment files once every event inside them has aged out, a fast, low-effort operation. This is far more efficient than having to search through and remove individual expired entries one at a time from a single enormous file.
| Retention style | What triggers cleanup | Typical use |
|---|---|---|
| Time-based | A segment’s data becomes older than the configured retention period | Event history, auditing, and replay scenarios |
| Size-based | The topic’s total stored data exceeds a configured limit | High-volume topics with a fixed storage budget |
9Log Compaction: A Different Kind of Cleanup
Sometimes what matters is not how long data has existed, but whether it has since been replaced.
Keeping only the latest event per key
Log compaction is an alternative cleanup approach, where instead of removing data purely based on age or size, Kafka removes older events once a newer event sharing the exact same key has been written, keeping only the most recent event for each distinct key permanently. A key is an identifying label attached to certain events, letting Kafka know which older entries a newer one is meant to replace.
Why this fits certain kinds of data so well
Some topics are meant to represent the current state of something, such as the latest known address for every customer, rather than a complete history of every change that ever happened. Log compaction lets such a topic stay reasonably small over time, since it only ever needs to hold one entry per key, no matter how many times that key’s value has actually changed over the years.
Think of a filing cabinet that only ever keeps the most recently submitted form for each person, automatically shredding any older version of that same person’s form the moment a newer one arrives, rather than keeping every historical version stacked up forever.
Log compaction does not remove data based purely on age, the way ordinary retention does. A very old event can survive indefinitely under compaction if no newer event sharing its exact key has ever come along to replace it.
10How Log-Based Storage Powers Replaying Data
One of the most practically valuable side effects of this whole design.
Old events do not disappear when read
Because reading from a Kafka log does not remove or change any data, and events remain available until they are actually cleaned up by retention or compaction, a consumer can deliberately move backward to an earlier offset and reread events it has already seen, or events it never read at all, as long as they are still retained.
Why this matters for real systems
This ability to replay history is extremely valuable in practice. A brand-new service can rebuild its own internal understanding of the world by replaying a topic’s full retained history from the beginning. A team fixing a bug can reprocess exactly the period of time affected by that bug, correcting mistakes without needing a separate backup system just for this purpose.
Think of being able to rewind and rewatch an earlier part of a recorded broadcast, rather than only ever being able to watch it live, once, before it disappears forever. Log-based storage gives Kafka’s readers this same rewind ability, within whatever retention period has been configured.
11Advantages and Trade-offs of Log-Based Storage
No design is free of cost. Understanding the trade-offs helps use it wisely.
Advantages
- Extremely fast writes, thanks to sequential, append-only disk access
- Simple, efficient cleanup by removing whole segment files at once
- Naturally supports replaying and rereading historical data
- Benefits automatically from the operating system’s page cache
- Zero-copy reads make serving data to consumers highly efficient
Disadvantages / Trade-offs
- Not designed for quickly updating or looking up individual arbitrary records
- Retained history requires real disk space, which must be planned for
- Log compaction adds background work to continuously clean up superseded entries
- Understanding segments and indexes adds some conceptual overhead for newcomers
12Log-Based Storage Compared with a Traditional Database
Seeing the two approaches side by side makes the distinct design choices clearer.
| Aspect | Traditional Database Storage | Kafka’s Log-Based Storage |
|---|---|---|
| Typical write pattern | Frequent updates to existing records, in place | Only ever appending new entries to the end |
| Lookup style | Flexible queries across many fields | Sequential reading, or lookup by offset |
| Deleting old data | Removing individual rows | Removing whole segment files, or compacting by key |
| Best fit | Managing current, frequently changing records | Recording and replaying a continuous stream of events |
If your main need is quickly updating and querying individual records by arbitrary fields, a traditional database remains the better fit. If your main need is durably recording an ordered stream of events, and letting many readers replay them, Kafka’s log-based storage is purpose-built for exactly that.
13Best Practices and Common Mistakes
Practical lessons for working comfortably with log-based storage.
Best practices
Choose retention settings deliberately
Base time and size limits on realistic needs for replay and recovery, balanced against the real disk space available.
Use log compaction only where it truly fits
Reserve compaction for topics genuinely representing current state per key, rather than full historical records.
Provide brokers with enough memory
Generous available memory allows the operating system’s page cache to serve more reads quickly, improving overall performance.
Monitor disk usage against retention settings
Ensure available storage genuinely matches what your chosen retention period and traffic volume will actually require over time.
Take advantage of replay when it helps
Design systems to benefit from Kafka’s ability to reread history, rather than building separate, redundant backup mechanisms for the same purpose.
Mistakes beginners often make
Trying to use Kafka the way one might use a traditional database, expecting quick lookups by arbitrary fields, runs against how log-based storage is actually designed to work, which favours sequential access and lookup by offset.
Setting a long retention period without properly accounting for real disk usage can lead to brokers running out of storage space unexpectedly.
Enabling log compaction on a topic that actually needs its full historical record, rather than just the latest state per key, can silently discard events a system later needs, once older entries have already been compacted away.
14Frequently Asked Questions
No. Once an event is written, it cannot be modified in place. Data is only ever added at the end of the log, and old data is eventually removed entirely, whether through ordinary retention or log compaction, never edited.
Splitting into segments makes cleaning up expired data far more efficient, since an entire old segment can simply be deleted as a whole file, rather than needing to search through and remove individual entries.
Each segment has an accompanying index file that maps certain offsets to their approximate position, allowing Kafka to jump close to the right spot almost immediately, needing only a small amount of extra scanning from there.
No. This is a completely different meaning of the word “log”. Here, it refers to the ordered, append-only storage structure Kafka uses internally, not diagnostic messages an application prints for troubleshooting purposes.
Ordinary retention removes data purely based on age or total size, regardless of its content. Log compaction instead removes older events specifically because a newer event with the same key has replaced them, regardless of how old either one actually is.
No. Kafka durably writes its data to disk. It simply benefits heavily from the operating system’s page cache for frequently accessed, recent data, which can make many reads feel almost as fast as if the data were purely in memory.
Yes, as long as that data has not yet been removed by retention or compaction. Reading is non-destructive, which is exactly what allows consumers to replay and reread history when needed.
15Summary and Key Takeaways
Log-based storage is the simple but powerful idea underneath every Kafka partition: an append-only, ordered record where new events are only ever added to the end, never edited or rearranged in place. This design takes direct advantage of how disks perform best, splits naturally into manageable segments for efficient cleanup, and pairs with lightweight index files for fast lookups by offset. By leaning on the operating system’s page cache and using zero-copy techniques for sending data to consumers, Kafka turns this straightforward structure into a genuinely high-performance system. Retention and log compaction offer two different, purposeful ways to eventually let go of old data, while the non-destructive nature of reading gives every consumer the ability to replay history whenever it is needed. Understanding this foundation makes nearly everything else about how Kafka behaves easier to reason about.
Key Takeaways
- Each partition is stored as an append-only log — new events are only ever added to the end, never edited in place.
- Sequential writes are what make Kafka fast — leaning directly into how disks naturally perform best.
- Segments break the log into manageable pieces — making cleanup fast by removing whole files at once.
- Index files enable quick lookups by offset — avoiding the need to scan an entire segment from the start.
- The page cache and zero-copy boost performance further — serving recent, frequently read data very efficiently.
- Retention and compaction offer two cleanup strategies — one based on age or size, the other on replacing outdated keys.
- Non-destructive reads enable replaying history — a powerful capability built directly into this storage design.