Before we look inside a segment, we need to know where segments live and how they are named.

From topic to folder

A Kafka topic is a named stream of messages, such as orders. A topic is split into partitions, and each partition is an independent, ordered list of messages. On a broker, every partition that the broker hosts becomes one plain folder on the disk. The folder name is the topic name, a dash, and the partition number. So partition 0 of the orders topic becomes a folder called orders-0.

All of these folders sit inside the directories you list in the broker setting log.dirs. Despite the confusing name, this setting has nothing to do with application logs like error messages. It is the place where Kafka keeps its data. Some teams list several directories, one per physical disk, so the broker can spread partitions across disks.

graph TD
  A["log.dirs directory on the broker"] --> B["orders-0 folder"]
  A --> C["orders-1 folder"]
  A --> D["payments-0 folder"]
  B --> E["Segment files: .log, .index, .timeindex"]
  B --> F["Checkpoint and state files"]
  C --> G["Segment files"]
  D --> H["Segment files"]
        

Segment names are numbers with a meaning

Inside a partition folder you will see files whose names are long strings of digits, such as 00000000000000000000.log or 00000000000000368769.log. That number is the base offset of the segment. An offset is the position number of a message inside a partition. The first message ever written gets offset 0, the next gets 1, and so on. The base offset is simply the offset of the first message that lives in that segment.

The name is padded with zeros to a fixed length of twenty digits. This is a clever small trick. Because every name has the same length, sorting file names alphabetically also sorts them by offset. The largest possible offset in Kafka fits in nineteen digits, so twenty digits is always enough. A person listing the folder sees the history of the partition in perfect order without any special tool.

If one huge file could hold a whole partition, why does Kafka bother with many smaller ones?

The idea of a commit log is very old. You only append to the end, and you never change what was already written. Kafka builds on this idea, but it does not keep one endless file. It cuts the log into pieces, and there are four strong reasons for that.

The two kinds of segments

At any moment a partition has exactly one active segment. This is the newest one, and it is the only one that accepts new messages. Every other segment is closed, also called rolled or inactive. Closed segments never change their message content. This split matters a lot because many Kafka rules are stated in terms of it. For example, the retention system may only delete closed segments, and the log cleaner only works on closed segments.

A segment is not one file. It is a small family of files that share the same base offset in their names.

When people say “segment file”, they usually mean the .log file. But the broker creates several companions for it. Each companion has one job. Here is the complete family, from most important to least common.

Files that belong to the whole partition

Besides the per-segment files, the partition folder holds a few files that describe the partition as a whole. The most important is leader-epoch-checkpoint, a small text file that records which broker was leader during which range of offsets. You will also find partition.metadata, which stores the unique ID of the topic. And at the top level of each log.dirs directory there are checkpoint files such as recovery-point-offset-checkpoint, replication-offset-checkpoint, log-start-offset-checkpoint, and cleaner-offset-checkpoint, plus a meta.properties file that identifies the broker and the cluster.

The data file has almost no structure of its own. It is just batches placed one after another.

If you could look at the raw bytes of a .log file, you would not find a file header, a footer, a table of contents, or a special end marker. The very first byte of the file is already the start of the first record batch. When that batch ends, the next batch begins immediately, and so on until the file ends. This “nothing but data” design is one reason writes are so fast: the broker only ever pushes bytes to the end.

What is a record batch?

A record batch is a group of one or more messages that were packed together by the producer. Kafka calls each individual message a record. A producer does not send every record by itself. It waits a little, collects records for the same partition, and sends them as one batch. The controls for this are batch.size and linger.ms on the producer side.

The batch is the unit of everything important in Kafka: it is the unit of network transfer, the unit of compression, the unit of checksum checking, and the unit that is stored on disk. The broker does not usually pull a batch apart. It stores the very same bytes that arrived over the network.

graph LR
  A["Batch 1: offsets 0 to 4"] --> B["Batch 2: offsets 5 to 9"]
  B --> C["Batch 3: offsets 10 to 10"]
  C --> D["Batch 4: offsets 11 to 25"]
  D --> E["Next batch goes here"]
        

Offsets belong to records, batches carry the range

Every record has its own offset, but a batch does not store a separate offset for each record in full. It stores one base offset for the first record and a small offset delta for each record. To find the real offset of a record, you add its delta to the base offset of the batch. This saves a lot of space, especially for large batches.

Every batch starts with a fixed header of 61 bytes that tells the broker everything it needs to know without opening the records.

The current on-disk format is called message format version 2, and its magic value is 2. It was introduced with Kafka 0.11 together with idempotent producers and transactions. The header is fixed in size and is laid out in a strict order. Reading it is like reading the label of a parcel.

Why the checksum starts where it does

Look closely at the order of the fields. The checksum begins at the attributes field and covers everything after it. It deliberately does not cover the base offset or the partition leader epoch. This is a very smart design choice. When a batch arrives at the leader broker, the broker must stamp the real base offset and the current leader epoch into the header. If the checksum covered those two fields, the broker would have to recompute the checksum for every single batch. Because the checksum skips them, the broker can overwrite just those bytes and leave everything else untouched.

The producerId, producerEpoch, and baseSequence fields are the heart of exactly-once features. They let the broker recognize that the same batch arrived twice, for example because of a network retry, and quietly ignore the copy. They live inside every batch, even if you never turned on transactions, because idempotence is on by default in modern Kafka clients.

Sixteen bits, and only a handful of them are used. Each one changes how the whole batch behaves.

The attributes field is a row of on/off switches packed into two bytes. Understanding it explains a lot of Kafka behavior, from how compression works to how consumers skip aborted transactions.

Create time versus log append time

Timestamps are important because time-based retention and time-based lookups depend on them. With create time, the producer application writes the time it thinks the event happened. This is great for event data, but it means a badly set clock on a producer can put strange values into your log. With log append time, the broker overwrites the timestamp with its own clock when the batch arrives. This is more predictable but hides the real event time. The topic setting message.timestamp.type chooses between the two.

After the header comes the list of records. Each one is packed as tightly as possible.

A record contains a key, a value, an optional list of headers, and some small bookkeeping numbers. The clever part is the use of variable-length integers, often called varints. A normal 32-bit number always takes four bytes, even if the number is tiny. A varint uses only as many bytes as it needs: small numbers take one byte, and larger numbers take more. Since most lengths and deltas in real data are small, this saves a great deal of space when you have millions of records.

There is no separate storage for the record length in the header of the batch, which is why a reader must walk through the records in order. The batch header already tells you how many records to expect, and each record tells you how long it is, so walking is easy and exact.

Kafka compresses whole batches, and the header always stays readable.

When compression is turned on, the producer compresses only the records part of the batch. The header stays in plain form. That way the broker can read the base offset, the record count, the timestamps, and the producer identity without decompressing anything. The attributes field tells everyone which codec was used, so a consumer knows how to unpack the records later.

Because the whole batch is compressed as one block, bigger batches usually compress better. Repeated field names in JSON, repeated keys, and similar values across records all give the compressor more patterns to find. This is why a producer with a slightly longer linger.ms often sees smaller network traffic and smaller disks, even though each message waits a few extra milliseconds.

What the broker does with compressed data

In the normal case, the broker does not decompress and recompress. It checks the batch header, verifies the checksum, assigns the offset, and appends the bytes as they came. This is the fast path and it is the reason a broker can handle very high throughput on modest hardware. There are only a few cases where the broker must open the batch and rebuild it:

• The topic has a compression setting that forces a different codec than the producer used.
• The topic uses log append time, so the timestamps inside must be rewritten.
• The batch uses an older message format and must be converted.

Each of these costs extra CPU on the broker. When you see a broker’s CPU climbing unexpectedly, mismatched compression settings between producers and topics are a very common cause.

The batch size limit

The topic setting max.message.bytes limits the size of a batch, measured after compression. Its name is a leftover from old days when a batch held a single message. If a producer builds a batch bigger than this, the broker rejects it. Consumers and follower replicas must also be able to fetch a batch of this size, so if you raise the limit you must raise the matching fetch limits too.

The on-disk format has changed over the years, and each change solved a real problem.

The magic byte inside every batch is a version stamp. Looking at the history helps you understand why version 2 looks the way it does.

What version 2 fixed

To read message number 1,000,000 without scanning a gigabyte, Kafka keeps a small map next to each log file.

Suppose a consumer asks for offset 4,517,210. The broker first chooses the right segment by looking at the base offsets. But inside a segment that may be a full gigabyte, it still needs to find the exact spot. Scanning from the start would be painfully slow. So each segment has an offset index, stored in the .index file.

Sparse on purpose

The index does not list every message. It adds one entry only after a certain number of bytes have been appended since the last entry. That number is controlled by index.interval.bytes, which defaults to 4096 bytes. So roughly every four kilobytes of data, one small entry is added. This is called a sparse index.

What one entry looks like

Each entry is exactly eight bytes. Four bytes hold the relative offset, which is the offset minus the segment base offset. The other four bytes hold the physical position, which is the byte location in the .log file where a batch starts. Using relative numbers keeps them small enough for four bytes, and this is one of the hidden reasons segments are limited in size: the position must fit in a four-byte number, so a segment must stay below about two gigabytes.

The arithmetic is simple. A full one-gigabyte segment holds about 262,144 chunks of four kilobytes. At eight bytes each, that is about two megabytes of index. This is tiny compared to the data, and that is why the index can be memory-mapped without stress.

sequenceDiagram
  participant C as Consumer
  participant B as Broker
  participant S as Segment list
  participant I as Offset index
  participant L as Log file
  C->>B: Fetch from offset 4517210
  B->>S: Which segment holds this offset?
  S-->>B: Segment with the closest smaller base offset
  B->>I: Find largest entry not above the target
  I-->>B: Byte position in the log
  B->>L: Scan forward from that position
  L-->>B: Batch containing the target offset
  B-->>C: Return batches
        

How a lookup works

The broker keeps the list of segments sorted by base offset in memory. To find a segment, it looks for the largest base offset that is not bigger than the requested offset. Then it does a binary search inside that segment’s index, which means it repeatedly halves the search range, to find the largest indexed entry that does not go past the target. That entry gives a byte position. The broker starts reading the log from there and walks a few batches forward until it reaches the batch containing the target. Since entries are only four kilobytes apart, the walk is short.

Memory mapping and preallocation

The index file is memory-mapped, which means the operating system lets the program treat the file like a normal block of memory. The broker preallocates the file at its maximum size, controlled by segment.index.bytes with a default of ten megabytes. When a segment is closed, the file is trimmed to its real size. This is why an active segment’s index file looks bigger than expected: it is a reserved space, not real data. Operators who see ten-megabyte index files on many partitions sometimes panic, but it is normal.

Offsets answer “which message number”. Timestamps answer “what happened around 3 PM yesterday”.

Sometimes people do not know an offset. They know a time. A team may want to replay everything since a bad deployment at a certain hour. Kafka supports this through the time index, stored in the .timeindex file. Each entry is twelve bytes: an eight-byte timestamp and a four-byte relative offset.

An entry means “by the time we reached this offset, the largest timestamp we had seen so far was this.” The word largest is important. Real timestamps do not always increase, because different producers have different clocks. To make searching possible, the time index is always kept in increasing order by only recording the maximum timestamp seen so far.

The time index also drives retention

Time-based retention needs to know how old a segment is. Kafka uses the largest timestamp found in the segment for this, and that value is available from the time index without reading the log. A segment becomes eligible for deletion only when its newest record is older than the retention limit. This is the exact reason why a message can live longer than the retention period: the segment must be fully expired first.

Let us follow one fetch request from start to finish and see every file it touches.

A subtle point: whole batches only

The broker does not cut a batch in half. If a consumer asks for an offset in the middle of a batch, the broker returns the whole batch, starting from its first record. The client library then quietly throws away the records that come before the requested offset. This detail explains why the consumer sometimes receives slightly more data than it asked for on the wire, and it keeps the broker’s work extremely simple.

flowchart TD
  A["Fetch request with offset"] --> B{"Offset in valid range?"}
  B -- "No" --> C["Return out of range error"]
  B -- "Yes" --> D["Find segment by base offset"]
  D --> E["Search offset index for byte position"]
  E --> F["Limit by high watermark or last stable offset"]
  F --> G["Send file region to network"]
  G --> H["Consumer verifies and decompresses"]
        

New data always goes to the end of the active segment, until a rule says it is time to start a new one.

What happens on append

When a producer’s batch reaches the partition leader, the broker checks the batch for basic sanity: the format version, the size limit, the checksum, and whether it is a duplicate according to the producer sequence. If all is well, the broker assigns the next offsets, writes the base offset and the leader epoch into the header, and appends the bytes to the active segment. It also adds an entry to the offset index and the time index whenever the byte gap or timestamp rules say so. Then the followers copy the same bytes. Followers store exactly the same batches, byte for byte, which is how all replicas end up identical.

Notice that a write goes first into the operating system’s page cache, which is spare memory used by the operating system to speed up file access. Kafka does not force each write to the physical disk immediately. We will discuss why that is safe in the durability chapter.

The rolling rules

Rolling means closing the active segment and creating a new one whose base offset equals the next offset to be written. Kafka rolls a segment when any of these things happens:

What the roll actually does

On a roll, the broker flushes the old segment’s data to disk, trims the old index files to their true sizes, saves a snapshot of the producer state, and creates a new set of files named after the new base offset. From then on, the old segment is read-only. This is the moment when the old segment becomes eligible for retention and compaction rules.

Exactly-once behavior leaves footprints in the segment folder. Here is what each one does.

Duplicate detection and the producer snapshot

Every idempotent producer gets an ID from the cluster and numbers its batches in order. The leader remembers, for each producer and partition, the recent sequence numbers it has accepted. If the same batch arrives again after a network timeout, the leader sees the old sequence number and refuses to write it a second time. The broker keeps information about the most recent few batches from each producer, which is why the client limits how many unacknowledged requests may be in flight at once.

This producer state lives in memory, but memory is lost at restart. To avoid re-reading the whole log to rebuild it, the broker saves a producer state snapshot to a .snapshot file whenever a segment rolls. After a restart, the broker loads the latest snapshot and only replays the log after that point.

Control batches: the invisible markers

A transaction lets a producer write to several partitions and then either commit or abort all of it together. Data records are written right away, like normal ones, but flagged as transactional. When the producer finishes, the transaction coordinator writes a control batch into each partition involved. It contains a single marker that says commit or abort. Control batches take up offsets in the log, which is why you sometimes see gaps in the offsets a consumer receives. Client libraries never hand control records to your application.

The transaction index

A consumer in read-committed mode must skip data from aborted transactions. To do that quickly, the broker keeps a transaction index in the .txnindex file. Each entry lists an aborted transaction with the producer ID, the first offset, the last offset, and the last stable offset at the time. When a read-committed fetch happens, the broker attaches the list of aborted transactions for the returned range, and the client filters those records out. A segment that has never seen an aborted transaction has no such file at all.

graph LR
  A["Data batch: transactional"] --> B["Data batch: transactional"]
  B --> C["Control batch: commit marker"]
  C --> D["Data batch: transactional"]
  D --> E["Control batch: abort marker"]
  E --> F["Transaction index records the aborted range"]
        

Copies of a partition must agree. This small text file helps them agree after a leader change.

Each time a new broker becomes leader for a partition, the partition’s leader epoch number goes up by one. Every batch is stamped with the epoch that was current when it was written. The leader-epoch-checkpoint file lists each epoch together with the first offset written under it. It is a tiny text file with a version line, a count line, and then one line per epoch.

Why is this needed? Imagine the old leader wrote a few messages that were never copied to the followers, and then it crashed. A follower becomes the new leader and accepts different messages at the same offsets. When the old leader comes back, its log has offsets that mean something different from the new leader’s log. The returning broker asks the new leader, “Where did my last epoch end according to you?” The answer tells it exactly where to cut its own log, so it truncates the unsafe tail and then catches up. Without epochs, brokers would have to guess using only lengths, and a wrong guess could silently keep different data at the same offset.

Kafka never deletes single messages under the delete policy. It deletes whole segments, in a careful sequence.

The three retention limits

With the delete cleanup policy, a background task checks the closed segments on a regular schedule. The check interval is controlled by log.retention.check.interval.ms, which defaults to five minutes. A closed segment may be removed for any of these reasons:

The gentle two-step deletion

Deleting a file that a reader may still be using can cause trouble. So Kafka does it in two steps. First, the segment is taken out of the in-memory list, so no new reader will find it, and its files are renamed with a .deleted suffix. Second, after a delay controlled by file.delete.delay.ms, which defaults to one minute, the files are physically removed by a background thread. This means disk space is not returned at the exact moment retention decides to delete. There is a short gap.

Some topics keep only the latest value for each key. Segment files are the tool that makes it possible.

With the compact cleanup policy, Kafka keeps at least the latest record for every key, and older records with the same key may be removed. This is useful for things like the current profile of each user or the current price of each product. The log is split into a clean part, which has already been compacted, and a dirty part, which has not yet been processed. The active segment is never compacted.

How the cleaner rewrites segments

A background group of cleaner threads picks the partition with the highest share of dirty data, controlled by min.cleanable.dirty.ratio. It builds an in-memory map from each key to its latest offset by scanning the dirty part. Then it copies the older segments into new ones, keeping only records that are still the latest for their key. Several small old segments can be merged into one new segment during this rewrite.

Details that surprise people

• Offsets never change. Records that survive compaction keep their original offsets. So compacted segments have gaps in offset numbers, and the index still works because it stores real positions.
• Tombstones. A record with a key and an empty value means “delete this key”. The tombstone itself stays for a while, controlled by delete.retention.ms, so that slow consumers can see the deletion before it disappears.
• Empty batches may stay. The last batch of a producer may be kept even if all its records are removed, so that duplicate detection information is not lost.
• Compaction is not instant. Old values can still be read until the cleaner reaches them.

flowchart LR
  A["Old segments with duplicate keys"] --> B["Cleaner builds key to latest offset map"]
  B --> C["Copy only latest records into .cleaned files"]
  C --> D["Rename to .swap"]
  D --> E["Replace old segments"]
  E --> F["Delete old files after delay"]
        

Kafka does not flush every write to disk, yet it survives crashes. Here is how.

The role of the page cache

When the broker writes to a file, the operating system first places the bytes in the page cache and writes them to the physical disk later. If only the Kafka process dies, the operating system still holds the bytes and no data is lost. Data can be lost only if the whole machine loses power before the cache is written. Kafka’s answer is not frequent flushing, which would be slow, but replication. A write is normally acknowledged only after enough in-sync replicas have received it, so the chance that every copy is lost at the same instant is tiny.

What happens after an unclean shutdown

When a broker is stopped cleanly, it writes a small marker file called .kafka_cleanshutdown and flushes everything. On the next start, it sees the marker and loads quickly. If the marker is missing, the broker assumes a crash and enters recovery. It reads the recovery point from a checkpoint file, which marks the offset up to which data is known to be safely on disk. Every segment beyond that point is checked.

The file format was designed hand in hand with how operating systems work.

When zero-copy does not apply

Zero-copy works when the broker can send file bytes as they are. It is lost when the broker must touch the data, for example when it has to encrypt traffic with TLS, because encryption needs the bytes inside the application. In older releases it was also lost when converting message formats for old clients. That does not mean TLS is a bad idea. It means you should plan more CPU capacity on brokers when you turn it on.

graph LR
  A["Producer batch"] -->|"network"| B["Leader broker"]
  B -->|"append same bytes"| C["Log file in page cache"]
  C -->|"same bytes"| D["Follower replicas"]
  C -->|"zero-copy send"| E["Consumers"]
        

Segments are such a clean unit that they can be moved to cheaper storage without changing how clients read.

Keeping months of data on fast local disks is expensive. Tiered storage lets a broker keep only recent segments locally and copy older closed segments to a remote system, such as cloud object storage. The feature was proposed as a Kafka improvement named KIP-405 and reached production readiness in release 3.9.

Only closed segments are eligible for copying. The active segment always stays local. Once a segment is safely stored remotely, the local copy may be removed according to a separate local retention limit, while the overall retention limit decides when the remote copy dies. Consumers keep using the same protocol. When they ask for an old offset, the broker fetches the needed piece from the remote store on their behalf.

Kafka ships with a command line tool that reads segment files and prints what is inside.

The tool is named kafka-dump-log. You point it at a .log, .index, .timeindex, .txnindex, or .snapshot file, and it decodes the bytes into readable lines. For a log file, it prints one line per batch showing the base offset, last offset, record count, producer ID, epoch, sequence, whether the batch is transactional, the compression codec, the batch size in bytes, and the checksum. With an extra option it can also print the actual key and value of each record. For index files, it can check that the entries are sane.

A few settings decide almost everything about how your segment files behave.

Thinking about segment size

Choosing the segment size is a trade-off between two forces. Larger segments mean fewer files, fewer open file handles, and less overhead. But they also delay deletion and compaction, since only closed segments qualify. Smaller segments give finer control over retention and quicker cleaning, but they multiply the number of files, and each partition needs at least three open files per segment. For a busy topic, the default one gigabyte is a good starting point. For a low-traffic topic, reduce the roll time so segments close even when they are small.

You do not need to look at files every day if the right numbers are on a dashboard.

Operating system numbers to watch

• Disk free space on every log directory, with an alert well before it is full.
• Open file descriptors compared with the allowed limit.
• Memory-map count, because each index file uses one, and the system has a limit.
• Disk latency and utilization, to detect slow devices.
• Page cache hit behavior, seen indirectly through disk read rates. Consumers that read old data cause disk reads that push out hot data.

Segment files hold your raw business data, so they deserve the same care as a database file.

Deleting sensitive data

Because Kafka is append-only, personal data does not disappear the moment you decide it should. It disappears when its segment is deleted, or when compaction rewrites it. Teams under privacy laws must plan for this. Options include short retention for sensitive topics, compacted topics with tombstones and a short tombstone retention time, or encrypting personal fields inside the message with keys that can be thrown away later. The last approach makes data unreadable even while the bytes still sit in old segments or backups.

Every design choice has a price. Here is the honest balance sheet.

These mistakes appear again and again in real clusters.

A short checklist that turns the theory of this tutorial into daily habits.

Let us apply everything to situations that engineers meet in production.

Kafka began at LinkedIn as a way to move large streams of activity data reliably, and the segmented log was central to its design from the beginning. Today, teams in many industries use the same format for event sourcing, change data capture, metrics pipelines, and log aggregation. The reason the design has survived is exactly what this tutorial showed: it is simple, and simple things are fast, recoverable, and easy to reason about.

Let us put real numbers on everything we learned, so the design feels concrete instead of abstract.

Imagine a partition that receives one thousand records every second, and each record carries about two hundred bytes of key and value. To keep the arithmetic friendly, we will ignore compression and small overheads for the moment. The partition therefore grows by about two hundred thousand bytes every second.

How fast do segments roll?

One gigabyte is a little over one billion bytes. Dividing that by two hundred thousand bytes per second gives roughly five thousand four hundred seconds, which is about ninety minutes. So this partition closes a segment about every hour and a half. In seven days, which is one hundred sixty-eight hours, that means about one hundred thirteen segments. Each segment has three regular files, so the partition folder holds around three hundred forty files, and the folder holds about one hundred twenty-one gigabytes of data.

How big are the indexes?

A full segment holds about 262,144 blocks of four kilobytes. Each block earns one offset index entry of eight bytes, so a full offset index needs about two megabytes. The time index uses twelve bytes per entry, so at most about three megabytes. Together, the helper files add around five megabytes to a one-gigabyte segment. That is half of one percent, a tiny cost for fast lookups.

How much does batching matter?

Suppose the producer waits five milliseconds before sending. In that time about five records collect for the partition, so it sends roughly two hundred batches per second. A batch of five records carries the fixed 61-byte header once, plus the five records with their small overheads. The header cost is spread across five records, a bit over twelve bytes each. Now imagine the producer sends every record alone. Each record carries its own 61-byte header, which is about a quarter of its total size. Over a week, that waste adds up to tens of gigabytes on this single partition, and it also multiplies the number of requests, checksums, and index updates.

The life of one record, start to finish

A partition folder is created, copied, moved, and finally removed. Segments play a role at every step.

Creation

When a topic is created, the cluster decides which brokers will hold each partition and its copies. Each chosen broker creates the folder, an empty first segment with base offset zero, and the small metadata files. The first segment’s files are created immediately, even before a single message arrives.

Replicas do not have to look identical

It is easy to assume that every replica of a partition contains exactly the same segment files. In truth, all replicas contain the same batches at the same offsets, but the way those batches are grouped into segments can differ. Each broker rolls its own segments according to its own clock and its own size counting, and each broker runs compaction on its own schedule. A replica that joined late and caught up by fetching a large amount of history may have cut its segments at different places than the leader. This is perfectly fine, because clients only care about offsets, and offsets agree everywhere.

Adding a broker or moving a partition

When a partition is reassigned to a new broker, the new broker becomes a follower. It fetches the log from the leader beginning at the leader’s log start offset, and writes what it receives into its own segments. Nothing is copied file by file. It is an ordinary replication fetch, just a very long one. This is why moving a partition with hundreds of gigabytes takes hours, and why throttling the replication traffic is important, so that regular clients keep good performance while the move runs. Tiered storage reduces this cost, since less of the history lives on local disks.

Failing disks and several directories

Brokers can use several log directories, one per disk. If one directory fails, for example because of disk errors, the broker marks it offline. The partitions in that directory go offline on this broker, and leadership for them moves to other replicas. The rest of the broker keeps running from the healthy directories. Operators can then replace the disk and let the broker rebuild the missing replicas by fetching from other brokers. Monitoring the count of offline directories is therefore very important.

Deleting a topic

When a topic is deleted, each broker renames the partition folder by adding a delete marker and a unique ID to its name, and then removes it in the background. The rename is quick and makes sure a new topic with the same name can be created straight away without clashing with the folder that is still being erased.

flowchart TD
  A["Topic created"] --> B["Partition folder and first segment created"]
  B --> C["Segments roll and grow over time"]
  C --> D["Old segments removed by retention or compaction"]
  C --> E["Replica moved to another broker"]
  E --> F["New replica rebuilds segments by fetching"]
  C --> G["Topic deleted"]
  G --> H["Folder renamed with delete marker"]
  H --> I["Folder erased in the background"]
        

Understanding what Kafka chose not to do makes its choices easier to remember.

Most databases store data in structures built for fast random updates, such as trees of pages that are changed in place. That design is wonderful when you need to change a single row and read it back instantly. But it needs a lot of random disk movement, careful locking, and complex repair work after a crash. Kafka chose a very different goal. It wants to accept an endless stream of facts as fast as possible and let many readers replay them in order. For that goal, an append-only log is the natural shape.

The consumer position is not in the segment

In many queue systems, the queue itself tracks which messages were consumed. In Kafka, the segment files know nothing about consumers. The position of each consumer group is stored separately, in an internal topic that is itself a log made of segments. Because of this, adding a tenth consumer group costs the log nothing. It only adds another position number. This separation is why one topic can serve dozens of independent applications at once.

Keep this table nearby when you read Kafka documentation or debug a storage problem.

Short answers to the questions people ask most often about segment files.

It is one slice of a partition’s log, made of a data file and its helper index files, all named after the offset of the first message in the slice. A partition is a chain of such slices.

The name is the base offset padded to twenty digits, so that alphabetical order equals offset order.

You will see mostly unreadable bytes, since it is a binary format, and compressed batches will look like noise. Use the dump tool to read it properly.

Index files are preallocated to their maximum size while the segment is active. They are trimmed to real size when the segment closes.

The broker rebuilds it from the log file when it starts, because the log is the only source of truth. The rebuild takes time on large partitions.

Gaps come from transaction control markers, from compaction removing older records, or from aborted transactions. The offsets themselves never get reused.

No. It deletes whole closed segments once the newest record in them is old enough, and it checks on a schedule, so deletion can lag.

For the process crashing, yes, because the operating system still holds the bytes. For a whole machine failing, safety comes from having other replicas with the same data.

No. Byte positions in the index are four-byte numbers, so a segment must stay under about two gigabytes.

Batches are returned whole. If your requested offset falls inside a batch, you receive the entire batch and the client drops the earlier records.

It is the only one that receives writes, and it cannot be deleted by retention or compacted by the cleaner until it rolls.

Yes. Compaction only rewrites closed segments into new ones, keeping the same batch format and the original offsets of surviving records.

A Kafka partition is a folder. That folder holds a chain of segments, each named after the offset of its first message. Inside each segment, a plain data file stores record batches one after another, and two small sparse indexes, one by offset and one by time, make lookups quick. Extra files record transaction facts, producer state, and leader history. Only the newest segment accepts writes, and once it rolls, it becomes immutable and eligible for retention, compaction, and tiering. Every part of this design serves the same goal: do the least possible work on the hot path, and make failure recovery simple and safe.