Before we meet the files themselves, we need to feel the problem they were invented to solve.

Kafka in one paragraph

Kafka stores messages, which it calls records, in named streams called topics. To spread the work across many machines, each topic is cut into pieces called partitions. A partition is an append-only log: new records are only ever added at the end, and old records are never edited in place. Each record in a partition gets a whole number called its offset, starting at 0 and growing by one for every record. Offset 41 is the record right before offset 42, always, inside that partition.

The problem: a huge log and a picky reader

A consumer, which is a program that reads records, does not always want to start at the beginning. Very often it says, “I already processed everything up to offset 8,399,999, so please start from 8,400,000.” A single partition can hold hundreds of gigabytes. Reading from the start and counting records until we reach offset 8,400,000 would take a very long time and would waste disk and network. Kafka must jump straight to the right spot, and it must do that for thousands of consumers at once, all day long.

There is a second, different question. Sometimes a person does not know an offset at all. They only know that “the bug started at 2:15 in the afternoon” and they want to replay everything from that moment. Offsets are just counters, so they say nothing about the clock. Kafka needs a second helper that translates a time into an offset.

The two helpers

Words you will see again and again

Index files do not live alone. They are one member of a small family of files that together describe a segment.

Partitions become directories, logs become segments

On a broker’s disk, every partition is a directory. The directory name is the topic name, a dash, and the partition number, so the third partition of a topic called payments is stored in a directory named payments-2. Inside that directory you will not find one giant file. You will find many segments, and each segment is a small group of files that share the same name prefix.

The prefix is the base offset of the segment: the offset of the first record that the segment can hold. It is written as a twenty-digit number padded with zeros, so a segment starting at offset 368,769 has a name that begins with the digits 00000000000000368769. Because the name is the base offset, Kafka can sort the segments simply by name, and it can tell which offsets each segment covers without opening a single file.

flowchart TD
  T["Topic: payments"] --> P0["Partition directory: payments-2"]
  P0 --> S1["Segment with base offset 0"]
  P0 --> S2["Segment with base offset 368769"]
  P0 --> S3["Active segment with base offset 737912"]
  S2 --> L2["log file: the records"]
  S2 --> I2["index file: offset to position"]
  S2 --> TI2["timeindex file: time to offset"]
  S2 --> TX2["txnindex file: aborted transactions"]
  P0 --> CK["checkpoint and epoch files"]
        

The family of files

In this tutorial we focus on the two files with the words index and timeindex in their names. The others matter because they show that Kafka uses the same idea, “keep a small side file next to the big data file”, for several different questions.

Why segments exist at all

Why not keep one endless log file per partition? There are three strong reasons. First, deleting old data would be painful: to remove the oldest hour you would have to rewrite a giant file, while with segments Kafka simply deletes whole old files. Second, indexes would grow without limit; with segments each index stays small and bounded. Third, after a crash, Kafka can check and repair segments one by one instead of scanning one monstrous file.

Active segment versus closed segments

Only the newest segment, called the active segment, accepts writes. All older segments are closed: their log and index files never change again until they are deleted. This simple rule makes reading old data very safe and very cache friendly, because closed files are read-only and their pages in memory are never dirtied by new writes.

The offset index is a tiny sorted table that turns “which offset?” into “which byte?”.

What one entry looks like

Every entry in the offset index is exactly eight bytes. The first four bytes hold a relative offset. The last four bytes hold a physical position, which is the number of bytes from the start of the .log file where a batch of records begins. The pair says: “Around this offset, start reading at this byte.”

Why relative? Instead of storing the full offset, which could be a big number needing eight bytes, Kafka stores how far the offset is from the segment’s base offset. If the segment starts at 368,769 and the entry is for offset 368,800, Kafka stores 31. The full offset is rebuilt by adding the base offset from the file name. This halves the size of the number, and it explains two hidden limits: a single segment cannot cover more than about two billion offsets, and a segment file cannot be larger than about two gibibytes, because both numbers must fit in four bytes.

Sparse, not dense

An obvious idea would be to write one index entry for every single record. That is called a dense index. Kafka deliberately does not do that. It writes an entry only after roughly a certain number of bytes have been appended since the last entry. The gap is controlled by a setting whose default is 4,096 bytes. This is called a sparse index.

Let us do the arithmetic for a full one gibibyte segment. With the default gap, the index holds at most 262,144 entries, and at eight bytes each that is about two mebibytes. Now imagine records that are only 100 bytes each. A one gibibyte segment then holds roughly ten million records, and a dense index would need about eighty mebibytes for that one segment. The sparse index is around forty times smaller, small enough that the indexes of thousands of segments can live comfortably in memory.

The price of being sparse

A sparse index cannot land exactly on the record you want. It lands on the closest signpost at or before it, and then Kafka reads forward through the log file until it reaches the exact batch. Because signposts are at most about four kibibytes apart, that forward walk is short: a few thousand bytes that are already in the page cache. A tiny bit of extra reading is a very good trade for a much smaller index.

Sorted and append-only

Entries are added only at the end, and each new entry has a larger offset and a larger position than the one before. The file is therefore always sorted, and a sorted table can be searched with binary search, the “guess the middle and halve the range” method. Finding one entry among a quarter of a million takes only about eighteen steps.

Preallocated files

When Kafka creates a new segment, it creates the index files at their full maximum size at once. The maximum is controlled by a setting whose default is ten mebibytes. The file is filled with zeros, and Kafka tracks how many entries are valid. When the segment is closed, Kafka trims the index down to its real size. Preallocation avoids the cost of growing the file repeatedly and lets the operating system map it into memory once.

The time index lets Kafka work with clocks even though offsets are only counters.

What one entry looks like

Every time index entry is twelve bytes. The first eight bytes are a timestamp in milliseconds. The last four bytes are a relative offset, counted from the segment’s base offset just like in the offset index. The entry means: “By the time we reached this offset, the biggest timestamp we had seen in this segment was this one.”

Notice the careful wording: the biggest timestamp seen so far, not the timestamp of that single record. This matters because record timestamps can arrive out of order. One producer machine may have a clock that runs a bit slow, and another may run fast. By always storing the largest timestamp seen so far, and by writing an entry only when that value grows, Kafka keeps the time index sorted. A sorted table again means binary search works.

Two kinds of timestamps

Each topic decides where its record timestamps come from. This choice changes how trustworthy the time index is.

Why the time index was invented

Before Kafka 0.10.1, there was no time index. Kafka decided when to delete old data by looking at the last modified time of the segment file on disk. That sounds reasonable until you remember that files get copied. When a partition is moved to a new broker, or a follower rebuilds its copy, the new files get fresh modified times. Old data suddenly looked young, so it stayed on disk far longer than intended. A proposal known as KIP-33 added the time index and per-record timestamps so that retention and searching could use the real data times instead of file times.

What the time index is used for

Every append to the log makes a small decision: is it time to drop another bookmark?

The write path, step by step

flowchart TD
  A["Batch arrives at the leader"] --> B["Assign offsets and validate timestamps"]
  B --> C["Append bytes to the active log file"]
  C --> D["Update largest timestamp seen so far"]
  D --> E{"Bytes since last entry at least the index interval?"}
  E -- "No" --> F["Done. No index entry this time"]
  E -- "Yes" --> G["Write offset index entry"]
  G --> H["Write time index entry if the timestamp grew"]
  H --> I["Reset the byte counter"]
  I --> J{"Segment full or too old?"}
  F --> J
  J -- "Yes" --> K["Roll to a new active segment"]
  J -- "No" --> L["Wait for the next batch"]
        

Bytes, not records

The gap is measured in bytes, not in number of records. If your records are large, say one kibibyte each, an entry appears roughly every four records. If your records are tiny, an entry might appear only every few dozen records. In both cases the distance the broker has to scan after a lookup is about the same, and that is exactly the point of measuring in bytes.

Batches are the unit of appending

Kafka appends whole batches, and index entries are decided after a batch is written. So an index entry always points to a batch boundary, never to the middle of a batch. When a consumer asks for an offset in the middle of a batch, the broker returns the whole batch and the consumer library quietly skips the earlier records it did not need.

Followers build their own indexes

This is an important and often misunderstood point. Followers do not copy the index files from the leader. They fetch the record batches, append them to their own log, and run the very same gap check to build their own index files. The offsets inside the batches are already assigned by the leader, so every replica ends up with identical records at identical offsets, while the exact index entries may differ slightly because the batch boundaries seen by followers can be different.

When a segment is closed

A segment is rolled, meaning closed and replaced by a fresh active segment, when any of these happens: the next batch would push the log past the segment size limit; the segment has been open longer than the segment time limit; the offset index or time index is full; or the relative offset would no longer fit in four bytes. Rolling is a normal, cheap event and we will follow it in detail in chapter 8.

Now we follow a real read request from the moment it arrives to the moment bytes leave the machine.

The scenario

A consumer sends a fetch request to the leader of a partition: “Give me records starting at offset 8,400,000.” The broker must find the right segment, the right byte inside the segment, and then send the data. Three small lookups do all the work, and each one is cheap.

sequenceDiagram
  participant C as Consumer
  participant B as Broker
  participant M as Segment map
  participant I as Offset index
  participant L as Log file
  C->>B: Fetch from offset 8400000
  B->>M: Which segment has the biggest base offset at or below 8400000
  M-->>B: Segment with base offset 8391200
  B->>I: Find last entry at or below relative offset 8800
  I-->>B: Entry for relative offset 8790 at position 5316608
  B->>L: Scan forward from position 5316608 to the right batch
  L-->>B: Batch found at position 5318912
  B-->>C: Send bytes from that position, up to the limits
        

Lookup 1: choose the segment

The broker keeps the segments of each partition in a sorted in-memory map whose keys are base offsets. It asks the map for the entry with the largest key that is not bigger than the requested offset. Because the map is sorted, this takes a handful of comparisons. If the answer is the segment with base offset 8,391,200, we now know that the record must be inside that segment, or the offset is beyond the end of the log.

Lookup 2: binary search in the offset index

The broker subtracts the base offset from the requested offset to get a relative offset, here 8,800. It then binary searches the segment’s offset index for the last entry whose relative offset is less than or equal to 8,800. That entry, say for relative offset 8,790, gives a byte position. Since the index is memory mapped, this search touches only a few memory pages and usually never reaches the disk.

Lookup 3: a short scan in the log file

From that position the broker reads batch headers one after another until it finds the batch that contains offset 8,400,000. It only reads headers, not whole payloads, so the scan is tiny. If the requested offset is smaller than the first index entry, the scan simply starts at position zero.

Sending the bytes

Once the position is known, the broker does not load the records into its own program memory. It asks the operating system to copy bytes from the file’s page cache directly to the network connection. This technique is called zero-copy, and it is one reason Kafka can serve enormous read rates on modest hardware. The broker also respects limits: the maximum bytes the consumer asked for, and the boundary of what is safe to expose, which is the high watermark, or the last stable offset for consumers that read only committed transactions.

When the offset does not exist

If the requested offset is lower than the oldest record still kept, or higher than the newest, the broker answers with an “offset out of range” error. The consumer then applies its reset policy: jump to the earliest offset, jump to the latest, or fail loudly. The index files are what make this check quick, because the first and last offsets are known from the segment names and the log end.

Time lookups reuse the offset index, so the two files work as a relay team.

The question being asked

A consumer program, an admin tool, or a stream processing job calls a lookup that says, “Return the earliest offset whose timestamp is greater than or equal to this time.” The answer is an offset. From that offset, everything continues exactly like chapter 6. This lookup is exposed to clients as a method that turns timestamps into offsets, and to operators through tools that list offsets by time and reset consumer groups to a date.

flowchart TD
  Q["Request: first offset at or after time T"] --> S1["Walk segments in order and pick the first one whose largest timestamp is at least T"]
  S1 --> S2["Binary search that segment's time index for the last entry with timestamp at most T"]
  S2 --> S3["Take the offset stored in that entry as a starting hint"]
  S3 --> S4["Use the offset index to turn that offset into a byte position"]
  S4 --> S5["Scan the log forward until a record has timestamp at least T"]
  S5 --> R["Return that record's offset"]
        

Step by step

Why a scan is still needed

The time index stores the largest timestamp seen so far, which is a summary, not a record-by-record truth. Suppose a record with a late timestamp appears early in the segment. The index entry after it will show that big timestamp, even though many later records have smaller ones. The lookup therefore uses the index only to land safely before the answer, and lets the final scan pick the exact first record. The result is correct even for messy clocks.

Edge cases you should know

Index files are born, grow, are frozen, and finally are removed together with their segment.

flowchart LR
  A["Created and preallocated"] --> B["Active: entries appended"]
  B --> C["Roll trigger fires"]
  C --> D["Trim indexes and flush"]
  D --> E["Closed: read only"]
  E --> F["Retention says too old"]
  F --> G["Renamed with a deleted suffix"]
  G --> H["Files removed after a delay"]
        

Birth

When a new active segment starts, Kafka creates its log file and preallocates both index files at their maximum size. The name of the segment is the offset of the first record it will receive. From this moment the segment grows only by appending.

Roll triggers

The time limit deserves a comment. On a topic with little traffic, a segment might never reach one gibibyte in a week, and without a time trigger the data in it could never be deleted, because only closed segments can be deleted. The optional jitter spreads rolls of many partitions over time so they do not all happen at the same second.

Closing: trim, flush, freeze

At a roll, Kafka trims the index files from their preallocated size down to the actual number of valid entries and flushes them to disk. It also writes one final time index entry that records the segment’s largest timestamp, so that the closed segment carries an accurate “newest data” label for retention. A new active segment then takes over.

Retention uses the time index

A background task on the broker regularly checks closed segments. With time-based retention, a segment is eligible for deletion when its largest timestamp is older than the retention period. It is not the file modified time, and it is not the smallest timestamp. This is a deliberately safe choice: a segment is removed only when even its newest record is old enough. With size-based retention, the oldest closed segments are removed until the partition fits under the limit. Deletion can also happen when the log start offset moves forward, for example after an administrator deletes records up to a point.

Deletion is two-phase

Kafka first renames the segment’s files with a special suffix that marks them as deleted, and removes them from its in-memory segment map right away, so readers no longer see them. A scheduled task deletes the renamed files after a short delay, sixty seconds by default. The delay avoids blocking request threads on file system work and gives in-flight readers time to finish.

The speed of index files comes from the operating system, and so do their biggest operational limits.

What memory mapping means

Kafka opens each index file and asks the operating system to map it into the broker’s address space. After that, reading or writing the index looks like reading or writing an ordinary array in memory. The operating system quietly loads the needed pieces of the file into the page cache, and writes changed pieces back to disk later. Kafka does not need to copy data between kernel and program buffers, and the index behaves like fast memory that happens to be backed by a file.

Warm and cold parts of an index

Most lookups ask for recent data, because most consumers are close to the end of the log. Kafka’s index search takes advantage of this. It runs the binary search in two zones: a small “warm” zone at the end of the index, which holds the newest entries and is almost always in memory, and the larger “cold” zone before it. Searching the warm zone first keeps the pages that recent readers need hot in the page cache, and stops occasional old reads from scattering random accesses across the whole file.

Memory is off the Java heap

Index pages live in the operating system’s page cache, not in the Java heap that the garbage collector manages. This is the reason Kafka administrators are told to give brokers a modest heap and leave the rest of the machine’s memory to the operating system. More free memory means more of the logs and indexes stay cached.

The hidden limit: memory map count

Every memory-mapped file uses one mapping area, and Linux limits how many mappings a process may hold. The setting is called the maximum map count, and a common default is 65,530. Every segment has two mapped index files, so a broker with many partitions and many segments can run out. When that happens, the broker fails with an out-of-memory error whose message says a map failed, even though plenty of memory is free. This confuses many teams the first time they see it.

A quick calculation shows how easy it is. A broker hosting 4,000 partition replicas with 10 segments each has 40,000 segments and 80,000 mappings, which is already above the default limit. The remedy is twofold: raise the operating system limit to a comfortably higher number, and avoid creating needlessly tiny segments.

File handles are a separate limit

Log files need open file handles, and brokers with many segments need a high limit on open files per process. Index files, once mapped, do not need their own long-lived handle, which is why the map count is the limit that usually bites for them, while the open file limit bites for the log files. Both should be raised on any serious broker.

Where you must not put these files

Log directories should be on local disks or on well-behaved block storage with a standard local file system. Network file systems can behave poorly with memory mapping and with the file rename and delete patterns that Kafka uses, and are a well-known source of strange failures.

Index files are a cache of facts that already exist in the log, so they can always be thrown away and rebuilt.

The key idea: the log is the truth

Everything in the index files can be recomputed by reading the log file. This single fact shapes all of Kafka’s recovery design. If an index is damaged, missing or suspicious, Kafka does not try to repair it byte by byte. It deletes it and builds a new one from the log.

Clean shutdown versus crash

When a broker stops cleanly, it flushes everything, trims indexes, writes a marker that says “clean shutdown”, and stores checkpoint files that remember up to which offset data is known to be safely on disk, called the recovery point. On the next start, the marker tells the broker that it can load segments quickly without deep checks.

After a crash or power loss, there is no marker. Kafka must assume that the newest segments may have half-written data. It takes each partition’s recovery point and examines the segments at and after it: it reads through the log, drops any incomplete or corrupted bytes at the end, and rebuilds the offset and time indexes for those segments from what remains.

flowchart TD
  A["Broker starts"] --> B{"Clean shutdown marker present?"}
  B -- "Yes" --> C["Load segments and run light index checks"]
  B -- "No" --> D["Read recovery point for each partition"]
  D --> E["Scan unflushed segments and cut off bad bytes"]
  E --> F["Rebuild offset index and time index from the log"]
  C --> G{"Index passes sanity check?"}
  G -- "Yes" --> H["Partition goes online"]
  G -- "No" --> F
  F --> H
        

What the sanity checks look for

Cost of rebuilding and how to control it

Rebuilding indexes means reading log data, so the time grows with the number and size of segments that need recovery. A broker that crashed with a large amount of unflushed data can take many minutes to come back. Kafka reduces this by recovering many partitions in parallel, using a configurable number of recovery threads per log directory. Setting this higher, up to the number of disks and cores that the machine can really use, is a common tuning step for brokers with many partitions.

Why Kafka does not fsync every write

Kafka by default leaves flushing to the operating system and relies on replication for durability. Your data is safe because other brokers hold copies, not because one disk was told to flush after every message. The index files follow the same philosophy: cheap to write, safe to lose, and always rebuildable. This is a rare case where “throw it away and rebuild” is the correct and fast design.

The offset and time indexes have to stay correct while the log around them is copied, cut and rewritten.

Truncation after leader changes

When leadership moves, a new leader may have less data than an old leader had. A follower that had extra records must cut its log back to match the new leader. Kafka uses leader epochs, numbered eras of leadership, stored in the epoch checkpoint file, to find the exact offset where two replicas stop agreeing. The follower then truncates its log to that offset.

Truncation must cut the indexes as well. Kafka removes offset index entries at or beyond the truncation point, and removes time index entries that point to removed offsets. If the truncation point lies inside a segment, that segment’s indexes are shortened. If it lies before a whole segment, that segment is deleted entirely. After this, indexes and log agree again, and appending continues from the truncation point.

Compaction and the offset map

For compacted topics, Kafka keeps the latest record per key. A background log cleaner reads old segments, builds an in-memory map of the newest offset per key, and writes new, smaller segments that contain only surviving records. This offset map is a different structure from the offset index. It is a temporary hash table used during cleaning, while the offset index is a permanent sorted file used for reading. Students often confuse the two, so keep them apart in your mind.

Each cleaned segment gets fresh index files built while it is written. Because the original offsets are preserved, the offsets in the cleaned segment have gaps. Since index entries are only signposts and lookups pick the nearest entry at or before the target, gaps do not break anything.

Records that are not data

Transactions add control records, which are markers that say a transaction committed or aborted. These consume offsets but are hidden from consumers, which is one more reason offsets in a partition are not always a smooth run of visible records. The lookup logic already handles this because it works with batch positions rather than counting visible records.

The other side files

When old segments move to cheap remote storage, their indexes have to travel with them.

The idea of tiered storage

Keeping months of data on expensive local disks is costly. Tiered storage, introduced through KIP-405, lets a broker keep only recent segments on local disks, the local tier, and copy closed segments to cheaper object storage, the remote tier. Consumers still read old data through the same Kafka interface, and the broker fetches what it needs from the remote tier when required.

What travels with a segment

When a segment is copied to the remote tier, Kafka uploads more than the log file. The offset index, the time index, the transaction index, the producer snapshot and the leader epoch information are uploaded too. Without the offset index, the broker would have to download and scan an entire remote segment to find one offset. With it, the broker can fetch just the small index, compute the byte position, and request only the needed byte range from remote storage.

A cache for remote indexes

Downloading an index for every remote read would be slow, so brokers keep a configurable local cache of index files for remote segments. Frequent readers of old data hit this cache. Sizing it well is part of running tiered storage, because a cache that is too small forces repeated downloads and higher latency for historical reads.

Only a few settings influence index files. Knowing what each one trades away is the real skill.

The settings that matter

Reading the trade-offs

How to choose segment size

A useful rule of thumb is to size segments so that each partition holds a healthy number of them, not just one or two, and not thousands. For a busy topic with a retention of a few days, the default one gibibyte is usually fine. For a low-volume topic, the time limit matters more than the size limit, because the size limit may never be reached. A topic that receives a few megabytes per day and keeps data for thirty days will hold roughly a week of data in each segment, and retention will lag behind by up to a week, since the newest record in a segment decides when it can go.

Choosing the timestamp type

Choose CreateTime when event time matters and you trust your producers’ clocks. Choose LogAppendTime when you want a clean, always-increasing arrival clock, for example on topics fed by many unreliable devices. Also consider limiting how far a record’s timestamp may differ from the broker’s clock. Newer Kafka versions offer separate limits for timestamps in the past and in the future, and rejecting extreme values protects retention and time lookups from bad data.

Index files hold no message payloads, but they still live next to data that must be protected.

What an index file can and cannot reveal

The offset index holds offsets and byte positions. The time index holds timestamps and offsets. Neither holds keys, values or headers. Even so, they reveal patterns: how fast data arrives, at what hours, and how large batches are. Treat them as sensitive operational metadata, especially in regulated industries, where activity patterns can be confidential.

Protect the directory, not just the topic

Kafka access control lists protect topics at the network and API level. They do nothing for someone who can read the broker’s disk. Anyone who can open the log directory can read the actual records from the log files, so the whole log directory needs tight operating system permissions, and the broker should run as a dedicated user with no unnecessary access.

Deletion and the right to be forgotten

Index files do not change what happens when data must be removed. Data disappears when its segment is deleted by retention, or when compaction removes older values for a key after a tombstone. Because deletion happens per segment, the newest record in a segment controls when the whole segment goes. Teams with legal deletion deadlines should set retention and segment roll times with that delay in mind, and should test that data is really gone from disk, not only from consumers’ view.

Index problems are rare, but when they appear they are confusing. Knowing the symptoms saves hours.

What to watch on a healthy broker

Messages you may see in the broker logs

Inspecting index files safely

Kafka ships a log dump tool that can open segment files and print their contents, including the offset index and the time index, and it has options to verify that an index is consistent with its log. Use it on a copy of the files, or on a stopped broker’s data, and remember that it can print record contents unless you ask it not to.

Good habits keep index files invisible. Bad habits make them the cause of a midnight incident.

Patterns that work

Anti-patterns to avoid

Offset and time lookups quietly power some of the most common operations in Kafka-based companies.

Let us follow one small, imaginary partition from its first record to a consumer’s fetch, using round numbers so every step is easy to check.

The setting

Imagine a topic called orders with a partition that has just started a new segment. The segment’s base offset is 1,000,000, so its files begin with that number padded to twenty digits. Producers send batches of ten records, and each record is about five hundred bytes, so a batch is roughly five thousand bytes. The index interval is the default, 4,096 bytes. All numbers in this example are illustrative, chosen to show the mechanism and not to match a real system exactly.

Writing the first few batches

Reading it back

Now a consumer asks for offset 1,150,004. The broker looks at the sorted map of segments and picks the segment with base offset 1,000,000, since that is the largest base offset not above the request. It subtracts to get relative offset 150,004 and binary searches the offset index. The search lands on the entry for the batch that starts at relative offset 150,000, which points to some byte position, say around seventy-five million. The broker reads the header of that batch, sees that it covers offsets 1,150,000 to 1,150,009, and knows that offset 1,150,004 is inside it. It sends that batch, and the consumer library drops the four records it already has.

flowchart TD
  R["Consumer asks for offset 1150004"] --> A["Pick segment with base offset 1000000"]
  A --> B["Relative offset is 150004"]
  B --> C["Binary search the offset index"]
  C --> D["Entry for relative offset 150000 gives a byte position"]
  D --> E["Read batch header at that position"]
  E --> F["Batch covers 1150000 to 1150009"]
  F --> G["Send the batch and the consumer skips the first four records"]
        

Now a time question

An engineer asks, “What is the first offset at or after 10:30 in the morning?” The broker skips older segments by checking their largest timestamps, and settles on our segment. The binary search in the time index finds the last entry that is not later than 10:30 and reports an offset, perhaps 1,073,000. The offset index turns that into a position. The broker reads forward until it sees the first record whose timestamp is at least 10:30, and returns its offset. The engineer resets a consumer group to exactly that offset and the replay begins.

Index files are small, but small things multiplied by thousands of segments deserve a plan.

How big are all the indexes together?

For every four kibibytes of log data, the broker can write at most one offset index entry of eight bytes and one time index entry of twelve bytes. That is a maximum of twenty bytes of index for every 4,096 bytes of log, a bit under half of one percent. If a broker holds twelve tebibytes of log data, its indexes will take up to roughly sixty gibibytes in the worst case. In practice it is usually less, because large batches produce fewer entries than the maximum, and the time index only gets an entry when the largest timestamp grows.

A capacity worksheet in plain words

A short example with numbers

Suppose a broker carries twelve tebibytes of local data on 3,000 partition replicas. With one gibibyte segments, that is about twelve thousand closed segments plus three thousand active ones, so about fifteen thousand segments and thirty thousand memory mappings. That is under half of a limit of 65,530, which is comfortable. Now imagine someone lowers the segment size to 128 mebibytes to make deletion “more precise”. The count becomes roughly a hundred thousand segments and two hundred thousand mappings, far above the limit, and the broker fails. One innocent-looking setting decided the outcome.

Tuning tips that actually help

Clearing away wrong pictures is as valuable as learning right ones.

Six wrong ideas about Kafka index files

Kafka’s sparse index compared with a database index

People who know databases often ask how this compares with a database index. The ideas are related, but the goals are different. A database index must support arbitrary lookups and constant updates in the middle of the data. Kafka’s log is written only at the end, so its indexes can be much simpler.

How the design grew over time

The offset index has been part of Kafka from its early years, because reading from an arbitrary offset was always a core promise. The other pieces arrived as needs appeared. The approximate order is shown below, and exact version numbers should be checked against the official release notes for your installation.

Three habits of engineers who never fear index files

The first habit is to treat the log as the single source of truth and the indexes as disposable helpers. When an index looks wrong, they do not panic or hand-edit bytes. They stop the broker cleanly, remove the damaged index files of one segment, and let Kafka rebuild them from the log during startup. This habit turns a scary situation into a routine, well-understood maintenance step that takes a few minutes and risks no data.

The second habit is to watch the operating system, not just Kafka. Most production surprises connected to indexes come from limits outside Kafka itself: the memory map count, the open file limit, the free memory left for the page cache, and the speed of the disks. Engineers who add these numbers to their dashboards and alerts find problems days before their users do, because segment counts grow slowly and predictably as topics, partitions and retention settings change.

The third habit is to keep timestamps honest. Because retention, time-based rolling and seek by time all depend on record timestamps, one careless producer can quietly distort all three. Teams that validate client clocks, limit how far a timestamp may differ from the broker’s clock, or choose broker-assigned time for untrusted sources enjoy predictable disk usage and trustworthy replays. This is a small amount of discipline that pays back every time an incident needs a clean replay from a known moment.

What to remember for real-world work

When something is slow or strange, follow the questions in order. Is the request asking for an offset or for a time? Is the target segment on a local disk or in the remote tier? Are the index pages in memory? Is the machine near its limits for mappings, handles or disk? Most puzzles about “index problems” end up being answered by one of these four questions, and none of them requires touching the index files themselves.

Short, direct answers to the questions that come up most often.

A dense index would be many times larger and would not fit comfortably in memory for big partitions. A sparse index is tiny, and the short forward scan after each lookup costs almost nothing, so the trade is very favorable.

No. Followers replicate record batches and build their own index files while appending. Since the log is the source of truth, each replica can always produce a correct index by itself.

The broker rebuilds it from the log at startup. This can take time on large segments, but no data is lost, because the index holds no information that the log does not already have.

Kafka preallocates index files to their maximum size when a segment is created and trims them when the segment is closed. The size you see is reserved space, not stored entries.

No. It stores the largest timestamp seen so far at the moment an entry is written, and only when that value grows. This keeps it sorted, small and searchable even when producer clocks disagree.

Common reasons are that the segment is still the active one, that a record in the segment has a far-future timestamp, or that the segment’s newest record is still younger than the retention period. Deletion works on whole closed segments, judged by their largest timestamp.

The offset index is a permanent sorted file that helps readers find a position in a segment. The offset map is a temporary in-memory table the log cleaner builds to know the latest offset for each key while compacting. They serve different jobs.

No. Positions in the offset index are stored in four bytes, which limits a segment file to about two gibibytes. This is why the segment size setting has an upper bound.

Each segment’s two index files use memory mappings, and the operating system limits how many mappings a process can hold. When the limit is reached, mapping fails and the broker reports an out-of-memory error unrelated to free memory.

It picks the first segment whose largest timestamp reaches the requested time, searches that segment’s time index for the last entry not later than the time, converts that offset to a byte position with the offset index, and scans forward to the first record that matches.

Almost never. The default already keeps the post-lookup scan very short. Look first at page cache size, disk speed, fetch sizes and network settings.

No. Consumers only talk to brokers through the Kafka protocol. The broker uses its index files internally to answer fetch requests and offset-by-time requests.

Every Kafka partition is a long, ever-growing log cut into segments, and every segment carries a small family of files that make the log searchable. The offset index turns an offset into a byte position. The time index turns a timestamp into an offset. Both are sparse, sorted, memory mapped and always rebuildable from the log, which makes them cheap to write, quick to read and safe to lose. Once you see the log as the single source of truth and the indexes as rebuildable signposts, most of Kafka’s storage behavior, from retention to recovery to tiered storage, becomes easy to reason about.