One quiet memory decision can make a Kafka cluster feel lightning fast or painfully slow.

Apache Kafka is a system that stores streams of messages and hands them to programs that want to read them. Many teams notice that Kafka can serve enormous amounts of data using fairly plain machines. The secret is not magic. Kafka keeps its own memory use small and trusts the operating system to keep recently used file data in spare memory. That spare memory is called the page cache.

A workload is called read-heavy when many more bytes leave the brokers than enter them. This happens when one topic feeds many different applications. One producer writes an order event, and then billing, shipping, fraud checks, search indexing, dashboards, and a data lake all read that same event. Every extra reader multiplies the reading work, while the writing work stays the same.

If those reads are answered from memory, the disks stay calm and delivery is quick. If those reads fall through to the disks, every reader waits in line for the disk, and the disk becomes the bottleneck. Sizing the page cache is the act of deciding how much memory you leave free so that the reads land in memory.

The gap between memory and disk is enormous. Even fast solid state drives are many times slower than memory when a hundred readers ask at once. So a cluster with a well sized cache can serve the same traffic with fewer machines and fewer surprises, and that is why the topic deserves a full guide.

Before sizing anything, you need a picture of what a read actually is.

Kafka organizes messages into topics. A topic is a named stream, like a labelled conveyor belt. Each topic is cut into partitions, and each partition is an ordered list of messages that only grows at the end. Every message in a partition has a number called an offset, which is simply its position in line: first, second, third, and so on.

A consumer is a program that reads messages. It remembers the offset it has reached and asks the broker for the next batch starting from there. A broker is one Kafka server. Each partition lives on several brokers as copies, and one copy is the leader. Readers talk to the leader unless the cluster is set up to allow reading from a nearby copy.

On disk, a partition is stored as a series of files called segments. Only the newest segment is being written. Older segments are frozen and read only. Beside each segment sits a small index file that helps Kafka jump quickly to a given offset without scanning everything.

When a consumer asks for messages, the broker finds the right segment, uses the index to find the spot, and sends a batch of bytes back. Notice that the broker does not build the answer. It only copies existing bytes from a file. That simple fact is what lets the page cache do so much of the work.

It is not a Kafka feature. It belongs to the operating system.

Computers read files from disks, but disks are slow. To hide this slowness, the operating system keeps copies of recently used file pieces in memory. A piece is called a page, and it is usually four kilobytes. The set of all such pages is the page cache. When a program asks for file data, the system first looks in the page cache. If the page is there, the answer comes back at memory speed. This is called a cache hit. If it is missing, the system reads it from the disk, stores it in the cache for next time, and then answers. This is a cache miss.

You can see the page cache on any Linux machine. Tools that show memory usage report a line for buffers and cached data. Many beginners panic when they see almost no free memory on a healthy Kafka broker. That is normal. The system fills unused memory with cache on purpose, because empty memory is wasted memory. The important number is how much memory is available for programs when they need it, since cache can be given back instantly.

flowchart LR
  A["Consumer asks for data"] --> B["Kafka broker"]
  B --> C{"Page in memory?"}
  C -- "Yes: cache hit" --> D["Send bytes from memory"]
  C -- "No: cache miss" --> E["Read page from disk"]
  E --> F["Keep page in cache"]
  F --> D
        

The diagram shows the two roads. Every read takes one of them. The cache hit road is short and cheap. The cache miss road is longer and uses the disk, which is shared by everyone. Your goal as an operator is to make the short road the common road.

Kafka chose to lean on the page cache instead of building its own cache.

Many databases build a large cache inside their own process. Kafka does not. The designers reasoned that the operating system already has an excellent cache, and that a second cache inside the Java process would just store the same bytes twice. Storing them twice wastes memory. Worse, a big cache inside a Java program creates lots of objects, and the garbage collector must sweep them, which can pause the program.

By keeping data in the page cache, Kafka gets several gifts at once. The cache lives outside the Java heap, so it does not add garbage collection work. The cache survives when the Kafka process restarts, because it belongs to the operating system. That means a restarted broker can come back with a warm cache instead of a cold one. And the operating system is very good at deciding which pages to keep.

This design bet is the reason sizing memory is different for Kafka than for many other systems. You do not hand Kafka a giant heap. You give it a modest heap and leave the rest of the machine’s memory alone so that the operating system can use it as cache.

Reads are fast mostly because writes put the data in the right place first.

When a producer sends messages, the leader broker appends them to the end of the newest segment file. It does this with an ordinary file write. The operating system accepts the bytes into the page cache and answers right away. The bytes are marked dirty, which simply means they have not yet been saved to the disk. Later, a background job of the operating system writes dirty pages to the disk at a convenient time.

Kafka does not force each write to the disk by default. Instead, it relies on copies. Other brokers hold replicas of the same partition, so if one machine loses its memory in a crash, the data still exists elsewhere. This is a deliberate choice that trades a small risk on one machine for far greater speed, protected by replication across machines.

The important consequence for reading is this. Fresh data is already sitting in the page cache the moment it is written. Nobody has to load it. A consumer that reads new messages seconds later finds them in memory. This is why sizing focuses on how long fresh data stays in the cache before newer data pushes it out.

A cache hit gets even cheaper because Kafka can skip most copying.

In an ordinary program, sending a file over a network involves several copies. The data is read from disk into a kernel buffer, then copied into the program’s memory, then copied into another kernel buffer for the network card, and finally sent. Each copy burns processor time and memory bandwidth.

Kafka avoids this with a system feature commonly called zero-copy. The broker asks the operating system to send bytes from a file straight to the network connection. The data moves from the page cache to the network card without visiting the Java program at all. The broker’s own processor barely notices the transfer, and the Java heap stays quiet.

This is why cache hits matter twice. First, they avoid the slow disk. Second, they allow the cheap zero-copy road, where the processor does almost nothing. A single broker can then push out a very large stream of data to many readers, limited mostly by the network card.

flowchart LR
  subgraph Ordinary["Ordinary way: many copies"]
    D1["Disk"] --> K1["Kernel buffer"] --> P1["Program memory"] --> S1["Socket buffer"] --> N1["Network"]
  end
  subgraph ZeroCopy["Kafka zero-copy way"]
    PC["Page cache"] --> N2["Network card"]
  end
        

Later chapters will show that a few common settings quietly turn zero-copy off. Encryption of the connection is the best known one. When that happens, the broker must touch every byte, and the cost of reading rises sharply. Keep this in mind, because it changes how much capacity a read-heavy cluster really has.

Not all readers are equal. Two kinds behave very differently for the cache.

A tail reader is a consumer that keeps up with the newest data. It reads messages moments after they were written. Because fresh messages sit in the page cache, tail readers almost always get cache hits. They are the happy customers of Kafka.

A catch-up reader is a consumer that is behind. Maybe it was switched off for maintenance, maybe it was slow because of a bug, or maybe it is a brand new application that wants to replay yesterday’s history. Such a reader asks for old offsets. Old data may no longer be in the cache, because newer data has pushed it out. So the broker has to read it from disk.

The trouble does not stop at the slow disk read. When the broker loads old pages into the cache for the catch-up reader, those pages take room. That room comes from other pages, which might be the fresh pages that the tail readers need. This is called cache pollution, and it is the main enemy of a read-heavy cluster. One greedy catch-up reader can make everybody else slower.

Before you size a cache, describe the workload in numbers and words.

Read-heavy does not have a single shape. Some clusters have a modest write rate and a huge number of consumer groups. Others have one enormous consumer group with hundreds of members. Some serve mostly tail readers, while others serve batch jobs that scan hours of history. Each shape needs a different answer.

The first idea to grasp is fan-out. Fan-out is the number of separate groups that read the same data. If ten groups read the same topic, the broker sends each message ten times, plus copies to followers. The good news is that all ten reads hit the same cached pages. The cache holds one copy, and ten readers share it. So fan-out raises network and processor work, but it does not raise the amount of cache you need, as long as all groups read at roughly the same position.

The second idea is spread. Spread is how far apart the readers are inside the log. If all ten groups are at the tail, they share one small hot area. If they are scattered across the last six hours, the hot area is six hours wide. Cache need grows with spread, not with the number of readers.

flowchart TD
  W["Producers write new data"] --> L["Partition log in page cache"]
  L --> G1["Billing group at the tail"]
  L --> G2["Search group at the tail"]
  L --> G3["Dashboard group a few minutes behind"]
  L --> G4["Nightly job reads old history"]
  G4 -. "loads old pages" .-> D["Disk"]
        

The whole log is huge. The part you must keep in memory is much smaller.

Suppose your brokers store seven days of data. That could be tens of terabytes. Nobody expects the page cache to hold all of that. What matters is the hot set: the slice of data that readers touch again and again within a short time. For a cluster full of tail readers, the hot set is the last few minutes of writes on each broker.

The working set is a slightly larger idea. It is the data that active readers need across a chosen period, such as one busy hour. If the working set fits into the cache, nearly every read is a hit. If the working set is larger than the cache, the system keeps throwing out pages just before they are needed again, and the hit rate falls.

This is why capacity planning for the cache starts with a question about people and programs, not about machines: how far behind is the slowest reader we still want to serve from memory? That answer, converted into a length of time, becomes the width of your hot set.

A Kafka machine has memory for three things: the operating system, the Java heap, and the cache.

Java programs keep their objects in a region called the heap. Kafka’s heap holds bookkeeping such as partition state, request objects, and buffers. It does not hold message bodies for long. Because of this, a modest heap is usually enough. Many teams run brokers with a heap in the range of a few gigabytes, often around six. Very large heaps bring longer garbage collection pauses and steal memory from the cache for no benefit.

The operating system itself needs some memory for its own work, network buffers, and running other small services such as monitoring agents. Whatever is left is what the page cache can use. So the rule is simple to say: give the heap only what it needs, keep the operating system happy, and let the rest be cache.

flowchart LR
  subgraph Machine["One broker machine, for example 64 GB"]
    H["Java heap: about 6 GB"]
    O["Operating system and agents: about 4 GB"]
    C["Page cache: the remaining memory, about 54 GB"]
  end
        

Other services on the same machine also compete. A log shipper, a security scanner, or a co-located application can quietly consume memory. On a machine where the cache is the star, every side program has a cost that appears as extra disk reads.

You can size the cache with a short chain of easy steps and a little arithmetic.

The method turns the question how much cache do I need into a multiplication. The main idea is that the cache must hold the newest data for as long as your slowest protected reader might lag. So you multiply the rate at which new data arrives on one broker by the length of time you want to cover, and then add a safety margin.

Let us walk through the method once with round numbers.

Imagine a cluster of six brokers. Producers send a combined 200 megabytes per second of new data. The replication factor is three, meaning every message is stored on three brokers. So the cluster as a whole absorbs 600 megabytes per second of writes once copies are counted. Divided across six brokers, each broker takes in about 100 megabytes per second.

The team decides they want to serve any reader that is up to five minutes behind straight from memory. Five minutes is 300 seconds. So each broker needs to keep about 100 times 300, which is 30,000 megabytes, or roughly 30 gigabytes of fresh data. With a safety margin of two times, they aim for about 60 gigabytes of page cache.

Add a heap of about 6 gigabytes and about 4 gigabytes for the operating system. The total is around 70 gigabytes. In practice the team would pick the next standard machine size above this, such as one with 96 or 128 gigabytes, and enjoy extra room for growth.

Copies and partitions change how much data each broker must hold in memory.

The replication factor is the number of copies of each partition. A factor of three is the common choice for important data. Every copy is written by a different broker, so each of those brokers needs to fit its own fresh data in its own cache. This is why the write rate in the sizing method is multiplied by the replication factor before dividing by the number of brokers.

The number of partitions matters in a quieter way. Each partition is a separate set of files with its own active segment. With thousands of partitions on a broker, the fresh writes are spread over thousands of file locations. Total fresh bytes do not change, but the pattern becomes scattered. Scattered access can weaken the benefits of sequential disk writing and read ahead. Extra files also mean extra open file handles and extra index pages.

Spreading load evenly is the last piece. If one broker leads many busy partitions, its cache need is larger than the average you calculated. A cluster with uneven leadership will have one hot machine whose cache is too small while others sit half empty. Balanced leadership is therefore part of cache sizing.

When memory is full, something must leave. The choice matters.

The page cache cannot grow without limit. When new pages arrive and memory is short, the operating system removes older pages. Linux roughly keeps two lists. Pages touched only once go on a list of inactive pages. Pages touched more than once move to a list of active pages. When space is needed, the system usually evicts pages from the inactive list first.

This design gives Kafka some natural protection. A page read by many groups becomes active and tends to stay. A page read once by a lonely catch-up reader stays inactive and leaves first. In theory, that already limits pollution from one-time scans.

Theory has limits, though. A very large scan can flood memory faster than pages are promoted, and it can squeeze out pages that would have been used soon. In addition, fresh writes are new pages that are not yet active. If a burst of old reads arrives at the same time as a burst of writes, fresh data can be evicted before consumers reach it. Then the tail readers, who should have been safe, get cache misses.

flowchart LR
  N["New page arrives"] --> I["Inactive list"]
  I -- "touched again" --> A["Active list"]
  I -- "memory is short" --> E["Evicted"]
  A -- "long unused" --> I
        

A slow reader hurts more than its own speed.

Consumer lag is the number of messages a consumer group still has to read. A small, steady lag is normal. A growing lag means the group is falling behind. Every minute of lag pushes the group’s position farther into the past, and older data is more likely to have left the cache.

The dangerous moment is the turning point. While the group is inside the cached window, its reads are cheap. Once it slips outside, each read costs a disk trip. A disk trip is slower, so the group falls further behind, which means even more disk trips. This is a downward spiral. The reader that needed cache help the most is the one that has lost it.

The spiral spreads. Disk reads from the slow group compete with disk writes and with other readers’ misses. Latency rises for everyone. Producers may see slower acknowledgements, and followers may fall out of sync. A single lagging group can therefore create trouble far from where it started.

The lesson for sizing is to pick your protection window with the spiral in mind. Extra cache gives slow groups more time to recover before they cross the danger line. It works like a shock absorber.

Some settings look harmless but make every read cost far more.

Chapter 6 described zero-copy, which lets bytes flow from the cache to the network card without touching the Java program. A few common features force the broker to read, transform, and copy data. When that happens, the processor and memory bandwidth become new limits, and the cheap road is closed.

Encrypting client connections with TLS is the most common cause. To encrypt data, the broker must load each byte into user space, scramble it, and send the scrambled result. The page cache still helps by avoiding disk reads, but the cost per byte is higher. Clusters that enable encryption for all traffic often need more processors and sometimes more brokers to move the same volume of data.

Another cause is the conversion of message formats for old clients. If a consumer speaks an old version of the protocol, the broker may have to convert stored messages into the old format before sending. This is called down-conversion, and it defeats zero-copy for those readers. Keeping clients reasonably up to date avoids this hidden tax.

A third cause is when the broker must change how data is compressed. Normally producers compress batches and the broker stores them unchanged, so readers receive the same compressed bytes. If broker settings force recompression, the broker burns processor time on every batch.

The cost of a cache miss depends on what is under the cache.

A spinning hard disk has a moving arm. Reading data in a straight line is fine, but jumping around is slow. Kafka’s access pattern is mostly sequential, which is why hard disks worked well for it for years. Still, when many readers each ask for different places, the arm must jump between them, and speed falls a lot.

A solid state drive has no moving parts and handles scattered reads much better. An NVMe drive, which connects through a faster path, is better still. On such disks a miss is far less painful, and the cache can be smaller for the same experience. Yet even the fastest drive is slower than memory, and it has limited throughput shared between writes, replication, and reads.

Cloud volumes add another twist. Network attached volumes have limits on operations and throughput per volume, and bursts may be allowed for a short time and then throttled. A cache miss on such a volume competes with these limits. A miss that costs a few milliseconds in a test may cost much more when the volume is throttled.

flowchart TD
  M["Cache miss"] --> Q{"Which storage?"}
  Q --> H["Spinning disk: slow when readers scatter"]
  Q --> S["Solid state: good with scattered reads"]
  Q --> V["Network volume: limited by throughput and operation caps"]
  Q --> L["Local NVMe: fastest, but tied to the machine"]
        

The operating system guesses what you will read next, and Kafka’s pattern makes the guess easy.

When a program reads a file in order, the operating system notices and starts loading the next pieces before they are requested. This is called read-ahead. For a catch-up reader scanning a partition from old to new, read-ahead can hide much of the disk delay, because the next pages are already arriving when the reader asks for them.

Read-ahead has a cost as well. It pulls pages into the cache that may not be needed, and those pages take space. It works best for a few long sequential scans. It works poorly when hundreds of readers each request small pieces from different files, because the system must guess for each and its guesses fill the cache with pages nobody wants.

There is a tuning knob for the size of read-ahead on each disk device. Larger values help long scans and hurt scattered reads. Kafka’s default patterns rarely need extreme values, but teams with many replays sometimes tune it after measuring. As always, measure before changing.

The small helper files also live in the cache, and they need room too.

Each segment has index files. An offset index maps message numbers to positions in the segment file. A time index maps timestamps to offsets. When a consumer asks for a specific offset, the broker checks the index to find where to start reading. These index files are small compared with data files, but they are used on every fetch.

Kafka reads indexes through memory mapping, a feature that lets a file appear as part of the program’s memory. Behind the scenes, the pages of the index still live in the page cache. If an index page is not cached, the lookup causes a disk read before the data read can even begin. That doubles the delay of a miss.

Since indexes are tiny, keeping them cached costs very little. But with thousands of partitions and long retention, the total index size grows. The number of memory mappings a process may hold is limited by a system setting, and brokers with many segments need that limit raised. If the limit is too low, brokers can fail in surprising ways when they try to open more segments.

Fresh data has to reach the disk eventually. How that happens affects reads.

Dirty pages are cached pages that have changed but are not yet saved. The operating system has two levels of control. Below a certain amount of dirty data, a background helper writes pages out gently. Above a higher limit, programs that write are forced to wait while the system catches up. Both limits are set as a share of memory.

On a machine with a very large amount of memory, a share can become a very large absolute number. If the limits are too high, dirty data can pile up into a huge lump, and then the system writes it all in a rush. During that rush the disk is saturated and reads that miss the cache must wait behind the writes. Latency spikes appear at regular intervals.

Many operators lower the background limit so that flushing starts earlier and proceeds smoothly. Smaller and steadier writes leave more disk time for reads. Others use the byte based versions of the limits so that they do not scale with the total memory. The right values depend on disk speed and write rate, so measure the effect.

sequenceDiagram
  participant P as Producer writes
  participant C as Page cache
  participant B as Background flusher
  participant D as Disk
  P->>C: New bytes marked dirty
  C->>B: Dirty level rises
  B->>D: Gentle early writeback
  Note over C,D: Steady flushing keeps disk time free for reads
  P->>C: Many more writes arrive
  C->>D: Forced flush if limit is passed
  Note over P,D: Writers now wait, and misses queue behind writes
        

When memory gets tight, the system must choose between cache pages and program pages.

Swap is disk space that the operating system uses when memory is short. It can move rarely used program memory out to disk to free room. For most programs, swapping is a safety net. For a broker, swapping the Java heap is a disaster, because heap pages may be needed at any moment and bringing them back from disk takes ages. Long pauses cause the broker to miss heartbeats, and the cluster may think it is dead.

A setting called swappiness tells the system how eager to be about swapping program memory compared with dropping cache pages. Kafka guidance commonly suggests setting it very low so that the system prefers dropping cache over swapping the heap. Cache pages can be reloaded from files, while heap pages have nowhere else to come from.

Memory pressure from other sources also matters. If another process grows and uses memory, the cache shrinks. A shrinking cache is a silent loss of read capacity. You will see it as rising disk reads and slower fetches, long before any error message appears.

In containers, the page cache may count against the limit you gave the process.

Many teams run Kafka in containers on orchestration platforms. Containers are boxes with limits on memory and processor. On Linux, these limits are enforced by a feature called control groups. In many setups, the memory used by the page cache for files that the container reads and writes is charged to that container’s memory budget.

This has two effects. First, the cache is limited by the container’s limit, not by the host’s total memory. A host with 256 gigabytes but a container limit of 16 gigabytes gives Kafka roughly 16 minus heap minus overhead as cache. Second, when the container gets close to its limit, the system reclaims cache pages inside the container, and an oversized heap can push the container over the edge and cause it to be terminated.

The safe approach is to set the memory limit deliberately as heap plus system needs plus the cache target you computed. Do not set a limit equal to the heap and expect the cache to appear from nowhere. Also remember that other containers on the same host may compete for the host’s total memory.

Cloud instance families trade memory, processors, network, and disks in different ratios.

Public cloud providers offer families of machines. Some have a lot of memory per processor, some have a lot of processors, and some have fast local disks. For a read-heavy Kafka cluster the important figures are memory size, network bandwidth, and disk throughput. Processors matter mostly when encryption or compression conversions are involved.

Sizing the cache in a cloud setting means choosing the family and size that gives the memory you computed while also providing enough network. A machine with lots of memory but a small network card cannot send data to many readers, and the cache advantage is wasted. Likewise, a machine with a huge network card and little memory forces readers to the disk.

Instances with local NVMe drives deliver fast misses but lose data if the machine is replaced, so the cluster must rebuild those replicas by copying from other brokers. That rebuild is a burst of catch-up reading that pollutes the caches of the source brokers. Network attached volumes keep data across replacements but bring throughput limits. Neither is perfect.

A few low-level choices can help or hurt the way cache pages behave.

On large servers with several processor sockets, memory is divided among the sockets. Reaching memory attached to another socket is slower. This design is called NUMA. If a Java process allocates most of its memory on one socket while the page cache fills evenly, odd imbalances can appear, and some machines report memory pressure on one socket while others have plenty free. Teams with big servers sometimes spread memory evenly or bind the broker carefully, and they test the results.

Transparent huge pages let the system use larger memory pages to save bookkeeping. For some workloads this is helpful, but for others it causes unpredictable pauses when the system compacts memory. Many production guides suggest checking this setting and, if pauses appear, turning it to a less eager mode.

The file system also matters. Kafka guidance has long favored file systems such as XFS and ext4 for their good handling of large sequential files. Mount options that skip updating the last access time on every read avoid extra writes that would otherwise happen each time a file is read. That small change removes needless disk work on a read-heavy machine.

How long you keep data and how you cut it into files both shape memory use.

Retention is the rule that says how long Kafka keeps messages before deleting old segments. Long retention does not directly demand more cache, because old data that nobody reads does not stay in memory. The cache holds what is being read, not what is being stored. So a seven day retention with tail readers needs the same cache as a one day retention with tail readers.

Long retention does create a risk, though. It makes replays possible, and replays are exactly what pollutes the cache. A cluster that keeps a month of history invites new applications to start from the beginning. If those replays run on the production brokers, they drag old pages through the cache. So retention is a policy decision that must be paired with a plan for where replays run.

Segment size also matters a little. A segment is closed and a new one started when the file reaches a size or an age. Very small segments create many files and many index entries, which increases open files and mappings. Very large segments make deletion coarse, since Kafka deletes whole segments. A middle path suits most clusters, and the default is a sensible place to start.

Moving cold data to cheaper storage changes what the local cache must do.

Tiered storage is a feature in newer Kafka versions that lets brokers keep only recent data on local disks and move older segments to cheaper remote storage, such as an object store. Readers who need old data are served from the remote tier. Readers who need recent data are served from the local disk and the page cache as before.

For cache sizing this is good news. The local tier holds a short window, so replays from far back do not walk through local files and do not pull old pages through the local cache in the same way. Reads from the remote tier are handled by a separate path with its own limits. The heavy replays are moved away from the hot area.

There are trade-offs. Remote reads are slower than local reads, and the remote tier has its own limits on throughput and request rates. Teams must watch how the remote fetches behave and make sure a big replay does not overwhelm network bandwidth on the broker. Check the current documentation and the maturity level for your version before you depend on this feature, since details have evolved over time.

flowchart LR
  T["Tail readers"] --> LC["Local disk and page cache: recent window"]
  R["Replay readers"] --> RS["Remote object storage: older segments"]
  LC -. "old segments move out" .-> RS
        

More brokers can answer reads if the cluster lets them.

By default, consumers read from the leader of each partition. In a read-heavy cluster, that concentrates all the read traffic on leaders. A newer feature lets consumers fetch from a follower replica that is closer to them, for example in the same cloud zone. This was designed mainly to reduce network cost and latency across zones, but it also spreads the read work across more brokers.

From a cache viewpoint, followers already hold the same fresh data in their caches, since they copied it as it arrived. So a follower is also a good place to read tail data. Spreading reads across leaders and followers puts more caches to work. The catch is that followers can lag slightly behind the leader, so readers may see data a moment later than they would from the leader.

Spreading reads by adding partitions can also work. More partitions mean more leaders, and the leaders can be placed on different brokers. But partition count is a heavy decision with its own costs, and it cannot be undone easily, so consider it carefully.

Consumers decide how much to ask for, and that changes cache behavior.

Consumers control how much data they ask for in each request and how long the broker may wait to gather it. A minimum fetch size tells the broker to wait until at least that many bytes are ready. A maximum wait time tells it how long to wait for those bytes. A maximum fetch size caps the response. These are consumer settings, and they can be different for each application.

Tiny fetches mean many requests, each with overhead. Huge fetches mean big reads that may load many pages at once. A moderate size lets read-ahead work well, keeps requests efficient, and does not flood the cache. For tail readers, waiting a little to gather a bigger batch reduces the request count and the processing load on the broker, at the price of a small delay.

Batch jobs that scan history can use large fetches and should be limited in speed by quotas, which the next chapter explains. The main point is that consumer settings are part of capacity planning. A team that changes them without telling the platform team can change the read profile of the cluster overnight.

Sometimes the best cache plan is to limit who can take from it.

Kafka supports quotas, which limit how many bytes per second a client or a group of clients may read or write. A quota on fetch traffic can cap a batch job so that it cannot pour data out of the brokers at full speed. Slower catch-up reading means less disk contention and slower pollution, giving the tail readers breathing room.

Quotas work best when they are assigned by identity. Each application uses its own client identity or authenticated user, and the platform team assigns each one a share. Applications that are business critical get generous limits, and experimental or replay applications get lower limits. When someone needs a bigger share, they ask, and the change is discussed in the open.

There is a more radical form of isolation: separate clusters. A dedicated cluster for replays, analytics, or data lake export can copy data from the main cluster using a single well behaved reader. All the other analytics applications then read from the copy. The main cluster serves only its tail readers, and its cache stays healthy.

flowchart LR
  P["Producers"] --> M["Main cluster: tail readers only"]
  M --> S["Critical apps at the tail"]
  M -- "one steady copier" --> X["Analytics cluster"]
  X --> A["Replays, experiments, batch jobs"]
        

A restarted broker remembers, but a replaced broker starts from nothing.

When a broker process restarts, the operating system’s cache usually remains intact, so the broker comes back warm. This is one of the benefits of using the page cache rather than the heap. But a machine reboot empties the cache, and a replaced machine has never had a cache at all. The broker begins with cold pages and every read is a miss until the cache fills.

A cold broker faces a second problem: it must catch up. After being away it has to copy the messages it missed from the leaders. This is a large catch-up read on the leaders, and it can pollute their caches. Rolling restarts of a whole cluster can therefore create a chain of catch-ups, one after another, each disturbing the caches of the neighbors.

Good practice is to restart one broker at a time and wait until it is fully in sync and its cache is reasonably warm before the next. Throttling the replication speed during recovery also helps. Leaders should not be moved back to a freshly restarted broker until it has warmed up, since it would begin serving reads with a cold cache.

A cache that is barely big enough today will be too small next quarter.

Traffic grows. More producers appear, existing ones send more data, and new consumer groups join. Each increase in the write rate moves your hot window boundary closer. If writes double, the same cache now covers half the time. A reader that was safely five minutes behind may now be outside the cache at only two and a half minutes of lag.

Spikes create the same problem in a shorter time. A sale event or a bug that floods a topic can multiply the write rate for a few hours. During a spike the cache covers less time, and any slow reader is in more danger. Sizing for the average day and ignoring the peak day is a common cause of surprise trouble.

A helpful practice is to size for the busiest realistic hour and then keep a headroom target, such as always keeping the hot window at least twice as long as the slowest protected reader normally lags. Review that number every quarter and after any large launch. Adding brokers reduces per-broker write rate and gives the same effect as adding memory.

You cannot manage a cache you cannot see, so pick a few honest signals.

Kafka does not report a cache hit rate, because the cache belongs to the operating system. So you learn about it from signs around it. The clearest sign is disk read activity on brokers. On a healthy read-heavy cluster serving mostly tail readers, the disks show heavy writes and very little reading. When reads climb, the cache is missing.

The second sign is fetch latency, the time the broker takes to answer a consumer or follower request. Kafka exposes request timing metrics, and they usually show a rise when misses start. Watch the high percentiles, not the average, because a few slow reads hide inside a healthy-looking mean.

The third sign is consumer lag measured in time, not only in message counts. A lag of a million messages means little until you know how fast messages arrive. Convert it to minutes and compare it with your cache window. The fourth sign is memory itself: available memory, cache size, swap use, and page fault activity on each broker.

Some tools let you peek inside and answer the question directly.

On Linux, there are tools that report how much of a given file is currently in the page cache. Pointing such a tool at the newest segment files shows what share of them is resident in memory. Others trace cache hits and misses over time and print a summary every few seconds. These tools are often part of standard performance packages, and using them is safe on a running system when done with care.

Simple system reports are useful too. A memory report shows the size of the cache and available memory. A virtual memory report shows swapping and block reads and writes over time. A disk report shows read and write rates and how busy each device is. Comparing these while a suspicious consumer runs tells you a lot in a few minutes.

A good experiment is to start a controlled replay from a test client on one broker and watch. If disk reads jump and fetch times worsen for other clients, you have learned how sensitive your cluster is to catch-up reading. This is safer to learn on a quiet afternoon than during an incident.

Sharing a cache between teams raises fairness and safety questions.

The page cache is shared by everything that reads files on the broker machine. There is no per-team slice. When many teams share a cluster, one team’s replay can hurt another team’s speed. This is a fairness problem more than a secrecy problem, because access to the data itself is controlled by authentication and permissions inside Kafka.

Encryption in transit protects data on the wire and, as discussed, costs some read efficiency. Encryption at rest, provided by the storage layer or the file system, means pages in the cache are already decrypted for use by the operating system. Cached data is protected by machine access controls, so restrict who can log into brokers and who can read the data directory.

Access rules, called ACLs, decide which users can read which topics. Combined with quotas, they give each team a boundary in both permission and bandwidth. Good governance assigns each application its own identity, so that when a read pattern hurts the cache, you know whom to call.

Every design choice has a price. Here is the honest summary.

Relying on the page cache made Kafka simple and quick. Reads for recent data are cheap, restarts are gentle, and garbage collection stays under control. Engineers do not need to write cache logic, and the operating system improves over time.

The price is control. You cannot pin the hot pages, cannot mark some topics as more important than others in the cache, and cannot easily prevent one reader from changing what is stored. Your tools are indirect: more memory, better isolation, limits on readers, and careful operations.

A short list of habits that keep read-heavy clusters healthy.

Good cache sizing is a habit, not a one-time calculation. The practices below come from the ideas in earlier chapters. None is exotic. Most teams that follow them see calmer clusters and fewer late-night incidents.

These mistakes appear again and again in real clusters.

Some popular beliefs about Kafka memory are simply wrong.

Three short stories show how the ideas play out.

Large companies that run Kafka at great scale, such as social networks, streaming services, and payment companies, have shared talks and blog posts about similar lessons, including isolating replays, planning brokers by network and memory, and upgrading clients. Check their public writings for the latest details, as practices change over time.

It is spare memory that the operating system uses to keep copies of recently used file pieces, so that reading them again is fast.

No. Kafka writes to files and depends on the operating system to keep hot file pages in memory.

It depends on your write rate and the lag window you want to protect. Multiply the per-broker write rate by the window, add a margin, then add heap and system memory.

Because the heap does not store message bodies for long. A big heap takes memory from the cache and can cause long garbage collection pauses.

The system uses spare memory as cache. That is normal. Look at available memory and disk reads instead.

Not if they read the same recent data. They share cached pages. They do add network and processor work.

A read answered from memory. A read that must go to the disk is a cache miss.

It happens when reads of old, rarely needed data push out fresh, popular pages, so other readers start missing.

A way to send bytes from the page cache straight to the network card without passing through the Java program.

Yes, for encrypted client connections the broker has to process every byte, so it uses more processor time.

Not directly. Use disk read rates, fetch latency, and operating system tools that inspect the cache.

Usually not. Kafka relies on replication for safety, and forced flushing hurts speed.

It reads old data from disk and loads it into the cache, taking room from fresh pages and using disk time.

They limit how fast a client can read, so a batch job cannot flood the disks and the cache.

No. It moves old data away from local disks, but the local hot window still has to fit in memory.

The cache usually survives, since it belongs to the operating system. A machine reboot or replacement starts cold.

A low value makes the system drop cache pages before swapping the Java heap, which avoids long pauses.

Often the cache counts toward the container’s memory limit, so a tight limit leaves little room for it.

Both lengthen the time the cache covers. Adding brokers also adds network and disk capacity, while adding memory is simpler for a quick fix.

Every quarter, and after any big launch, traffic change, or new consumer group with heavy reading.

Kafka is quick because it lets the operating system keep recent data in spare memory and sends it out with almost no copying. Sizing the page cache means deciding how many minutes of fresh data must sit in memory so that readers, even slightly slow ones, never fall to the disk. You measure the write rate per broker, choose a lag window, add a margin, and then add the small memory needs of the heap and the system. After that, you protect the cache from replays, keep clients modern, tune the system gently, and watch lag in minutes.