Speed in Kafka is not one magic feature; it is a stack of small decisions that all point in the same direction.

Apache Kafka is a distributed event streaming platform. In simple words, it is a system where programs called producers write records (small messages describing something that happened) and programs called consumers read them, possibly much later, possibly many times. The servers in the middle are called brokers. A broker’s whole job is to accept records, store them safely, and hand them back on request.

Because a broker stores everything on disk, a newcomer often expects it to be slow. Disks are famously slower than memory. Yet in practice a well-run Kafka cluster serves reads at speeds that look more like memory than disk. The reason is that Kafka was designed around how the operating system already behaves, instead of fighting it.

There are four design choices that combine to make this work. First, Kafka only ever appends to the end of a file, so disk access is sequential and predictable. Second, Kafka does not build its own memory cache; it lets the operating system’s page cache do that job. Third, records are stored in exactly the same byte format on disk, in memory, and on the network, so nothing has to be converted on the way out. Fourth, when data is ready to leave, Kafka asks the kernel to send it straight from the cache to the network card using a special call, avoiding wasteful copies.

flowchart LR
    P["Producer"] --> B["Kafka broker process"]
    B --> PC["OS page cache in RAM"]
    PC -. "background flush" .-> D["Disk log segments"]
    PC --> N["Network card"]
    N --> C["Consumer"]
        

Read the figure from left to right. A producer sends records to the broker process. The broker appends them to a file, but the write actually lands in the page cache first, which is just a region of RAM managed by the kernel. The kernel copies dirty (not yet saved) pages to the disk later in the background. When a consumer asks for data, the broker points the kernel at the cached file region, and the kernel pushes it to the network card. The dotted line shows that disk writes happen off to the side, not in the path of a consumer read.

A few operating-system ideas make everything else in this tutorial easy to understand.

Before we talk about Kafka specifically, we need a shared vocabulary. Do not worry if these words are new; each one is explained with a simple picture in mind. Once you have them, the rest of the tutorial reads like a story.

Why copies are expensive

Moving data from one memory area to another sounds harmless, but at high volume it is one of the most expensive things a server does. When the CPU copies bytes, it reads them from one memory location and writes them to another. That uses CPU time, uses memory bandwidth (the width of the road between the CPU and RAM), and also pushes useful data out of the small, very fast CPU caches. A server pushing several gigabytes per second can spend a shocking share of its power simply moving the same bytes around.

There are two kinds of copies to know about. A CPU copy is done by the processor itself and burns processor time. A DMA copy is done by a hardware controller on the side, leaving the CPU free to do other work. Zero-copy is not literally zero copies; it is about eliminating the CPU copies and the extra trips through user space, while the unavoidable DMA transfers still happen.

Disks, memory and the network card

A disk stores data permanently but takes microseconds (SSD) to milliseconds (spinning disk) to respond. Memory, or RAM, forgets everything when power is lost but responds in nanoseconds. The network card sends bytes to other machines. A read request from a consumer means data must, in one way or another, travel from wherever it lives (disk or RAM) to the network card. The whole trick is making that trip short.

To understand what gets cached, we first need to know what Kafka actually puts on disk.

A Kafka topic is a named stream of records, such as “orders” or “payments”. Each topic is split into partitions, and each partition is an ordered, append-only sequence of records. Every record in a partition gets a number called an offset, starting at zero and only ever going up. Consumers remember which offset they have read up to, which is how they resume after a restart.

On disk, a partition is a directory. Inside it, the log is not one giant file. It is cut into pieces called segments. Only the newest segment, called the active segment, receives new writes. Once a segment reaches a configured size or age, Kafka closes it and starts a new one. Old segments are deleted or compacted according to retention rules. This makes housekeeping cheap: Kafka can drop a whole old file at once instead of hunting for individual old records.

Record batches, not single records

Kafka does not store records one by one. Producers group records into batches, and the batch is the unit of storage, replication and fetching. A batch has a small header (with offsets, timestamps, and a checksum) followed by the records, and it may be compressed as a whole. The broker keeps this batch exactly as it arrived. It does not unpack it and repack it later. This one decision is what makes zero-copy possible, because the bytes on disk are already in the exact shape the consumer expects.

flowchart TD
    T["Topic: orders"] --> P0["Partition 0"]
    T --> P1["Partition 1"]
    P0 --> S1["Segment 1 - closed"]
    P0 --> S2["Segment 2 - closed"]
    P0 --> S3["Segment 3 - active"]
    S3 --> L["Log file"]
    S3 --> OI["Offset index"]
    S3 --> TI["Time index"]
        

The figure shows how one topic fans out. Each partition owns a stack of segments. The first two are closed and never change again, so they are perfect candidates for staying cached or being read straight from disk. Segment 3 is active, and new records are appended to its log file, while its two index files are updated so lookups keep working. Every one of these files is an ordinary file that the operating system can cache.

Every modern operating system quietly keeps recently used file data in spare RAM. Kafka leans on that habit heavily.

The page cache is a part of RAM that the kernel uses to hold copies of file contents. The kernel slices memory into fixed-size blocks called pages, commonly four kilobytes each. When any program reads a file, the kernel first checks whether the needed pages are already in the page cache. If they are, the read is served from RAM. This is called a cache hit. If they are not, the kernel loads them from disk into the cache, and then serves the read. This is a cache miss, and it costs a disk trip.

How the page cache behaves

Writes and dirty pages

The page cache also works for writes. When a program writes to a file, the kernel usually copies the bytes into cache pages and returns immediately, saying “done”. Those pages are now called dirty, meaning they hold newer data than the disk does. A group of kernel background threads later writes dirty pages to disk. This is called write-back. By default, Linux starts writing dirty data in the background after it has been dirty for about thirty seconds, or sooner if too much memory becomes dirty.

Why the page cache is “free”

Every Linux machine already has this cache; no program needs to install or configure it. The kernel automatically uses almost all otherwise-unused memory for it, and instantly gives it back when a program needs that memory. When you look at a server and see very little “free” memory, that is usually a healthy sign that the cache is doing its job, not a sign of a leak.

The most surprising decision in Kafka’s design is what it chose not to build.

Most databases build their own buffer pool, a private cache managed by the application. Kafka’s designers went the other way. They decided the broker should not maintain a large in-process cache of log data at all. Instead, the broker writes to files, reads from files, and lets the kernel decide what stays in memory. This is sometimes described as a page-cache-centric design.

The double-caching problem

Whenever a program reads a file in the usual way, the kernel caches the file pages. If the program then also keeps its own copy in a private cache, the same data sits in memory twice. On a machine with, say, thirty-two gigabytes of RAM, a large private cache could waste a big slice of the space that the kernel would otherwise use to cache more log data. By skipping the private cache, Kafka lets nearly all spare memory hold useful data once.

Garbage collection and the heap

Java automatically frees objects that are no longer used, in a process called garbage collection. The more objects live in the heap, the more work collection has to do, and sometimes the whole application must pause while it happens. A broker that pauses is a broker that cannot answer fetch requests or heartbeat to the cluster. Keeping the cache outside the heap means a Kafka broker can run with a fairly small heap (often a few gigabytes) and still cache tens of gigabytes of log data.

A warm restart

Servers are restarted for upgrades, patches and configuration changes. A private cache disappears with the process and takes a long time to refill. The page cache belongs to the kernel, so if only the Kafka process restarts, the recently used log pages are still in memory. The broker comes back and starts serving from cache almost immediately. That is a real operational benefit during rolling upgrades of a big cluster.

Reads get the spotlight, but what happens on the write side decides what is sitting in the cache when a consumer arrives.

When a producer sends a batch of records, the broker’s network threads receive the bytes and hand them to request-handling threads. These threads validate the batch (checking things like the checksum and the format), assign offsets, and append the batch to the end of the active segment file. The append is a normal file write. The kernel copies the bytes into page cache pages and the call returns quickly. From the broker’s point of view the record is now “in the log”.

sequenceDiagram
    participant Prod as Producer
    participant Brk as Leader broker
    participant PC as Page cache
    participant Fol as Follower broker
    participant Dsk as Disk
    Prod->>Brk: Send record batch
    Brk->>PC: Append to active segment
    PC-->>Brk: Write returns immediately
    Fol->>Brk: Fetch new data
    Brk->>PC: Read newly written pages
    PC-->>Fol: Data served from memory
    Brk-->>Prod: Acknowledge
    PC->>Dsk: Background write-back later
        

Look at the middle of the diagram. The follower fetches data that the leader has only just written. Because that data is still fresh in the page cache, the follower is served from memory and the disk is not involved at all. This is why replication in Kafka is so cheap on the leader: the same cached pages serve the followers and the consumers.

Durability without forcing the disk every time

A natural worry appears here. If a write only reaches RAM before the broker acknowledges it, what happens if the machine loses power? Kafka’s answer is replication. A record is normally copied to several brokers, each in a different failure zone. With the producer setting that waits for all in-sync replicas, and a cluster setting that requires a minimum number of them, a record is only acknowledged once several independent machines have it in their page caches. It is very unlikely that all of them lose power at the same instant.

What Kafka does flush

Kafka does have settings that force data to disk after a number of messages or a period of time, but by default they are not used, so the operating system decides when to flush. Kafka does flush when a segment is closed and rolled, and on a clean shutdown. It also records a recovery point, a marker saying “everything before here is known to be on disk”. After a crash, Kafka only needs to check and repair the log after that marker, which keeps restarts fast.

Segment rolling and the boundary flush

Every segment eventually reaches its size or age limit, and at that moment Kafka rolls to a new one. Rolling is an important moment for the page cache, because the closing segment is flushed so that its contents are known to be on disk before it becomes a read-only file. On a busy partition this happens regularly, and on a broker with thousands of partitions many rolls occur every minute. Each one triggers a small burst of write-back that competes briefly with reads for disk bandwidth.

Two consequences follow. First, very small segments mean many rolls, many open files, and many small flushes, which is wasteful. Second, very large segments mean fewer rolls but bigger deletions later, because retention removes whole segments at a time. Most clusters keep the default large segment size and change it only after measuring, since the kernel is already good at absorbing the flushing work in the background.

The page cache works best when the data it fetches from disk is read in a straight line. Kafka’s append-only log gives it exactly that.

Disk speed depends heavily on how you use the disk. A sequential access reads or writes neighbouring bytes one after the other. A random access jumps to unrelated locations. On a spinning disk, random access forces a mechanical arm to physically move, which takes milliseconds each time. Sequential access lets the arm stay put while the platter spins past, so throughput is thousands of times higher. Even on SSDs, sequential access is friendlier, because it lets the drive and the operating system read big chunks efficiently and prefetch what comes next.

Kafka’s log is naturally sequential for writes, since everything is appended at the end, and naturally sequential for most reads, since consumers walk forward through offsets. When a consumer reads and the data is not yet cached, the kernel notices the forward pattern and starts read-ahead: it loads the following blocks before the consumer has asked for them. By the time the consumer’s next request arrives, the data is often already waiting in the page cache.

Why the index is sparse

The offset index does not list every record. It lists one entry roughly every few kilobytes of log data. When a consumer wants offset 8,000,000, Kafka finds the nearest earlier index entry and then scans forward a short distance in the log. That small scan is sequential and cheap. Keeping the index small also means it fits comfortably in memory, and Kafka maps its index files into memory so lookups feel instant.

To appreciate zero-copy, first see how wasteful the ordinary way of sending a file over a network is.

Suppose a normal program wants to send a chunk of a file to a client over the network. In most programming languages, the natural way is to read the file into a buffer, then write that buffer to a network socket. It looks like two simple lines. Underneath, the machine does a surprising amount of work.

sequenceDiagram
    participant App as Application (user space)
    participant Ker as Kernel
    participant Dsk as Disk
    participant Nic as Network card
    App->>Ker: read (switch 1)
    Dsk->>Ker: DMA copy into page cache
    Ker->>App: CPU copy into app buffer (switch 2)
    App->>Ker: write (switch 3)
    Ker->>Ker: CPU copy into socket buffer
    Ker->>Nic: DMA copy to network card
    Ker-->>App: return (switch 4)
        

Count the waste. Two of the four copies are done by the CPU, and the application only acted as a middleman that never modified the data. It read the bytes in and immediately wrote the same bytes out. The bytes made a round trip into user space and back, purely because the ordinary programming interface demands it.

Zero-copy removes the middleman: the kernel itself moves bytes from the file cache to the network connection.

Operating systems offer a special system call for exactly the “send this file region over a socket” job. On Linux it is called sendfile. Instead of asking the application to read and then write, the application makes one request: “Kernel, please send bytes from this file to that connection.” The data never enters the application’s memory at all.

sequenceDiagram
    participant App as Application (user space)
    participant Ker as Kernel
    participant Dsk as Disk
    participant Nic as Network card
    App->>Ker: sendfile (switch 1)
    Dsk->>Ker: DMA copy into page cache (only on a miss)
    Ker->>Nic: DMA gather copy straight from page cache
    Ker-->>App: return (switch 2)
        

Compare this figure with the previous one. Two context switches instead of four. Two transfers, both done by DMA hardware, instead of four transfers of which two used the CPU. The Java process never sees the data, so nothing lands in the heap, and garbage collection has nothing to clean up.

A note on older hardware

On network cards that cannot gather scattered pages, the kernel has to make one CPU copy from the page cache into the socket buffer before the card sends it. That is three copies rather than two, and still better than four. Modern network cards commonly support the gather feature, so the two-copy version is what you usually get on a current server.

Kafka is written for the Java virtual machine, and Java exposes sendfile through a single channel method.

Java’s file channel offers a method named transferTo. On Linux, when the destination is a plain network socket, the Java runtime translates this method into a sendfile system call. Kafka’s log code represents a slice of a segment file as a “file records” object. When the broker builds a fetch response for a consumer, it does not read these bytes into the heap. Instead, it remembers the file, the start position and the length, and later asks the channel to transfer those bytes straight to the socket.

Why the on-disk format matters

Kafka keeps record batches on disk in the same wire format that clients use on the network. The producer builds a batch, possibly compresses it, and sends it. The broker stores those very bytes. A consumer fetching the batch receives those very bytes and does the decompression on its own side. Because the broker never has to unpack and repack the data, the bytes in the file are already a valid response, and sendfile can ship them untouched.

flowchart LR
    A["Producer builds and compresses batch"] --> B["Broker appends same bytes to log"]
    B --> C["Page cache holds same bytes"]
    C --> D["sendfile ships same bytes"]
    D --> E["Consumer decompresses and reads"]
        

The figure shows a single unbroken chain of identical bytes. This is a subtle design triumph. Compression happens once at the producer, the network carries the smaller data, storage holds the smaller data, the page cache holds the smaller data (so more history fits in memory), and only the final consumer pays the cost of decompressing. The broker in the middle is a very efficient courier that never opens the parcel.

Batch size, fetch size and how much each sendfile call moves

A fetch request carries limits that decide how many bytes the broker may return: a maximum for the whole response and a maximum for each partition. The broker slices up to those limits from the file and hands the whole slice to the kernel. If limits are tiny, the broker makes many small transfers and pays the fixed cost of a system call each time. If limits are generous, one transfer moves a large, contiguous region, which suits both read-ahead and the network card. This is why consumers that fetch a lot of data per request tend to be gentler on a broker than consumers that ask for a few kilobytes at a time.

There is also a waiting rule. A consumer can tell the broker to wait until a minimum amount of data is ready, or until a short time passes. That rule reduces the number of nearly empty responses, which saves CPU on both sides and lets more data accumulate in the cache so each transfer is worth the trip.

Why the follower path matters as much as the consumer path

Replication multiplies traffic. With three copies of every partition, the leader serves two followers in addition to all consumer groups. Because followers use the same fetch mechanism, they gain from the same cached pages and the same sendfile shortcut. A leader with a warm cache can feed its followers almost for free, while a leader whose cache has been polluted has to read from disk for the followers too, which slows replication, which in turn can shrink the set of in-sync replicas. In other words, a cache problem can start as a latency issue and grow into an availability issue.

The shortcut is fragile. Several perfectly reasonable configuration choices quietly switch it off.

Because zero-copy needs the bytes on disk to be the bytes on the wire, anything that forces the broker to touch the data can break it. When that happens, the broker falls back to the traditional path: read the data into memory, process it, then write it to the socket. The system still works, so no error appears, but CPU use rises, garbage collection increases, and throughput per broker drops. This is why these situations are dangerous: they are silent.

The TLS trade-off in detail

TLS (Transport Layer Security) protects data from being read or changed while it travels over the network. Many organisations require it, and rightly so. The cost is that the encrypting work must happen inside the Java process, which means the bytes must be read into memory, encrypted, and then written out. A broker serving TLS traffic therefore loses the biggest advantage of zero-copy for those connections and needs noticeably more CPU for the same data rate.

flowchart TD
    Q["Fetch request arrives"] --> T{"Is the listener using TLS?"}
    T -- "Yes" --> S["Read into memory, encrypt, then write to socket"]
    T -- "No" --> V{"Client needs an older record format?"}
    V -- "Yes" --> C["Convert batches in memory, then write to socket"]
    V -- "No" --> Z["sendfile: page cache straight to network card"]
        

Follow the arrows in the figure. The first decision asks about encryption; the second asks about format conversion. Only the bottom-right path is the fast one. Two of the three outcomes send data through the Java process. A platform team should be able to say, for each cluster, which of these three paths its traffic mostly takes.

Two consumers reading the same topic can have wildly different effects on a broker, depending on how far behind they are.

A consumer is caught up when it reads records almost as fast as they are produced. It sits at the tail of the log, and the data it wants was written seconds ago. That data is almost certainly still in the page cache, so its reads are served from RAM and shipped by sendfile. On a cluster where consumers are mostly caught up, the disks can be almost silent on reads, even while gigabytes flow out through the network.

A consumer is lagging when it is far behind, perhaps because it was down for maintenance, is slow, or is a new application replaying history. Its wanted data is old. It may have already been pushed out of the cache, so the kernel has to read it from disk.

flowchart LR
    subgraph Hot["Caught-up consumer"]
    H1["Requests recent offsets"] --> H2["Page cache hit"] --> H3["sendfile to network"]
    end
    subgraph Cold["Lagging consumer"]
    C1["Requests old offsets"] --> C2["Page cache miss"] --> C3["Disk read plus read-ahead"] --> C4["Pages enter cache"] --> C5["sendfile to network"]
    end
        

The figure shows that both consumers get zero-copy delivery. The difference is the first stage. The lagging path involves the disk, and disk reads are much slower and much more limited than memory hits.

The cache pollution problem

The cost of a lagging consumer goes beyond its own slowness. When the kernel loads old pages for it, those pages need room in memory. The kernel makes room by evicting other pages, and those may be the very tail-of-log pages that hundreds of caught-up consumers and follower replicas depend on. Suddenly the healthy consumers start seeing cache misses too. Disk reads rise, latency climbs, and the broker can appear to be “slow for everyone” because of one heavy reader.

When read-ahead helps and when it hurts

Read-ahead is a bet that you will keep reading forward. For a lagging consumer scanning a partition in order, the bet pays off: the next blocks arrive before they are requested, and the disk stays busy doing large, efficient reads. For a reader that jumps around, the bet is lost, because the extra blocks loaded are never used and they occupy cache space that something else could have used. Kafka mostly benefits from read-ahead because its consumers overwhelmingly read forward.

The lesson for operators is to be cautious with very large read-ahead values. They help sequential replays on spinning disks, but on a broker with many partitions read at once, each partition’s read-ahead multiplies, and the total amount of speculative data can crowd the cache. Moderate values that suit the storage device are usually the safest starting point.

The page cache is only as good as the memory you leave for it, so sizing a broker is really a question of how RAM is split.

On a Kafka broker, machine memory is divided into a few regions. The JVM heap holds Java objects for the broker’s internal work, such as request handling, metadata, and buffers for producer requests. Native memory holds things outside the heap, such as thread stacks and network buffers. Everything left over is free for the operating system, and the kernel fills nearly all of it with page cache.

flowchart TD
    R["Total machine RAM"] --> J["JVM heap: small, often a few gigabytes"]
    R --> N["Native and OS overhead: modest"]
    R --> P["Page cache: everything else"]
    P --> H["Hot tail of every partition"]
    P --> I["Mapped index files"]
    P --> O["Recently read older segments"]
        

The figure emphasises how lopsided the split should be. A common starting point for a busy broker is a heap of roughly six gigabytes, on a machine that has far more RAM in total. The exact number depends on partition count and workload, but the principle is stable: give the heap enough to run comfortably without frequent pauses, and let the rest be cache.

How much cache do you actually need?

A useful way to think about it: the cache needs to hold the data that active readers will want. For caught-up consumers, that is the most recent stretch of each partition’s log. Estimate it like this. Take the write rate per second for the whole broker, multiply by the number of seconds of history you want to keep cached (for example, thirty seconds to a few minutes for real-time readers), and that is the minimum useful cache. If some consumers regularly lag by an hour, the cache needed to protect them grows accordingly.

Other programs on the same machine

Anything else running on a broker host competes for memory. A monitoring agent, a log shipper, or an unrelated application that uses lots of RAM shrinks the cache. That is why Kafka brokers are best run on dedicated hosts, or on hosts where the neighbours have small, well-understood memory needs.

A worked example in plain words

Imagine a broker that receives data at a steady rate of one hundred megabytes each second. Real-time consumers are usually within a minute of the tail. Sixty seconds of writes is about six thousand megabytes, or roughly six gigabytes. That is the amount of recent data that should stay cached for most reads to be memory hits. Add some room for index files, for other partitions’ tails, and for a few consumers that lag by several minutes, and a working target of twenty to thirty gigabytes of page cache is sensible.

Now add the heap, native memory and the operating system. A machine with sixty-four gigabytes of RAM comfortably covers everything: a heap of a few gigabytes, some native and system memory, and a page cache of well over forty gigabytes. A machine with only sixteen gigabytes would leave perhaps ten for cache, which is thin for this workload, and the operator would see disk reads climb whenever any consumer fell behind. The example is deliberately simple, but it shows the method: start from the write rate, choose how much history you want in memory, and work backwards to a machine size.

A few operating-system knobs decide whether the kernel behaves like a friend or a nuisance for a Kafka broker.

Kafka works out of the box on a default Linux system, but a handful of settings are commonly adjusted on production brokers. The names below are kernel parameters that operators set in system configuration. They are described here in words, since the point is to understand what each one controls and why.

Choosing the file system and mount options

Kafka is commonly run on XFS or ext4, and XFS is often preferred for its good behaviour with large files and heavy parallel writes. It is also common to mount the data volumes so that the kernel does not update a “last accessed” time on every read, because that extra tiny write per read is wasted work. Keeping Kafka data on its own volumes, separate from the operating system, also stops noisy neighbours from interfering with flushes.

Why smoother flushing helps

If dirty data piles up for a long time and is flushed all at once, the disk suddenly receives a big wave of writes. During that wave, reads that miss the cache queue behind the writes, and produce requests can be delayed if the hard dirty limit is reached. Lowering the background threshold makes the kernel start flushing sooner and in smaller waves, so the disk load is steady rather than spiky. Raising the hard limit keeps normal bursts from ever hitting the wall.

Because the cache lives in the kernel, you must look outside Kafka’s own metrics to see whether it is healthy.

Kafka does not report a “page cache hit ratio” of its own. Operators infer cache health from indirect signs: how much the disks are being read, how long fetches take, how idle the broker’s threads are, and what the operating system says about memory. A good habit is to build one dashboard that shows both sides together.

Special tools for looking into the cache

Several small utilities can show which files are cached and how well the cache is working. Some report how much of a given file is currently in memory. Others trace kernel cache accesses and report hits and misses per second. Standard system tools also show per-disk throughput and CPU wait. Learning one tool from each group is enough for most investigations.

flowchart TD
    S["Slow fetches or high broker load"] --> D{"Are disk reads high?"}
    D -- "Yes" --> L["Find lagging readers and check cache size"]
    D -- "No" --> C{"Is CPU high?"}
    C -- "Yes" --> Z["Suspect TLS, conversion or recompression"]
    C -- "No" --> W["Check network limits and client behaviour"]
        

This figure gives a quick first cut. High disk reads point to the cache; high CPU with quiet disks points to data flowing through the heap; neither points to the network or to the clients. Real incidents can combine causes, but this three-way split resolves a surprising number of cases in minutes.

Three dashboard patterns and what they usually mean

These three patterns are not the only ones, but they teach the right habit. Never look at one graph alone. Pair each Kafka symptom with an operating-system graph, and ask which of the two questions from the start of this tutorial is being answered badly: was the data in memory, and how many times was it copied?

Speed features touch security in a few important ways, and a good architect keeps both in view.

The page cache holds plain, unencrypted copies of whatever is in the log files. That is normal for any file cache, but it means that protection has to come from the machine boundary: who can log into the host, who can read the data directory, and how disks are encrypted at rest. If the underlying volume uses disk-level encryption, the kernel decrypts as it loads pages, and the page cache still holds clear data in RAM.

Quotas as a security and fairness tool

Client quotas let an operator cap how many bytes per second a user or client application may read or write. In the light of this tutorial, quotas are more than a fairness feature; they are how you protect the shared page cache. A quota on fetch rate bounds how much cold data a single tenant can drag into memory in a given time.

Good architects turn the ideas of this tutorial into repeatable decisions, and learn to spot the decisions that quietly undo them.

Modern deployment environments add new twists to a design that was born on bare-metal servers.

Kafka in containers and Kubernetes

When a broker runs in a container, the container has a memory limit enforced by the kernel through a mechanism called a control group. Page cache pages used by files that the container reads and writes are usually counted against that container’s memory. This surprises many people. A dashboard might show container memory use climbing to the limit and look alarming, when much of it is reclaimable cache that the kernel will happily discard under pressure.

The right way to read container memory is to look at the “working set”, which subtracts cache the kernel can easily drop, and to remember that too tight a limit leaves no room for the cache. If you set the container’s memory limit only slightly above the heap size, the page cache will be tiny and reads will drift towards disk. A memory limit for a Kafka broker container should deliberately include generous space for cache, and the memory request should reflect that too.

Cloud block storage and network disks

In the cloud, log data usually sits on network-attached volumes instead of local disks. These volumes have per-volume limits on throughput and operations per second, and their latency is higher than local flash. The page cache is even more valuable here, because every miss now travels across a network to the storage service. On the write side, write-back to a network volume can lag, so watching dirty memory and flush time matters more.

Tiered storage

Newer Kafka versions support tiered storage, where older closed segments are moved from broker disks to a cheaper remote object store, while recent data stays local. This changes the picture for old reads. Recent data still lives in local files and benefits from the page cache and sendfile exactly as described. Older data that has been moved off the broker is fetched from the remote store, so those reads do not enjoy the same local, zero-copy path. In practice this makes the hot local tail smaller and more focused, but it also means historical reads have a different performance profile that should be tested separately.

Choosing between local disks and network volumes

Local disks attached to the machine give low and steady latency, and they make cache misses cheaper. The trade-off is that data on them must be rebuilt from replicas if the machine is lost, which shifts more work to replication after failures. Network volumes survive machine loss and are easy to resize, but every miss pays a network round trip and every volume has throughput ceilings. Teams that choose network volumes generally compensate with more memory for a bigger cache and with careful monitoring of volume limits, while teams with local disks compensate with more brokers and faster recovery plans.

The same two ideas appear again and again in the way large organisations run streaming platforms.

Kafka began at LinkedIn, where engineers needed a way to move huge streams of activity data between systems. The page-cache-centric and sendfile-based design was part of the original thinking and has remained a distinguishing feature as the technology spread to many industries. Large technology and financial companies now operate clusters that carry very large volumes of events, and many of them describe similar operational habits: memory-rich brokers, small heaps, careful control of lagging readers, and a strong preference for keeping most traffic on the fast path.

flowchart LR
    W["Producers write once"] --> PC["Page cache holds one copy"]
    PC --> R1["Consumer group A"]
    PC --> R2["Consumer group B"]
    PC --> R3["Consumer group C"]
    PC --> F["Follower replicas"]
        

This last figure captures the whole economic idea of Kafka’s read path. A record is written once. It sits in memory once. It may be sent out many times, to many consumer groups and replicas, each time by a handoff that costs the CPU almost nothing. That is why adding readers to a healthy Kafka cluster tends to hit the network limit long before the CPU or memory limit.

A short checklist of habits that keep the fast path fast.

Yes, when the needed data is not in the cache. Caught-up consumers are usually served from memory, but lagging consumers, restarted replicas, and reprocessing jobs cause real disk reads. The design makes the common case fast and the uncommon case acceptable.

Not if only the process crashes. The written data is already in the kernel’s page cache, which continues to exist and will be flushed by the kernel. Data can be lost only if the whole machine or kernel fails before write-back, and replication to other brokers is the protection against that.

It refers to the number of copies made by the CPU in user space on the way from the file to the network. The data still moves by DMA between the disk, memory and the network card, but the processor does not shuffle it through the application’s own buffers.

Not directly. The write path reads incoming bytes into buffers for validation and offset assignment. Zero-copy mainly benefits reads, which include consumers and follower replicas, and reads usually dominate traffic.

Encryption is done inside the Java process, so the data must be brought into user space, encrypted, and sent. That means sendfile cannot be used for those connections, and the broker has to do real cryptographic work for every byte.

Almost never. Kafka’s data is cached outside the heap, so extra heap takes memory away from the cache and can make garbage-collection pauses longer. Increase the heap only if monitoring shows real heap pressure.

The kernel fills unused memory with page cache on purpose. It is reclaimable immediately when a program needs it. Look at the “available” figure rather than the “free” figure to judge real pressure.

A consumer reading old data forces the kernel to load old pages, which can evict the recent pages other consumers rely on. Those consumers then suffer cache misses, and the disk becomes busier for everyone.

It helps. Compressed batches take less space in the cache and on the network, so more history fits in memory. The broker stores what the producer sent, and consumers do the decompression.

Yes. Kafka maps its index files into memory, so they are served by the page cache as well. They are small, and they are very valuable to keep resident because every fetch begins with a lookup in them.

No. Recent data is still local and benefits from the cache and sendfile. Tiered storage changes how older data is served, and it can shrink the amount of local data, but the hot tail remains the most important part of the memory plan.

You can infer it. With plaintext listeners and up-to-date clients, high network output with low broker CPU and quiet heap activity indicates the fast path. If CPU and garbage collection climb in proportion to traffic, some traffic is likely going through the heap, and you should check for TLS or format conversion.

Kafka’s read speed is not the result of a single trick but of a coherent design that cooperates with the operating system. Producers write batches that are appended sequentially. The kernel keeps recent data in the page cache, so most reads never touch the disk. The bytes in the file are the bytes on the wire, so the kernel can send them straight from cache to the network card with sendfile, skipping the application’s memory and the wasteful copies that come with it. Replication provides durability, so the broker does not need to force every write onto the disk. The price of this design is that it depends on plenty of spare memory, sensible operating-system settings, and vigilance about anything that breaks the fast path, such as TLS, format conversion, and heavy lagging readers.