A short refresher on the parts of Kafka that use disks and networks.

Apache Kafka is a platform for storing and moving streams of events. An event is a small record of something that happened, such as an order placed or a temperature measured. Kafka stores these events in order, on disk, so that many applications can read them now or much later.

The three resources every broker spends

Every broker spends four kinds of resources: CPU for processing requests and compressing data, memory for caching, disk for storing data, and network for moving it. Kafka is generally light on CPU compared with the amount of data it handles. Memory is used mostly as a cache. So in most clusters, the two resources that run out first are disk and network, and that is why this tutorial focuses on them.

Kafka’s design makes it very fast, but only if the disk and network underneath it behave well.

Kafka does two simple things over and over. It receives bytes from the network and appends them to a file, and it reads bytes from a file and sends them over the network. Everything else, such as replication, retention, and compaction, is a variation on those two actions. That simplicity is the source of Kafka’s speed, and also the reason its speed is bounded by disk and network capacity.

flowchart LR
  P[Producer] -->|network in| B[Broker]
  B -->|write to disk| D[Log files on disk]
  B -->|network out| F[Follower brokers]
  D -->|read from disk or cache| B
  B -->|network out| C[Consumers]
        

What goes wrong when either is short

To size disks wisely, you first need to picture how Kafka arranges its files.

Partitions become directories, logs become segments

Each partition is stored as a directory on the broker’s disk. Inside that directory, the partition’s events are written into a series of files called log segments. Kafka writes only to the newest segment, called the active segment. When the active segment reaches a size limit or an age limit, Kafka closes it and starts a new one. Older, closed segments are never modified, only eventually deleted.

flowchart LR
  A[Partition directory] --> B[Closed segment 1]
  A --> C[Closed segment 2]
  A --> D[Closed segment 3]
  A --> E[Active segment receiving new writes]
  E -->|reaches size or age limit| F[Closed and a new active segment starts]
        

Why sequential writing is a big deal

Because Kafka only ever appends to the end of a file, its disk writes are sequential. Sequential writing is dramatically faster than jumping around a disk, especially on spinning hard drives, where the mechanical head must physically move for random access. This is one of the main reasons Kafka can achieve huge throughput even on modest hardware.

But sequential can turn random

There is a subtlety. Each partition has its own active segment. If a broker hosts a great many partitions, it is appending to a great many files at once. From the point of view of the disk, this can start to look like random access, as the disk hops between all those files. That is one reason why very high partition counts per broker can hurt disk performance, especially on hard drives.

Kafka is fast partly because it leans on the operating system’s memory cache instead of building its own.

When a program writes to a file, the operating system does not usually push the bytes to the physical disk immediately. It first keeps them in a region of memory called the page cache, and writes them to disk a little later, in the background. When a program reads a file, the operating system first checks whether the data is already in the page cache, and if it is, it serves it straight from memory without touching the disk.

Kafka deliberately relies on this. It does not maintain a large cache of its own inside the Java process. Instead, it lets the operating system use all spare memory as a cache for the log files. This has two large benefits: the cache survives a Kafka process restart, since it belongs to the operating system, and Kafka avoids the heavy garbage collection pauses that come with keeping huge amounts of data on the Java heap.

Durability comes from replication, not from flushing every write

By default, Kafka does not force each message to be flushed to physical disk before acknowledging it. It leaves flushing to the operating system. Safety comes from keeping several replicas on different brokers, and from producers requiring acknowledgement from enough of them. A single broker crash may lose a few unflushed messages from its own copy, but the other replicas still have them.

sequenceDiagram
  participant P as Producer
  participant B as Broker
  participant C as Page cache in memory
  participant D as Physical disk
  P->>B: Send batch of records
  B->>C: Append to the log file in cache
  B-->>P: Acknowledge after replicas confirm
  C->>D: Operating system flushes later in the background
        

Tail reads and lagging reads

Most consumers read data soon after it is written, close to the tail of the log. Those reads are served from the page cache and never touch the disk, making them extremely cheap. But a consumer that has fallen far behind, or one that starts from the beginning of a topic, asks for old data that is no longer in memory. Those reads must fetch from disk, and worse, they bring old data into the cache and push out the fresh data that other consumers want. A single lagging consumer can therefore slow down an otherwise healthy cluster.

Another reason Kafka is fast is a trick that skips most of the copying, and a common security feature that switches it off.

What zero-copy means

Normally, sending a file over the network involves several copies of the data: from disk to the cache, from the cache into the application’s memory, from the application into a network buffer, and from there to the network card. Each copy costs CPU time and memory bandwidth. Kafka uses an operating system feature that lets the kernel send data straight from the page cache to the network card, without passing through the Kafka process at all. This is called zero-copy, and it makes serving consumers remarkably cheap.

flowchart LR
  subgraph PLAIN[Plaintext connection]
    A1[Page cache] -->|kernel sends directly| B1[Network card]
  end
  subgraph TLS[Encrypted TLS connection]
    A2[Page cache] --> C2[Kafka process encrypts data]
    C2 --> B2[Network card]
  end
        

Why encryption changes the picture

When a connection is encrypted with TLS, the broker must encrypt the data before sending it, and that work happens inside the Kafka process. The data can no longer flow directly from the cache to the network card, so the zero-copy shortcut is lost, and the broker spends much more CPU per byte served. On a busy cluster, enabling TLS can noticeably reduce the maximum throughput per broker and increase CPU usage.

There is no single best disk. The right choice depends on your workload, budget, and failure plan.

Hard drives versus solid state drives

Because Kafka writes sequentially, spinning drives can perform surprisingly well for simple, high-volume workloads with a manageable number of partitions. But as partition counts and replays grow, solid state drives give far more comfortable headroom, and many teams choose them for that reason. The best answer is the one you confirm with a realistic load test on your own hardware.

JBOD versus RAID

A broker can be given several directories for its data, each on a different physical disk. This arrangement, often called JBOD, meaning just a bunch of disks, lets Kafka spread partitions across the disks. Alternatively, disks can be combined into a RAID set, so the operating system presents one big volume.

Local disks versus network storage

Kafka expects fast, low-latency, local-style storage. Network file systems such as those shared over a file-sharing protocol are a well-known trap: they add latency, can behave unpredictably under load, and are generally not recommended. In the cloud, block volumes attached over the network are widely used and work well, but they share bandwidth with other traffic on some instance types, and their performance depends on the volume type and configured throughput. Local instance storage is very fast but disappears if the machine is stopped, which is acceptable only because replication can rebuild the data.

A well-chosen disk can still be crippled by careless operating system settings.

Memory: heap versus page cache

Give the Kafka process a moderate Java heap, generally a few gigabytes, since it does not need to hold the data itself. Leave the rest of the machine’s memory to the operating system for the page cache. As a rule of thumb, the more of your active data, meaning the last several minutes to hours of writes, that fits in the page cache, the fewer disk reads your consumers cause.

Separate disks for cluster metadata

The component that stores cluster metadata, whether ZooKeeper in older setups or the controller’s metadata log in newer ones, is sensitive to disk latency. Placing it on its own low-latency disk, separate from busy data directories, prevents heavy data traffic from delaying critical cluster decisions.

A simple formula gives a solid first estimate. The art lies in the headroom.

The basic formula

The total disk space needed across the cluster is the data written per second, multiplied by how long you keep it, multiplied by the number of copies, and then divided by the fraction of each disk you are willing to fill. Every part of this formula matters.

A worked example

flowchart TD
  A[Write rate per second] --> M[Multiply]
  B[Retention in seconds] --> M
  C[Replication factor] --> M
  M --> D[Total data stored]
  D --> E[Divide by the usable fraction of each disk]
  E --> F[Raw disk needed across the cluster]
  F --> G[Split across brokers with failure headroom]
        

Things the formula does not show

The rules that decide how long data stays on disk shape both cost and behaviour.

Time-based and size-based retention

Kafka can delete old data based on age, based on the size of a partition, or on both. Whichever limit is reached first triggers deletion. Setting a size limit is a useful safety net, because it guarantees that even an unexpected traffic explosion cannot fill a disk forever, though it may cause data to be deleted earlier than the time limit suggests.

Segments and when deletion actually happens

Kafka deletes only closed segments, never the active one. So data older than the retention limit may remain on disk until the segment containing it is closed and then checked. If segments are very large, or a low-traffic partition takes a long time to fill one, old data can linger far past the nominal retention time. Segment size and roll time settings therefore affect how promptly disk space is reclaimed.

Compacted topics

For topics that keep only the latest value per key, a background cleaner rewrites segments, discarding older values. This uses extra disk input and output, and needs temporary space. The cleaner works on the older, closed portion of the log, and it competes with normal traffic for disk capacity, so heavily compacted workloads need extra disk headroom and enough cleaner threads to keep up.

Tiered storage

Newer Kafka versions support tiered storage, where recent data stays on the broker’s local fast disks, and older segments are moved to cheaper remote storage such as an object store. Consumers can still read the older data, transparently. This decouples how long you can keep data from how much local disk each broker has, and it shortens recovery, because a rebuilt broker only needs to copy the recent local portion.

Every event crosses the network more than once. Understanding each crossing is the key to sizing bandwidth.

flowchart LR
  P[Producers] -->|1 unit in| L[Leader broker]
  L -->|1 unit out| F1[Follower 1]
  L -->|1 unit out| F2[Follower 2]
  L -->|1 unit out| G1[Consumer group A]
  L -->|1 unit out| G2[Consumer group B]
        

Simple arithmetic tells you how much network you need, and how much headroom to add.

The traffic formulas

A worked example

Headroom for failures

The steady state numbers are only the starting point. When one broker fails, its partitions’ leaders move to other brokers, so the survivors take on extra client traffic. When it returns, or when a replacement joins, a large volume of catch-up replication begins. A common guideline is to keep steady-state network utilisation at around half of capacity or below, so that failures and recoveries do not push the network into saturation.

The cheapest way to add capacity is often to send fewer bytes.

Kafka producers can compress batches of records before sending them. The compressed batch travels over the network, and the broker stores it on disk as it is, without decompressing. Consumers decompress it when they read. So compression reduces network use on the producer link, on the replication links, on the consumer links, and disk space, all at the same time, at the cost of some CPU on producers and consumers.

Batching makes compression work

Compression works on batches, and larger batches compress better because there are more repeated patterns to exploit. Producers gather records for a short time, controlled by a linger setting, and up to a size limit, before sending. A small wait and a larger batch size usually improve throughput and compression at the price of a tiny additional delay per record.

Inside the broker, a few settings control how efficiently network requests are handled.

The request pipeline

A broker handles requests through a small pipeline. Network threads accept connections and read and write bytes on the sockets. Requests are placed in a queue, and a separate pool of input and output threads processes them, doing the actual log work. Responses are queued back to the network threads to be sent. If either pool of threads is too small for the load, requests wait in queues and latency climbs, even though CPU and disk may look fine.

flowchart LR
  A[Client connections] --> B[Network threads]
  B --> C[Request queue]
  C --> D[IO threads process requests and touch the log]
  D --> E[Response queue]
  E --> B
  B --> A
        

The bandwidth-delay idea

On a long-distance link, the amount of data that needs to be in flight to keep the pipe full equals the bandwidth multiplied by the round-trip time. If the buffers are smaller than that amount, the sender pauses and waits for acknowledgements, and throughput falls far below the link’s capacity. This is why replication or mirroring across regions often needs larger buffers than traffic inside one data centre.

How a broker presents itself on the network causes many mysterious connection problems.

Listeners

A broker can accept connections on several listeners, each with its own address, port, and security protocol. A common design has one listener for client applications and a separate one for traffic between brokers. This allows the inter-broker traffic to travel on a dedicated network interface or a private network, so that a burst of client traffic cannot starve replication, and so that the two can have different security settings.

flowchart LR
  A[Client applications] -->|client listener| B[Broker]
  B -->|inter broker listener on a private network| C[Other brokers]
        

Advertised addresses: the classic trap

When a client connects to any broker, it receives metadata that lists the addresses of all the brokers, and then connects directly to the specific broker leading each partition. The addresses in that metadata are the advertised addresses configured on each broker, not necessarily the address the client used first. If a broker advertises an address the client cannot reach, for example an internal name that only works inside the data centre, the client can connect initially and then fail with confusing timeouts when it tries to reach the actual partition leaders.

In the cloud, the network is not just about speed. It is also about money.

Spreading brokers across zones

To survive the loss of a data centre, brokers are usually spread across several availability zones, with replicas of each partition in different zones. This is excellent for resilience, but it means that replication traffic, and often client traffic, crosses zone boundaries. Cloud providers commonly charge for data transferred between zones, and with Kafka’s multiplication of traffic, these charges can become one of the largest lines on the bill.

flowchart TD
  P[Producer in zone A] -->|writes to leader| L[Leader in zone B]
  L -->|replicates across zones| F1[Follower in zone A]
  L -->|replicates across zones| F2[Follower in zone C]
  C[Consumer in zone A] -->|reads from nearby follower| F1
        

Fetching from the closest replica

Recent Kafka versions allow consumers to read from a follower replica in their own zone instead of always reading from the leader. The broker and the consumer both have to be told their location, and a replica selector picks the nearest suitable replica. This can remove most of the cross-zone traffic caused by consumers, and lowers read latency, at the cost of slightly more delay in seeing the very newest data, since followers are a fraction behind the leader.

Other cloud network points

Moving and rebuilding data are the moments when disks and networks are pushed hardest.

When data must be copied in bulk

How long does a rebuild take?

The time to rebuild a broker is the amount of data it holds divided by the speed at which data can be copied. Suppose a broker holds five terabytes, and replication can safely proceed at 100 megabytes per second. Five terabytes is about five million megabytes, and dividing by 100 gives fifty thousand seconds, or nearly fourteen hours. During those hours, the cluster runs with reduced redundancy, so a second failure is more dangerous.

Throttling: protecting live traffic

If a bulk copy runs at full speed, it will compete with normal client traffic for network and disk, and clients will suffer. Kafka provides replication throttles that limit the rate at which data is moved during reassignments and recovery. The right throttle is a compromise: too low, and recovery takes forever, leaving the cluster vulnerable. Too high, and users notice slowness.

Knowing how things typically fail makes them much easier to diagnose.

The cascade to avoid

One slow disk can start a chain reaction. Its broker’s followers fall behind and leave the in-sync sets. Producers requiring several in-sync replicas may then be rejected. The remaining brokers take over leadership, receiving extra traffic and possibly saturating. If recovery is then started without throttling, the network and disks are pushed even harder. Understanding this chain helps you interrupt it early, for example by shifting leaders away from a sick broker before it drags others down.

flowchart TD
  A[One disk becomes slow] --> B[Followers on that broker lag]
  B --> C[Replicas leave the in sync sets]
  C --> D[Fewer replicas confirm writes]
  D --> E[Producers slow down or see errors]
  C --> F[Leaders shift to other brokers]
  F --> G[Other brokers take extra load]
  G --> H[Wider slowdown if not addressed]
        

You cannot manage what you cannot see. These signals show the health of disks and networks.

Alerts that give you time to act

When many teams share one cluster, one noisy client can hurt everyone. Quotas keep the peace.

Kafka lets administrators set quotas on clients, identified by user, client identifier, or both. A producer byte-rate quota limits how much data a client may send per second. A consumer byte-rate quota limits how much it may fetch. A request quota limits how much processing time a client may consume. When a client exceeds its quota, the broker slows down its responses rather than rejecting requests, so the client is gently throttled.

A few habits keep clusters healthy, and a few habits quietly build up trouble.

Helpful patterns

Anti-patterns to avoid

Different workloads stress disks and networks in different ways.

Let us follow a realistic incident from the first alert to a lasting fix.

A company runs a six-broker cluster serving many applications. On a Monday morning, producers start timing out and several consumer groups report growing lag. Nothing was deployed over the weekend.

The lesson is that the trigger was not a hardware fault, but an interaction between one heavy reader, the page cache, and an uneven spread of busy leaders. Looking at the pattern across brokers, and understanding how reads reach the disk, led directly to the cause.

Because it writes sequentially by appending to files, and serves most reads from the operating system’s page cache. Sequential access is fast even on spinning drives, and cached reads avoid the disk altogether.

Not by default. Writes go into the page cache and the operating system flushes them later. Durability comes from replication across brokers and from producers requiring acknowledgement from enough replicas.

A moderate Java heap, typically a few gigabytes, and leave the rest of the machine’s memory for the operating system’s page cache, which does most of the caching work.

Hard drives can work for simple, high-volume workloads with modest partition counts. Solid state drives give far better headroom for many partitions, lagging or replaying consumers, and fast recovery. Test with your own workload.

Not necessarily, because Kafka replicates data across brokers. Many clusters use several separate disks, while some use RAID 10 to tolerate a disk failure without rebuilding a whole broker. Avoid parity RAID because of its write penalty.

Multiply the write rate by the retention time in seconds and by the replication factor, then divide by the fraction of each disk you are willing to fill, usually around sixty to seventy percent. Add growth, peaks, and room for a broker failure.

Inbound to brokers is the producer rate times the replication factor. Outbound is the producer rate times the sum of replication factor minus one and the number of real-time consumer groups. Keep steady utilisation around half of capacity to allow for failures and recovery.

Encryption forces data through the broker process, so Kafka can no longer use zero-copy to send data directly from the cache to the network card. CPU use rises, so plan capacity with encryption enabled.

The affected data directory can go offline, partitions on it become unavailable on that broker, and the broker may fail. Recovery is difficult, so alert early and keep generous free space.

Kafka deletes only closed segments. Data in the active segment, or in a large segment that has not yet been closed and checked, can remain beyond the nominal retention time.

Enable consumers to fetch from a replica in their own zone, use compression, limit unnecessary consumer groups and replication, and keep heavy clients near the cluster. Producers still write to leaders, so some cross-zone traffic remains.

The initial connection works, but the advertised addresses in the broker metadata are not reachable from the client’s network. Check the advertised addresses for each listener.

The data on the broker divided by the replication speed. Five terabytes at 100 megabytes per second takes nearly fourteen hours. Smaller per-broker data sets, faster networks, and tiered storage shorten recovery.

It limits the bandwidth used to move data during reassignments and recovery, so bulk copying does not starve normal client traffic. Remember to remove it afterwards so normal replication is not held back.

Plan capacity for the failure case, with generous headroom on both disks and network, and monitor trends per broker instead of cluster averages, so problems are found and fixed before they become outages.

Kafka’s speed comes from a simple design: sequential appends to log segment files, reliance on the operating system’s page cache, and zero-copy transfers to consumers. That design makes disk and network the two resources that most often limit a cluster. Disks must be large enough to hold the write rate, retention, and replication with generous headroom, and fast enough to handle many partitions and lagging readers. Networks must carry the multiplied traffic of replication and many consumer groups, with room left for failures and recovery.

Good results come from choosing sensible hardware and operating system settings, keeping the Java heap moderate, compressing at the producer, separating client and replication traffic where useful, and paying attention to cloud costs and advertised addresses. Bulk data moves should be throttled and done in batches, quotas should protect shared clusters, and monitoring should watch per-broker disk, network, and latency trends. Above all, plan for the bad day: a cluster that still works with one broker down and recovering is a cluster you can trust.