Speed problems rarely come from where people first look.

When Kafka feels slow, most people say “we need more brokers” or “let us tune something.” Sometimes that is right. Often it is wrong, because the real limit sits somewhere else: a producer that sends tiny messages one by one, a network card that is already full, a disk that cannot write fast enough, or a single thread inside a broker that is always busy. Adding brokers does nothing for a problem that lives inside the producer.

Just enough vocabulary to follow everything else.

Apache Kafka is a system for moving streams of messages between programs. One program writes messages, others read them, and Kafka stores them safely in between. Here are the words we need.

Two facts that shape every ceiling

Three measurements that people often mix up.

A system can be excellent at one and poor at another. Sending millions of tiny messages stresses the processor and the request handling. Sending fewer huge messages stresses the network and the disk. Very fast delivery of each message (low latency) often means waiting less to form bundles, which lowers throughput. So we always say which measurement we mean, and we always state the message size.

The word “ceiling”

The throughput ceiling is the highest steady rate at which the system can accept data before something gets worse in a way you care about: queues start to grow, latency rises sharply, or requests time out. Notice the word “steady.” A machine may briefly go faster by using up stored energy, such as burst credits in the cloud, but that is not a ceiling you can live on.

The word “saturation”

A part is saturated when it is busy all the time and has no idle moments left. A saturated part becomes the bottleneck: the narrowest point that limits everything else. The whole game of ceiling analysis is finding the part that saturates first.

Every step on this path has a speed limit.

flowchart LR
  APP["Application creates a message"] --> SER["Producer turns it into bytes and picks a partition"]
  SER --> ACC["Message waits in a batch in the producer buffer"]
  ACC --> SND["Sender thread ships the batch over the network"]
  SND --> NET1["Network to the leader broker"]
  NET1 --> RQ["Broker network thread reads it and puts it in a queue"]
  RQ --> IO["Broker worker thread validates and appends it to the log"]
  IO --> PC[("Operating system page cache then disk")]
  PC --> REP["Follower brokers fetch and copy it"]
  REP --> ACK["Leader acknowledges to the producer"]
  PC --> CON["Consumers fetch it from the leader"]
    

Six resources can each become the ceiling.

The rule of the moving bottleneck

When you fix one limit, the next one becomes the new ceiling. Upgrading disks may reveal that the network card was the next limit. This is normal. A ceiling analysis is not one measurement but a small series: measure, find the weakest link, lift it, measure again. You stop when the ceiling is high enough for your needs, with comfortable room to spare.

The single biggest lever on the producer is how messages are grouped.

Sending one small message in one request is very wasteful. Each request has fixed costs: headers, waiting for the response, and work for the broker to parse it. If your message is 200 bytes and the overhead is comparable, most of the effort goes to overhead. Batching spreads that fixed cost over many messages.

How to tell if batching is the problem

Producers expose measurements for average batch size and for the number of records per request. If the average batch size is far below the configured maximum, and the linger time is zero, batches are leaving nearly empty. Raising the linger time by a few milliseconds often gives a large throughput gain with almost no visible latency cost. Recent Kafka versions have raised the default linger time above zero for exactly this reason, but older clusters and applications often still use zero.

Smaller data means less network and less disk, at the price of some processor work.

Producers can compress each batch before sending. The broker usually stores the batch as it arrived, still compressed, and consumers decompress on their side. This means one act of compression saves network on the way in, disk on the broker, replication traffic, and network on the way out. That is why compression is one of the strongest ways to raise a ceiling caused by network or disk.

Why bigger batches compress better

Compression finds repeated patterns. A batch of one message has few patterns to find. A batch of five hundred similar messages, such as JSON with the same field names, has many, and shrinks dramatically. So compression and batching help each other.

How safe you want each message to be changes how fast the system can go.

After sending a batch, the producer can wait for different levels of confirmation. This setting is usually called acknowledgments, or “acks.”

Does safety lower the ceiling?

Surprisingly, the ceiling for bytes per second is often similar for all three levels, because the followers copy the data anyway. What changes is the waiting time per request. Because the producer waits longer for each batch, it needs more requests in flight or bigger batches to keep the pipe full. If your settings allow only one or two requests in flight and the acknowledgment takes 10 milliseconds, the producer simply cannot send faster than one batch every 10 milliseconds per connection, no matter how strong the machines are.

Idempotence and order

Modern producers by default number every batch, so that a retry never creates a duplicate. This is called idempotence. It allows several requests in flight while still keeping order for a partition, so it usually does not hurt throughput and is a good default to keep on.

Sometimes the producer’s own machine is the bottleneck without anyone noticing.

A Kafka producer object has one background sender thread. That thread builds requests and sends them to every broker. In most programs this is enough. In very fast programs, that single thread, or the serialization and compression done before it, can become saturated. The tell-tale sign is a producer machine with one processor core at 100 percent while the others are idle, and brokers that are barely working.

Connections per broker

The producer keeps one connection to each broker it needs. Requests to one broker share that connection, so latency on that single connection can limit the rate to that broker. Spreading data over partitions on many brokers gives the producer many independent connections working at the same time.

Every byte crosses a network card several times.

A network card has a fixed speed, such as 10 gigabits per second. A gigabit is a billion bits, and there are eight bits in a byte, so 10 gigabits per second is about 1.25 gigabytes per second at best. In practice, protocol overhead and other factors leave roughly 85 to 90 percent usable. A network card also works in two directions: inbound and outbound, and modern cards handle both at the same time, each with its own limit.

Why the network fills up faster than you expect

With three copies of every partition, each producer byte enters one broker (the leader) and then leaves that broker twice, once for each follower. Consumers also pull the data out, once per consumer group. So the outbound side of a broker’s network card carries far more than the inbound side. We will do the exact arithmetic in a later chapter, but the key idea is already clear: outbound traffic is usually the first network limit to be reached.

Kafka is fast on disks because it treats them politely.

A partition is stored as a log: new messages are always added to the end of a file. Writing to the end of a file is a sequential write, meaning the disk head or the flash memory writes along a straight line and does not jump around. Sequential writes are much faster than random ones, especially on spinning disks, and are still helpful on modern solid-state drives.

Numbers to remember

Why reads are often much cheaper than you would guess.

Kafka does not keep its own big cache of messages. It leans on the operating system’s page cache, which is free memory that the operating system uses to hold recently used file contents. When a consumer reads data that was just written, the data is usually still in the page cache, so no disk read is needed at all.

Zero-copy: skipping the middle steps

Normally, sending a file over a network means copying bytes several times: from disk to the operating system’s memory, then to the application’s memory, then back to the operating system for the network card. Kafka can ask the operating system to send data straight from the page cache to the network card without passing through the application. This is called zero-copy, and it saves processor time and memory bandwidth.

The slow reader problem

A consumer far behind, for instance one replaying last week’s data, forces disk reads. Those reads also fill the page cache with old data, pushing out the fresh data that other consumers and followers need. This can make everything slower at once. Ceiling tests must include the situation where some readers are behind.

A broker is a small factory with three stations.

Inside a broker, requests pass through three stages. Understanding them explains a ceiling that is easy to miss: the broker can run out of thread time even when the disk and network are fine.

flowchart LR
  CL["Clients and followers"] --> AC["Acceptor accepts new connections"]
  AC --> NT["Network threads - read requests and write responses"]
  NT --> RQ[("Request queue - bounded")]
  RQ --> IT["I/O threads - process requests: append to log, read from log"]
  IT --> LOG[("Log files and page cache")]
  IT --> PG["Purgatory - waits for followers or for more data"]
  PG --> RS[("Response queue per network thread")]
  IT --> RS
  RS --> NT
  NT --> CL
    

Every request’s total time can be split into five parts, and the biggest part is the culprit.

Brokers measure how long requests spend in each stage. For a produce request, the total time is the sum of the following parts:

Every byte written is written again on other brokers.

Followers stay in sync by sending fetch requests to the leader, exactly like consumers. The leader replies with new data, and the follower appends it to its own log. The leader keeps a list of followers that are close enough to be called in-sync replicas. With the strongest acknowledgment setting, a write is confirmed only when all in-sync replicas have it.

Follower fetching and its own limits

Followers use a small number of fetcher threads per source broker. Each fetch request has a maximum size. If a follower must catch up on a very busy partition with only one fetcher thread, that one thread can become the limit, and the follower can fall out of the in-sync list. When that happens, writes with full acknowledgment may be refused if too few replicas remain in sync. So replication has its own ceiling, separate from the client-facing ceiling.

Simple formulas that predict broker network and disk load.

Let P be the total rate at which producers send data to the whole cluster, in megabytes per second. Let R be the replication factor (number of copies), and C the number of consumer groups that each read everything. Let N be the number of brokers, and assume load is spread evenly. Compression is already counted in P, since we measure bytes on the wire.

Now divide by the number of brokers to get the load on each. If N equals 6:

Finding the ceiling from the arithmetic

Now reverse the question. Suppose each broker can write 400 megabytes per second to disk, and can send about 1,000 megabytes per second out of its network card in practice. Then:

A cluster that is exactly at its ceiling cannot survive a single broker failure.

When a broker stops, its leaders move to other brokers, and its share of the load is spread over the survivors. In our six-broker example, five brokers must now carry what six carried. The per-broker load rises by one-fifth (six divided by five is 1.2 times), and during recovery, the restarted broker must also catch up on missed data, adding extra replication traffic.

How to plan

Where the leaders live decides who does the work.

The arithmetic above assumed that load is spread evenly. In real clusters it rarely is. Leaders handle all client traffic for their partitions, so a broker with more leaders, or with the busiest partitions, carries more load and reaches its ceiling first. The cluster’s ceiling is really the ceiling of its busiest broker.

The single-partition ceiling

One partition’s leader appends to one log, handled by one broker. The maximum speed of a single partition is therefore limited by that broker’s disk and by the way requests for that partition are handled. It is common for a single partition to peak at some tens of megabytes per second. If you need more for one topic, you need more partitions, so that many leaders share the work.

A few busy keys can spoil an otherwise balanced cluster.

Messages with the same key always go to the same partition. If one key is far busier than others, for example a very large customer or a default key like “unknown,” its partition receives a huge share of the traffic. That partition’s leader broker gets hot while the others are relaxed. Adding brokers does not help, because the hot key still lands on one partition on one broker.

Reading is part of the ceiling too, and it grows with the number of readers.

Every consumer group reads the full stream, so each additional group multiplies outbound traffic. Ten groups reading the same topic mean ten copies leave the brokers. Data that is still in the page cache costs little processor and no disk. Data that must be read from the disk costs disk reads on top of writes.

Some features quietly lower the ceiling.

Sometimes the ceiling is a rule, not a machine.

Kafka can limit how much data a client, user, or group of clients may send or receive per second. It can also limit how much processor time a client may use in the request handling threads. These quotas protect the cluster from one noisy application starving all the others.

Why speed feels fine, and then suddenly does not.

As you push more load into a system, three zones appear. In the first zone, everything is smooth: more load means more throughput, and latency stays flat. In the second zone, queues begin to form, and latency slowly rises. In the third zone, the system is saturated: throughput stops growing no matter how hard you push, and latency climbs steeply, often ending in timeouts. The bend between the second and third zones is called the knee.

flowchart LR
  Z1["Zone 1 - light load: throughput rises with load, latency flat, plenty of idle time"]
  Z2["Zone 2 - busy: throughput still rises, queues start, latency creeps up"]
  KN{"The knee - one resource is nearly always busy"}
  Z3["Zone 3 - saturated: throughput flat, queues grow, latency spikes, timeouts and retries begin"]
  Z1 --> Z2 --> KN --> Z3
  Z3 -->|"retries add even more load"| Z3
    

Why saturation is dangerous

Once a resource is saturated, requests wait in queues. Waiting requests can time out. The clients then retry, which creates even more requests. The extra load makes the queue longer, so more time-outs happen. This loop is called a retry storm, and it can keep a system down even after the original spike has ended.

The batching trade

Batching moves you along this curve in an interesting way. Longer waits for batches raise throughput, since the overhead is shared, but add a fixed delay to every message. The best setting is the smallest wait that gets the throughput you need, and it depends on how quickly your application needs each message delivered.

A step-by-step recipe that anyone on your team can repeat.

Bad tests give confident, wrong answers.

A simple path from symptom to the limiting resource.

flowchart TB
  S["Throughput stops rising and latency climbs"] --> Q1{"Are producer machines busy - processor near full or buffer full?"}
  Q1 -->|"Yes"| A1["Producer-side ceiling: batching, compression, serialization, or too few producer instances"]
  Q1 -->|"No"| Q2{"Are broker network threads or I/O threads almost never idle?"}
  Q2 -->|"Yes"| A2["Broker thread ceiling: more threads if processors are free, or fewer and larger requests, or add brokers"]
  Q2 -->|"No"| Q3{"Is a broker network card near its limit in either direction?"}
  Q3 -->|"Yes"| A3["Network ceiling: compress, spread leaders, add brokers, or upgrade network"]
  Q3 -->|"No"| Q4{"Is disk write speed or disk wait time at its limit?"}
  Q4 -->|"Yes"| A4["Disk ceiling: faster or more disks, more brokers, or less replication traffic"]
  Q4 -->|"No"| Q5{"Are followers falling behind - under-replicated partitions growing?"}
  Q5 -->|"Yes"| A5["Replication ceiling: more fetcher threads, check slow follower and cross-zone links"]
  Q5 -->|"No"| A6["Look for quotas, skew on one hot partition, or a slow client and consumer side"]
    

Reading it correctly

When the count of messages, not the bytes, sets the ceiling.

A metrics collector sends 1,000,000 tiny messages per second, each 100 bytes. The byte rate is only about 100 megabytes per second, small compared with disks and networks. Yet the producers struggle and the brokers show high processor use. What is going on?

The fix

After the changes, batches carry about 300 messages. The requests per second fall from about 330,000 to about 3,300, a hundred times fewer, and the I/O threads drop from 90 percent busy to about 10 percent. The ceiling is now far higher, and it is set by the disk and network again, as it should be.

A different workload with a different bottleneck.

An image-processing platform sends 2-megabyte messages at 400 per second, giving 800 megabytes per second, with 3 copies on 9 brokers. Per-broker disk writes are 800 times 3 divided by 9, about 267 megabytes per second. Each broker has a cloud volume purchased for 250 megabytes per second, so disk is already over its limit.

Options, in order of cost

Clouds add limits you cannot see on the machine itself.

Three families of remedies, and when each works best.

Where old data lives changes what the disk has to do.

By default, all retained data lives on each broker’s local disk. With tiered storage, older segments move to cheaper remote storage, and only recent data stays local. This shrinks the disk size that each broker needs and speeds up recovery and rebalancing, because there is less local data to copy.

Retention and cleanup work

Even without tiering, brokers perform background work: deleting old segments, compacting logs for compacted topics, and building indexes. Compaction in particular reads and rewrites data, which uses disk and processor time. When measuring a ceiling for compacted topics, include the cleaner’s work, since it competes with normal traffic.

A short calculation that prevents blocked applications.

The producer’s buffer memory holds batches that are waiting or in flight. If it is too small, the application is forced to wait whenever the buffer is full. A rough sizing rule is: the buffer should hold at least the number of partitions the producer writes to, times the batch size, with extra room for batches in flight to brokers.

The single-connection rate limit

Suppose one connection to a broker sends batches of 128 kilobytes, with 5 requests in flight, and each round trip takes 20 milliseconds. The maximum rate on that connection is 5 times 128 kilobytes per 0.02 seconds, which is 32,000 kilobytes per second, or about 32 megabytes per second. If the same producer talks to a broker in another region with a 100-millisecond round trip, the same setting gives only about 6.4 megabytes per second. Longer distance needs bigger batches or more in flight.

Put everything together in seven lines you can fill in for your own cluster.

Every step that raises the ceiling costs something else.

Protecting data costs capacity, and that cost should be planned rather than discovered.

What to watch so the ceiling never surprises you.

Exact metric names differ between Kafka versions and monitoring tools, so treat this as a map of ideas. Watch each resource in the chain, and watch how close it is to its limit.

Alert on headroom, not only on failure

Habits that keep the ceiling high and the surprises low.

Four popular mistakes and the better ways to work.

From symptom to likely cause to first check.

The same reasoning appears wherever Kafka runs at scale.

Everything you learned, told as one small story.

ClickWave collects website click events from customers’ sites into a Kafka cluster of six brokers, with three copies of everything. Traffic has grown, and the team wants to know how much more it can take.

A quick reference for the words used in this guide.

Quick answers to the questions beginners ask most often.

The whole story in a few short lines.