Before any formulas, let us agree on what we are trying to decide and why a wrong answer is so expensive.

What is a broker?

A broker is one Kafka server. It receives events from software called producers, stores them on disk, and hands them to software called consumers. Several brokers working together form a cluster. When we talk about sizing, we mean answering two questions: how many brokers do we need, and how much processor, memory, disk and network bandwidth should each one have?

The four resources

A broker uses four main resources, and any one of them can become the bottleneck:

• Disk holds the events. It answers “how much can we keep, and how fast can we write and read it?”
• Memory speeds everything up by keeping recent events close to hand.
• CPU, the processor, does the thinking: handling requests, encrypting traffic, compressing data.
• Network carries bytes in from producers, out to consumers, and between brokers for copying.

What goes wrong when sizing is off

You do not need to be an expert, but a few ideas explain almost every sizing decision that follows.

Topics, partitions and offsets

A topic is a named stream of events, such as “orders” or “payments”. Each topic is split into partitions, which are separate ordered logs. Partitions let Kafka spread work across many machines. Every event in a partition gets a running number called an offset.

Leaders and followers

Every partition has copies, called replicas. The number of copies is the replication factor, and three is a very common choice. One copy is the leader, and it serves all reads and writes for that partition. The others are followers, which keep themselves up to date by continuously fetching new data from the leader. This copying is the reason the network and disk work is larger than it first appears: every byte a producer sends is written several times.

An append-only log on disk

Kafka stores each partition as a series of files called segments. New events are only added at the end, which is called sequential writing. Reading old data is usually sequential too. Disks are much faster at sequential work than at jumping around, and this is one of the main reasons Kafka is fast.

Leaning on the operating system

Kafka does not build a large private cache of data inside its own program. Instead it relies on the operating system’s page cache, which is the free memory that the operating system uses to keep recently used file data. When a consumer reads events that were written a moment ago, they usually come straight from memory rather than from the disk. This design has a big consequence for sizing: memory that is not given to Kafka’s program is not wasted. It becomes cache that makes Kafka faster.

Zero-copy sending

When a broker sends stored data to a consumer over a plain, unencrypted connection, the operating system can pass the bytes from the page cache to the network card without copying them through the Kafka program. This trick, called zero-copy, saves a large amount of processor work. It matters because it is lost when encryption is switched on, which we will revisit in the CPU chapter.

The four resources are not independent islands. They are stages in one pipeline, and the slowest stage sets the speed of the whole thing.

flowchart LR
  P["Producers"] --> N["Network card"]
  N --> T["Network threads"]
  T --> Q["Request queue"]
  Q --> I["I/O threads"]
  I --> PC["Page cache in memory"]
  PC --> D["Disk"]
  PC --> R["Follower brokers"]
  PC --> C["Consumers"]
    

Walking through the pipeline

Data arrives at the network card. A small group of network threads reads the bytes off the connections and turns them into requests. The requests wait in a request queue until one of the I/O threads is free. That thread appends the data to the log, which first lands in the page cache in memory, and the operating system writes it to disk shortly afterward. Meanwhile follower brokers and consumers fetch the same data, mostly from the page cache.

Each box in the picture can be a bottleneck. If the network card is full, data cannot arrive. If the network threads are busy, connections stall. If the I/O threads are busy, the queue grows. If the disk is slow, the page cache fills with data waiting to be written. If memory is short, reads fall through to the disk.

How the resources trade with each other

• More memory means less disk work. A bigger page cache turns disk reads into memory reads.
• Compression trades CPU for network and disk. Squeezing data costs processor time but saves bytes everywhere else.
• Encryption trades CPU for safety, and it also removes the zero-copy shortcut.
• More copies trade disk and network for safety. A higher replication factor multiplies both.
• More partitions trade memory and overhead for parallelism.

You cannot size anything without numbers. Before opening a spreadsheet, collect the facts below. Even rough answers are far better than none.

Peak versus average

Two numbers describe traffic, and they answer different questions. The peak rate decides whether your network, CPU and disk speed can cope at the busiest moment. The average rate, multiplied by time, decides how much disk space you consume. Mixing them up is the most common source of wrong estimates: sizing storage from the peak wastes money, and sizing speed from the average causes outages.

Measured bytes versus guessed bytes

If you already run a similar system, measure it. If you are building something new, sample real messages, compress them with your chosen method, and see the true size. Producers using JSON often shrink to a third of their original size or better, while data that is already compressed, such as images, barely shrinks at all.

When some numbers are still unknown

New projects rarely know every figure. That is acceptable, as long as you write down a low, expected and high estimate for each unknown, and size for the expected case while checking that the high case is survivable. Keep a list of who will confirm each estimate and by when. As real measurements arrive, replace the guesses one at a time. A sizing sheet with honest ranges is far more useful than one with falsely precise numbers, because everyone can see which assumptions carry the most risk and deserve early testing.

Four short formulas turn your requirements into cluster-wide traffic. Everything later in this guide builds on them.

Let us give the numbers names

• W is the producer data rate, in megabytes per second, as stored (after any compression).
• R is the replication factor, the number of copies.
• C is the number of consumer groups that each read the full stream.

The four cluster-wide numbers

Think in messages as well as megabytes

The same 60 MB/s can arrive as 60,000 messages of one kilobyte, or as 600 messages of one hundred kilobytes. The first case creates far more requests for the broker to process, so it uses more CPU. Always ask for both the bytes per second and the messages per second. Producers that batch, meaning they group many small messages into one request, reduce the request count dramatically.

Spread evenly, then check the spread

Dividing the cluster totals by the number of brokers gives a per-broker figure, but only if the work is spread evenly. In real clusters it rarely is. A topic whose messages all use the same key lands in one partition, and therefore on one leader broker. Add a safety margin for imbalance, and keep an eye on the busiest broker, not just the average one.

Disk capacity is often the first resource to run out, and it is the easiest to calculate. Let us do it slowly, one line at a time.

The capacity formula

Total disk space equals the average data rate, multiplied by the number of seconds you keep data, multiplied by the number of copies, then increased to leave free room. Free room matters because a disk that is nearly full is slow, risky and hard to repair.

In this example, the cluster needs around 78 terabytes on day one, and should be able to grow toward 156 terabytes. With nine brokers, that is roughly 8.7 terabytes each today.

Why leave so much free space?

• Kafka deletes old data by whole segment files, so usage rises and falls in steps, not smoothly.
• When a broker fails and its partitions are rebuilt elsewhere, the receiving brokers need spare room to hold them.
• Traffic can surge beyond what you predicted, and retention by time then produces more data than planned.
• A very full disk performs worse and leaves no time to react to problems.

Other things that take space

Besides the events themselves, each segment has small index files that help Kafka find data quickly. Topics that use log compaction, which keeps only the latest value for each key, need extra room while the cleaner rewrites files. Temporary copies during partition moves also need space. Add a few percent for these, or simply rely on the generous free-space margin.

Compression changes the answer a lot

If producers compress their batches, the brokers store the batches in compressed form, and your measured data rate W should already reflect that. For text-like data such as JSON, a ratio of three to one is common, and structured logs may do better. For data that is already compressed, expect almost nothing. Always test with real samples, since this single ratio can shift your disk bill by a factor of three or more.

Tiered storage can shrink local disks

Newer Kafka versions support tiered storage, which keeps only recent data on the brokers’ fast local disks and moves older data to cheap object storage while still letting consumers read it. If, in our example, only one day stays local, local capacity falls to about 2.6 TB × 3 ÷ 0.7, which is around 11 TB in total, instead of 78 TB. The trade-off is that reads of old data now travel across the network from object storage, so measure that path if replays are common.

Space is only half the story. The disks must also be quick enough to absorb writes and, when the cache misses, serve reads.

How much write speed per broker?

Take the cluster disk write traffic from Chapter 5 and divide by the number of brokers. In our example, 180 MB/s across nine brokers is only 20 MB/s each, which even a modest disk handles. That is typical: for many clusters, capacity is the real constraint and raw speed is easy. Clusters with very high ingest or many catch-up readers are the exceptions.

Reads: mostly cache, sometimes disk

Consumers that keep up read data that is still in the page cache, so the disk sees few reads. But a consumer that starts from the beginning, or one that was offline for hours, reads old data that is no longer in memory. Those reads hit the disk, and if many such consumers run at once, they compete with the writes. Sequential reads are handled well by most disks, but many readers at different positions create a jumpy pattern that punishes slow drives.

Choosing the kind of storage

These numbers are only rough guides, since products change every year. The habit that matters is to check the manufacturer’s or cloud provider’s stated limits and then confirm with your own test.

One big array, or many separate disks?

A broker can be given several storage directories, one per disk, in a layout often called JBOD, which means “just a bunch of disks”. Kafka spreads partitions across those directories. The alternative is to combine disks into a RAID array. RAID 10 (mirrored and striped) gives resilience but halves usable capacity, and Kafka already keeps replicas on other brokers, so many teams find the extra cost unnecessary. RAID 5 and RAID 6 slow down writes and are usually a poor fit. A common practical choice is JBOD or RAID 10, depending on how you want to handle a single disk failure and how much your operations tooling supports.

File system and settings

• XFS is widely recommended for Kafka data directories, and ext4 is also used successfully.
• Mount data disks with the option that avoids updating “last accessed” times on every read (noatime), to cut needless writes.
• Never place Kafka data on a shared network file system such as NFS; it is unreliable for this workload.
• Keep the operating system and Kafka’s own log files on a different device from the data, so a full data disk does not make the machine unusable.

Kafka runs inside the Java virtual machine, and the first memory decision is how big to make its private working area, called the heap.

What the heap holds

The heap stores Kafka’s bookkeeping: information about partitions and replicas, buffers for requests in flight, connection state, and the internal structures used to track who has read what. It does not hold your message data for long. The data itself flows through the operating system’s cache. That is why brokers rarely need a huge heap.

A sensible starting point

Many production guides suggest starting with a heap of around six gigabytes for a busy broker, and moving to eight or even twelve only if measurements show pressure. Smaller brokers with modest partition counts can run happily with less. The heap needs grow with the number of partitions, the number of client connections and the size of the requests being handled.

Why not just give Kafka everything?

• It steals from the page cache. Every gigabyte given to the heap is a gigabyte the operating system cannot use to cache your data.
• It lengthens pauses. The Java garbage collector, the tidy-up process that frees unused objects, has more to scan in a big heap. Long pauses make the broker seem frozen, which can cause followers to fall out of sync or clients to time out.

Use a modern collector such as G1, which is designed to keep pauses short, and watch the pause times in your monitoring. If pauses regularly exceed a fraction of a second, look at your partition and connection counts before simply enlarging the heap.

Memory outside the heap

The Java process also uses memory outside the heap for network buffers, file mappings and internal machinery. And the operating system needs some for itself. As a rule of thumb, reserve a few gigabytes for these, in addition to the heap.

If Kafka has a superpower, it is this: recently written data is almost always still sitting in memory when consumers ask for it.

sequenceDiagram
  participant C as Consumer
  participant B as Broker
  participant M as Page Cache
  participant D as Disk
  C->>B: Fetch from an offset
  B->>M: Look for the data in memory
  alt Data is cached
    M-->>B: Return data quickly
  else Data is not cached
    B->>D: Read from disk
    D-->>B: Return data slowly
    B->>M: Keep a copy for next time
  end
  B-->>C: Send the data
    

How to size the cache

The cache should be large enough to hold the data that consumers are likely to read soon. A common rule of thumb is to have room for at least about thirty seconds of a broker’s incoming data, and more if you have readers that regularly lag by minutes. Remember that a broker caches the data it writes for its follower replicas too, so use the per-broker disk write rate, not just the producer rate.

Turning it into a machine size

Add the pieces: heap, a few gigabytes for the operating system and off-heap needs, and the page cache you want. For the 10-minute case with a zone failure, that is 6 GB of heap, about 4 GB of other overhead and 18 GB of cache, giving 28 GB. Round up to a 32 GB machine, and choose 64 GB if long replays matter. Because Kafka’s speed depends so much on this cache, memory is often the cheapest upgrade with the biggest effect.

The danger of cache pollution

When a consumer replays a week of history, it pulls old data into the cache and pushes out the recent data that real-time consumers need. Now the fast readers also start missing the cache, and the disk becomes busy for everyone. This is called cache pollution. There are several defences:

• Buy enough memory that replays do not evict the hot data.
• Set client quotas, so no single reader can flood the broker.
• Run big replays against a separate cluster or a copy of the data.
• Use tiered storage, so old data is served from object storage instead of the brokers’ disks.

Kafka is often described as light on the processor, and that is true until a few specific features change the picture. Knowing where the cycles go helps you avoid nasty surprises.

The main consumers of CPU

The broker’s thread pools

A broker does its work with groups of threads. Network threads read and write on client connections, and the default is three. I/O threads process requests and touch the disk, and the default is eight. Separate replica fetcher threads copy data from leaders, one by default per source broker. Background threads clean and delete logs. If the request queue fills because all I/O threads are busy, latency rises. You can increase the thread counts, but they only help if there are enough physical cores to run them.

How to choose a core count

There is no honest universal formula, because so much depends on message size, TLS and compression. The dependable method is to benchmark one broker on the hardware you plan to buy, using realistic messages, the real security settings and the real compression. Find out how much traffic it can carry while the request handler threads still have spare time. Then plan to run at roughly half of that at your peak. Many production brokers use somewhere between eight and thirty-two cores, but let your own test decide.

These numbers are illustrative, not promises. Their purpose is to show the method: measure, apply a safety factor, and divide.

The network is where beginners most often undercount, because replication and fan-out multiply traffic in ways that are easy to forget.

Bytes versus bits

Storage and data rates in this guide use bytes, while network cards are sold in bits. One byte is eight bits, so multiply megabytes per second by eight to get megabits per second. A 10 gigabit card can theoretically carry about 1,250 megabytes per second. In practice, plan to use a comfortable fraction of that, since protocol overhead, encryption and bursts all take a share.

Per-broker traffic in our example

From Chapter 5, the cluster receives 180 MB/s and sends 300 MB/s at peak. Spread across nine brokers, that gives each broker about 20 MB/s incoming and about 33 MB/s outgoing.

Even after a zone failure, the busiest direction uses well under half a gigabit. A 10 gigabit card is generous here, and even a 1 gigabit card would cope. In this example the network is not the limiting resource. That is a useful finding, not an accident: it tells you where not to spend money.

When fan-out changes everything

Now imagine the same 60 MB/s topic is read by ten consumer groups instead of three: analytics, search, alerts, machine learning and several more. Outgoing traffic becomes 120 for replication plus 600 for consumers, which is 720 MB/s across the cluster, or 80 MB/s per broker. That is 640 megabits per second per broker, which would already strain a 1 gigabit card and would leave little room for failure.

Keep a margin

• Add a margin of ten to twenty percent for protocol overhead and encryption.
• Plan the busiest broker, not the average, since load is never perfectly even.
• Include the extra traffic from rebuilding a failed broker, unless you throttle it.
• Include monitoring, log shipping and backup traffic that shares the same network card.

Getting the bandwidth number right is one job. Understanding where the traffic travels, and what it costs, is another.

flowchart LR
  P["Producer"] -->|"1 copy in"| L["Leader broker"]
  L -->|"2 copies out for replication"| F["Two follower brokers"]
  L -->|"1 copy per consumer group"| G["Consumer groups"]
    

Reading the picture

One byte arrives from a producer. The leader sends it on to two followers, so two more copies cross the network. Then it sends one copy to each consumer group that reads the topic. With three consumer groups, one incoming byte causes five outgoing bytes from the leader. Sizing the outgoing side is therefore the bigger job for most clusters.

Availability zones and the cost of crossing them

Production clusters usually spread brokers across three availability zones, which are separate data center buildings within one region, so a fire or power failure in one does not stop the others. That is excellent for safety, but many cloud providers charge for data that moves between zones, typically per gigabyte. With three copies in three zones, every byte a producer sends causes two cross-zone copies for replication. Producers and consumers that talk to a leader in another zone add more.

Ways to reduce cross-zone traffic

• Rack awareness. Tell each broker which zone it lives in, so Kafka places replicas in different zones. This is essential for safety, though it does create the replication cost above.
• Fetching from the closest replica. A feature added in Kafka 2.4 lets a consumer read from a follower in its own zone instead of the leader in another. Consumers declare their zone, and brokers are configured with a replica selector. This can remove a large share of consumer cross-zone traffic, at the price of slightly higher delay, since followers trail the leader a little.
• Compression. Fewer bytes cross the boundary, on every hop.
• Fewer full-stream consumer groups. Share one consumer that feeds several destinations, when this is practical.

Check the machine’s true network limit

Cloud machines often advertise network speed as “up to” a number. That top speed may be a short burst, with a lower baseline that you can sustain. A Kafka broker that runs hot for hours lives at the baseline. Also note that if the disks are network-attached volumes, the traffic to them may be counted against a separate storage bandwidth limit of the machine, so check both figures in the provider’s documentation.

Partitions are how Kafka spreads work, but each one has a price. Too few limit your speed; too many drain your brokers.

How many partitions does a topic need?

A well-known rule links partitions to the speed you need. First measure two numbers on your hardware: how fast a single partition can accept data from producers (call it p), and how fast a single consumer can process one partition (call it c). If your target rate is t, you need at least t ÷ p partitions for writing and t ÷ c partitions for reading. Use the larger answer. For example, if the target is 60 MB/s, one partition takes 10 MB/s from producers, and one consumer processes 5 MB/s, you need at least 6 partitions to write and 12 to read, so choose 12 or more. Because you cannot reduce partitions later, and adding them changes where keyed messages land, leave room for growth; 24 would be a reasonable choice.

What each partition costs a broker

Practical limits

Widely quoted guidance from Kafka experts is to keep each broker to a few thousand partition replicas, and clusters to a couple of hundred thousand in total on older setups that use ZooKeeper. Newer clusters using the built-in KRaft controller handle much larger counts, but the per-broker costs above still apply. Treat these numbers as starting points and always test with your own topics.

Balance matters

All brokers should lead about the same number of partitions and hold about the same amount of data. Uneven clusters have one hot broker that limits everyone. Tools such as Cruise Control, an open-source rebalancer originally built at LinkedIn, can watch the load and move partitions to even it out. Balance is worth checking often, since a size plan assumes an even spread.

A cluster that runs perfectly when everything is healthy is only half designed. The other half is what happens on the worst day.

flowchart LR
  subgraph AZ1["Zone 1"]
    B1["Broker 1"]
    B2["Broker 2"]
    B3["Broker 3"]
  end
  subgraph AZ2["Zone 2"]
    B4["Broker 4"]
    B5["Broker 5"]
    B6["Broker 6"]
  end
  subgraph AZ3["Zone 3 fails"]
    B7["Broker 7"]
    B8["Broker 8"]
    B9["Broker 9"]
  end
  AZ3 -.-> AZ1
  AZ3 -.-> AZ2
    

The arithmetic of losing something

With nine brokers, losing one broker leaves eight to carry the load, which raises each survivor’s share by nine eighths, or 12.5 percent. Losing a whole zone of three leaves six, which raises each survivor’s share by nine sixths, or fifty percent. The fewer brokers you have, the harder a single loss hits. Two brokers that lose one survive on half the capacity; nine brokers that lose one lose only a ninth.

Setting the target utilization

Decide the highest load you are willing to run at after the failure, perhaps seventy or eighty percent of a broker’s tested capacity, and work backwards. If the failure multiplier is 1.5 and the post-failure ceiling is 75 percent, normal peak load should stay at or below 50 percent. This is why the earlier chapters kept talking about running at about half of tested capacity.

Safety settings that keep the cluster writable

With three copies spread over three zones, a common safe setup asks producers to wait for all in-sync replicas (acks=all) and requires at least two replicas to be in sync (min.insync.replicas=2). Losing one zone still leaves two replicas, so writes continue and no confirmed data is lost. Losing two zones stops writes, which is the correct, safe behavior.

How long does recovery take?

When a broker is replaced with an empty disk, all of its data must be copied back from other brokers. Suppose each of our brokers holds about 8.7 terabytes. If you throttle the copying to 100 MB/s to protect normal traffic, the copy takes about 87,000 seconds, which is roughly 24 hours. During that time, the cluster has fewer safe copies than normal. Faster copying finishes sooner but competes with clients.

• Smaller disks per broker mean faster recovery, but more brokers to manage.
• Reattaching a surviving data volume to a replacement machine, where the platform allows it, can skip most of the copying.
• Set the copy speed limit deliberately with replication quotas, and rehearse the procedure before you need it.

Now we combine every earlier chapter into one decision. The method is simple: calculate a broker count for each resource, take the largest, then adjust for zones and failure.

flowchart TD
  R["Requirements"] --> W["Workload math"]
  W --> D["Disk answer"]
  W --> N["Network answer"]
  W --> C["CPU answer"]
  W --> M["Memory answer"]
  D --> X["Take the largest count"]
  N --> X
  C --> X
  M --> X
  X --> F["Add failure headroom and round to a multiple of zones"]
    

The requirements

Peak producer rate 60 MB/s, average 30 MB/s, messages of about 1 KB, seven days of retention, three copies, three real-time consumer groups, TLS on, three availability zones, and doubling of traffic expected within a year.

One answer per resource

Adjusting for reality

In our worked example, disk capacity decided the size. For other workloads a different resource takes the lead. Recognizing your profile early tells you where to look first.

Once you know the total resources you need, you can build them from a few large machines or from many small ones. Both work, and each has costs.

A practical middle path

Most teams settle on a moderate number of mid-sized brokers, often between six and thirty for a single cluster, in a multiple of their zone count. Beyond that, it is often wiser to run a second cluster for a separate set of applications than to keep growing one giant cluster. This limits the blast radius, meaning how many applications a single problem can hurt.

In the cloud you rent machines by the hour, so sizing becomes a menu of instance types and storage options. The same method applies; only the building blocks change. Details below are general, and cloud limits change often, so check the provider’s current documentation.

Matching instance families to profiles

Network-attached disks

Most clusters use network-attached block volumes because they can be resized and can survive a machine replacement. Their speed is set per volume, for example a general-purpose SSD volume type offers a baseline throughput that you can raise up to a documented maximum. In addition, each instance has its own cap on how much storage traffic it can carry in total. So the achievable speed is the smaller of the volume limit and the instance limit. Throughput-oriented hard disk volumes are cheaper for large, mostly sequential workloads, but are slower for replays.

Managed Kafka services

Services such as Amazon MSK let you pick broker instance sizes and storage per broker while the provider handles machine replacement, patching and much of the monitoring. You still do the sizing: how many brokers, which size, how much storage, how many partitions. Managed offerings have their own documented limits for storage per broker, throughput per broker and partitions per broker, and some can grow storage automatically. Some also absorb the cost of replication traffic between zones. Read the current limits and pricing pages before you finalize a plan, since they change.

Cost levers to remember

• Storage is often the largest line item, so compression, retention and tiered storage matter greatly.
• Cross-zone transfer can quietly rival the machine cost for high-throughput clusters.
• Commitments such as reserved capacity lower the hourly price once your size has stabilized.
• Right-sizing later is easier in the cloud than on purchased hardware, so start with a measured plan and adjust.

Brokers rarely live alone. Several other components share the platform, and each has its own needs. Giving them the wrong home can undo careful broker sizing.

Cluster controllers

The controllers manage cluster metadata: which broker leads which partition, topic settings and access rules. Modern Kafka runs them with its own consensus system called KRaft. Production clusters usually use three, or five for higher safety, dedicated controller nodes. They need modest processors and memory, a small but reliable fast disk for the metadata log and a dependable network. Running controllers and brokers on the same machines is fine for small or test clusters, but a busy broker can then slow down cluster management just when you need it most.

Kafka Connect workers

Connect moves data between Kafka and other systems. Its workers do real processing such as converting formats and applying small transformations, which can use a lot of processor time and memory. Run them on their own machines, not on the brokers, and size them by the number of connector tasks and the volume they handle.

Schema registry and other helpers

A schema registry stores the formats of your messages. It is light on resources but must be highly available, since producers and consumers consult it. Run at least two instances and give them a small, fast disk or rely on Kafka itself for storage as designed.

Stream processing applications

Programs built with Kafka Streams or similar tools keep local state on their own disks and use their own memory. They are clients from the broker’s point of view but add load: every restart may replay a lot of data. Include them when you count consumer groups and when you think about cache pollution.

Operating system settings that support your sizing

• Raise the open-file limit and the memory-mapping limit, as described in Chapter 13.
• Keep swapping essentially off.
• Use a modern file system on data disks with access-time updates disabled.
• Keep the clocks synchronized, since replication and logs rely on time.
• Set network buffer sizes suitable for high-bandwidth links, especially between zones or regions.

A sizing plan is a prediction. Testing is how you find out whether the prediction was right, before your users do.

How to run a good test

The dashboard that tells you if sizing is right

Brokers do not decide alone how hard they work. The programs that talk to them make choices that can double or halve the load. Sizing goes better when you influence those choices early.

Batching: the single biggest lever

A producer can wait a few milliseconds to gather many messages before sending them together. Two settings control this: how long to wait (linger.ms) and how large a batch may grow (batch.size). Bigger batches mean fewer requests, better compression and less CPU on the broker. The cost is a small added delay for each message. For most data pipelines, waiting five to twenty milliseconds is invisible to users and saves a large amount of broker work.

Choosing a compression method

Ask the client teams to compress, and ask them to keep the topic’s own compression setting at “producer”, which means the broker stores batches exactly as they arrive. If the topic demands a different method, the broker must decompress and recompress every batch, and that uses processor time you did not plan for.

Acknowledgements and safety

Producers choose how many confirmations to wait for. Waiting for all in-sync replicas is the safest and adds a little latency. It does not add much extra work for the broker, but it does mean producers hold requests open longer, so account for that in connection and buffer sizing.

How consumers ask for data

Consumers fetch data in requests too. If a consumer asks for data with no minimum size and no waiting time, and the topic is quiet, it can send thousands of empty requests per second. Setting a sensible minimum amount of data (fetch.min.bytes) and a maximum wait (fetch.max.wait.ms) lets the broker hold the request until there is something worth sending. This lowers request counts and CPU for idle or slow topics.

Connections

Every client connection uses memory and file handles on the broker. Thousands of applications that each open connections to every broker add up. Encourage long-lived connections instead of opening and closing them constantly, since each new secure connection costs a handshake that burns CPU.

To see how a different profile changes the answer, let us size a cluster where many teams read the same modest stream and nobody keeps data for long.

The requirements

Peak producer rate 25 MB/s and an average of 15 MB/s. Three copies, three days of retention, twelve consumer groups each reading the full stream, TLS on, and three availability zones.

The workload math

One answer per resource

• Disk capacity: at about 8 TB usable per broker, 17 TB needs three brokers.
• Disk speed: 75 MB/s of writes is trivial.
• Network: with six brokers, each sends about 58 MB/s, roughly 467 megabits per second. A one gigabit card at 60 percent (75 MB/s) fails the zone-loss test, so choose a ten gigabit card.
• CPU: suppose benchmarks show one broker can handle about 100 MB/s of combined incoming and outgoing traffic with TLS at a comfortable load. Then 425 divided by 100 needs at least five brokers.

flowchart LR
  D["Disk: 3 brokers"] --> X["Largest is CPU at 5"]
  N["Network: 2 brokers on 10 Gbps"] --> X
  C["CPU: 5 brokers"] --> X
  X --> Z["Round up to a multiple of 3 zones: 6"]
  Z --> F["Failure check with 6 brokers"]
  F --> R["Zone loss gives 106 MB/s per broker, above the 100 target"]
  R --> S["Choose 9 brokers: 71 MB/s after zone loss"]
    

The failure check changes the answer

With six brokers, normal load is 425 ÷ 6, about 71 MB/s each. Losing one zone leaves four brokers, each carrying 425 ÷ 4, about 106 MB/s, which is above our comfortable 100 MB/s. So six is not enough. With nine brokers, normal load is about 47 MB/s each, and after losing a zone six survivors carry about 71 MB/s each, which is comfortable. The alternative is six larger brokers with more cores, which could also work if benchmarks prove it.

What else can lower the cost?

• Fetch from the closest replica to cut cross-zone charges for the twelve readers.
• Ask whether several consumer groups could share one reader that feeds them, reducing full-stream copies.
• Use light compression so each byte to twelve readers is smaller.
• Check memory: at 75 MB/s of cluster writes across nine brokers, each writes about 8 MB/s, so ten minutes of cache is only about 5 GB, and 32 GB machines are plenty.

The day you launch is the easiest day of a cluster’s life. Traffic grows, new teams arrive, and retention requests creep upward. Good sizing includes a plan for the next two years.

Forecast the three curves

• Data rate: ask product teams about launches, campaigns and new regions, then add a buffer.
• Consumer fan-out: every new team that reads a stream adds outgoing traffic.
• Retention and partitions: requests for “just keep it longer” and “just add partitions” quietly consume disk and overhead.

Decide your triggers in advance

Adding brokers is not the end of the job

Here is a fact that surprises many newcomers: when you add a broker to a running cluster, Kafka does not automatically move existing partitions onto it. The new machine sits nearly empty until you run a partition reassignment, which copies chosen partitions over to it. That copying uses network and disk on the old brokers, so apply a speed limit and do it during a quieter period. Tools such as Cruise Control can plan and execute balanced moves for you.

Here is how the ideas above show up in real organizations. Some describe publicly known systems, and others are illustrative scenarios that show how a typical team would apply the same reasoning.

Patterns are proven habits, and anti-patterns are traps that many teams fall into. Knowing both shortens the path to a stable cluster.

Patterns worth adopting

Anti-patterns to avoid

Here is a sensible order for doing this work, followed by the mistakes that trip up most teams.

A sizing routine you can follow

The most common mistakes

• Forgetting the copies. Replication multiplies disk writes and network traffic.
• Ignoring consumer fan-out. Every additional group reading the full stream adds outgoing bytes.
• Trusting burst numbers. Sustained network and disk speeds are lower than advertised peaks.
• Skipping failure math. A cluster with no headroom becomes an outage when one machine fails.
• Over-partitioning. More partitions are not free and cannot be removed later.
• Mixing workloads on broker machines. Neighbors steal cache, disk and network.
• Tuning without measuring. Changes made by guess often move the bottleneck rather than remove it.
• Never revisiting the plan. Traffic grows, and yesterday’s right answer becomes today’s risk.

A short self-check

• Do we know our peak and average rates, and did we use each in the right place?
• Could we survive losing a full zone without running above our ceiling?
• Do we know how long a broker rebuild would take?
• When did we last test with encryption on, at length, with realistic data?
• Which alerts tell us to grow, and what will we do when they fire?

Quick answers to the questions that come up most often when teams size Kafka for the first time.

Broker sizing turns a vague worry into a repeatable calculation. Four resources, a handful of formulas and a habit of testing will take you a long way.