A short refresher on the parts of Kafka that a producer benchmark touches.

Apache Kafka is a system for moving streams of messages between programs. A producer is a program that sends messages. The servers that receive, store and serve them are brokers, and a group of brokers working together is a cluster. Messages are organised into named channels called topics. Each topic is split into partitions, which are ordered lists of messages stored on brokers. Splitting a topic into partitions lets many brokers share the work.

Each partition has one leader broker that handles all reads and writes for it, and copies called replicas on other brokers. Replicas are the safety net: if the leader dies, another replica takes over. When a producer sends a message, it goes to the leader of the right partition, and the leader passes it to the replicas that follow it.

Measuring is how guesses become decisions.

Kafka is used for very different jobs. One team streams tiny click events at huge volume. Another moves large files as messages. A third needs every message safely stored on three machines before it is acknowledged. Each job stresses the system differently, so published numbers from other people’s tests rarely match your case. Benchmarking gives you your own numbers.

A small program that sends a chosen number of messages as fast, or as slowly, as you ask, and reports how it went.

The tool is a command-line program included with Kafka. It creates a producer, generates messages of a size you choose, sends them to a topic, and prints progress and a final summary. Depending on how Kafka was installed it may be a shell script under the name kafka-producer-perf-test, with a “.sh” ending in plain Apache downloads, and it wraps a Java class inside the Kafka tools. It needs no code from you.

What it is not

What happens between pressing Enter and seeing the final line.

flowchart LR
  G["Message generator
size and payload"] --> P["Producer buffers and batches"]
  P --> N["Network"]
  N --> L["Partition leader broker"]
  L --> R["Follower replicas"]
  R --> L
  L -->|"acknowledgement"| P
  P --> M["Latency and throughput recorded"]

Two clocks, two answers

Throughput is how much work is done per second: messages per second, or megabytes per second. Latency is how long one message takes from being sent to being acknowledged. They are related but different. A system can move enormous volume while each message waits a while in a batch. Or it can answer each message very fast while moving less in total. A good benchmark reports both together.

The tool prints lines while it runs and a summary at the end. Each number has a precise meaning.

Why percentiles beat averages

Suppose 990 messages take 5 milliseconds and 10 messages take 500 milliseconds. The average is about 10 milliseconds, which sounds fine. But one message in a hundred waited half a second. A user who happens to hit those slow ones sees a slow service. Percentiles show this. The 99th percentile here is close to 500 milliseconds, which tells the truth the average hides.

A latency number is only as meaningful as the stopwatch behind it.

In this tool, latency for each message is generally the time between handing the message to the producer and receiving the broker’s acknowledgement in a callback. That includes time waiting in the producer’s buffer for a batch to fill, the network trip, the broker’s work including replication when the settings require it, and the trip back. It does not include the time for a consumer to read the message. So this is send latency, not end-to-end latency.

Most of the difference between a slow and a fast run lives in a handful of producer settings.

Safety settings are performance settings

Choosing acks to be all and enabling idempotence makes messages safer. It also changes speed. A benchmark that uses the fastest, least safe settings will show impressive numbers that production, which needs safety, can never reach. Always test with the durability settings you will actually use.

The producer is only half the story. The topic you send to shapes the answer.

A topic with one partition is limited by one leader broker, one disk path and one lane. If your real topic has twelve partitions and three brokers, the test topic should look the same. Otherwise the test measures something you do not run.

You cannot maximise both at once. Every benchmark chooses a point on a curve.

Batching is the main reason. Waiting to fill a batch moves more data per trip, so throughput rises, but the first message in the batch waits, so latency rises too. Push the load higher and higher and you eventually hit the saturation point, where something in the path is fully busy. Past it, extra load does not raise throughput. It only makes messages queue, and latency climbs sharply.

flowchart LR
  A["Low load
fast, low throughput"] --> B["Moderate load
good balance"]
  B --> C["Near saturation
throughput flattens, latency rises"]
  C --> D["Overload
queues build, latency explodes"]

The throughput option decides whether the tool pushes as hard as it can or sends at a steady chosen rate.

Setting the throughput target to minus one removes the limit, and the tool sends as fast as the producer and cluster allow. This finds the maximum throughput. Setting a positive target, for example 50,000 messages per second, makes the tool pace itself to that rate. That measures latency at a given load, which is how real applications behave: they send at the rate their users create, not as fast as possible.

A subtle measurement trap makes systems look faster than they are under stress.

Imagine a shop where customers arrive every minute, but the cashier sometimes freezes for ten minutes. A naive stopwatch measures each customer’s service time from the moment the cashier starts on them. The frozen period barely shows up, because the queue of customers who arrived during the freeze is never timed from their arrival. This hides the real delay users felt. The trap is called coordinated omission: the measurement slows down along with the system, so the worst moments are under-counted.

A load generator that sends the next message only after the previous one is handled, or that pauses when the system is slow, can fall into this. Kafka’s producer is asynchronous and buffers messages, so the situation is less severe than with a simple one-at-a-time client. Still, if the producer’s buffer fills and the send call blocks, time spent blocked may not be counted in the message’s measured latency. The result is that overload can look milder than it is.

What you send changes how fast it goes, especially when compression is on.

The tool can generate messages of a given size from generated data, or read sample messages from a file you provide. Generated data is often simple and repetitive, so it compresses far better than real business data such as JSON with many different values. If you enable compression and test with easy data, the numbers will look wonderful and mean little.

The first seconds of a run are not typical. Measure the part that is.

At the start, several slow things happen at once. The producer must find the cluster, open connections and load metadata. The Java runtime is still optimising its code as it runs. Broker caches are cold. Operating-system memory for the files is not yet filled. Results from this period are often worse than the steady rate that follows. A run that is too short measures mostly the warm-up.

The same command on two setups gives two answers. Describe the setup fully.

A useful sanity check is to do the arithmetic. If one producer sends 200 megabytes a second to a topic with three copies, the cluster receives 200 from the producer and moves 400 more between brokers for the followers, and writes 600 megabytes a second to disks in total. If the network link or disks in your setup cannot carry that, the benchmark will find that limit long before Kafka itself is the bottleneck.

Good benchmarks are small experiments with a question, a hypothesis and controlled conditions.

A matrix lists the combinations you will try, so the work stays organised and complete.

Notice how each row differs from a neighbour in one way, so comparisons stay clean. A full grid of every combination would explode in size. Start with the questions that matter most, and add rows only when a result raises a new question.

A result is a symptom. The useful question is which part ran out first.

At the ceiling, one resource is fully busy. Finding it tells you what to change. Improving anything else is wasted effort. The method is to watch all the main resources during the test and see which one is pinned.

flowchart TD
  S["Throughput stops rising"] --> Q1{"Client CPU near 100 percent?"}
  Q1 -- "Yes" --> C1["Client is the limit: add producer processes or machines"]
  Q1 -- "No" --> Q2{"Network link near its limit?"}
  Q2 -- "Yes" --> C2["Network is the limit: compress, or use faster links"]
  Q2 -- "No" --> Q3{"Broker CPU near 100 percent?"}
  Q3 -- "Yes" --> C3["Broker processing is the limit: more brokers or threads"]
  Q3 -- "No" --> Q4{"Disk busy or slow?"}
  Q4 -- "Yes" --> C4["Disk is the limit: faster storage, more disks or brokers"]
  Q4 -- "No" --> C5["Check partitions, settings, replication and pauses"]

Let us walk through a small study from question to conclusion. The numbers are made up to show the method, not to describe any real cluster.

Most bad benchmarks fail in the same handful of ways.

Two correctness features change performance and deserve their own test rows.

An idempotent producer makes retries safe, so a message resent after a timeout is not stored twice. The broker tracks sequence numbers per producer and partition to detect duplicates. This adds a little bookkeeping and puts some limits on other settings, but the cost is often small compared with the safety it gives. In recent Kafka versions it is enabled by default, though you should confirm this for your version.

A transactional producer groups several sends into an all-or-nothing unit, used for exactly-once processing across topics. The tool can be configured with a transactional identifier and a transaction duration, so that it commits after a chosen number of milliseconds. Transactions add extra round trips and coordination, so throughput and latency are affected, and the effect depends strongly on how long each transaction lasts. Short transactions pay a fixed cost often. Long transactions pay it rarely but hold messages back from readers longer.

Encryption and login cost performance, and test tools can create risk if handled carelessly.

The tool’s summary tells you what happened. Monitoring tells you why.

One tool process can only push so hard. To test bigger clusters you need more load.

A single producer process has limits: its own threads, its network link, and the partitions it happens to be writing to. Real systems have many producer applications at once. To model that, run several copies of the tool in parallel, on separate machines if needed, and add up their results. Give each copy its own share of the target rate.

The producer test is half of a full picture.

Kafka also ships a consumer performance tool that reads messages and reports how fast it does so. It is used the same way in spirit: choose a topic and a number of messages, and read the throughput. There are also end-to-end approaches, in which messages carry a creation timestamp and the consumer subtracts it from the arrival time, giving the delay from creation to reading. That is what a user of the data really feels.

The goal of most benchmarks is a decision: how much do we need?

Suppose tests show that each broker safely handles about 60 megabytes a second of incoming data at the knee with three copies. For 50 megabytes a second of peak load, three brokers would run near their safe limit with no failure allowance. Adding a fourth or fifth broker gives room to survive a broker failure and to grow. The exact numbers will differ for you, but the steps are the same.

A result without its conditions cannot be trusted or reproduced.

A benchmark you run once helps once. A benchmark you can rerun helps every release.

Because the tool is a command-line program, it fits easily into scripts and automated pipelines. A short, standard benchmark can run after each significant change: a new Kafka version, a configuration change, a new instance type, or a new network policy. Comparing each result against a stored baseline reveals slowdowns early, when the cause is still easy to find.

Teams reach for producer benchmarks in a few recurring situations.

Batching is the single biggest lever on producer throughput, so it deserves a closer look.

The producer keeps one open batch per partition it is writing to. A batch is sent when it is full, when its linger time runs out, or when the producer is closing or flushing. Several ready batches for the same broker are grouped into one request. So the size of what actually travels depends on three things: how fast messages arrive, how many partitions they spread over, and the two limits, batch size and linger time.

flowchart TD
  M["Message arrives"] --> B{"Open batch for its partition?"}
  B -- "No" --> N["Start new batch and linger timer"]
  B -- "Yes" --> A["Add to batch"]
  N --> A
  A --> F{"Batch full?"}
  F -- "Yes" --> S["Ready to send"]
  F -- "No" --> T{"Linger time reached?"}
  T -- "Yes" --> S
  T -- "No" --> W["Wait for more messages"]
  S --> R["Grouped with other ready batches for the same broker"]

Worked numbers

Suppose messages are 1 KB and the batch limit is 16 KB. A full batch holds about sixteen messages. If 60,000 messages a second are spread over twelve partitions, each partition receives about 5,000 a second, so a batch could fill in about three milliseconds. A linger of five milliseconds is then long enough for nearly full batches. If the same rate spreads over 240 partitions, each gets about 250 a second, a batch would need sixty-four milliseconds to fill, and linger will send small batches long before that. The lesson: the right settings depend on the arrival rate per partition, not just the total.

Compression trades processor time for smaller data. Whether it wins depends on which resource is scarce.

The producer compresses each whole batch before sending, so bigger batches usually compress better, since there are more repeated patterns inside them. Brokers typically store the batch in its compressed form and pass it on to consumers, who decompress it. So a good compression ratio saves network from producer to broker, disk space, replication traffic and network to consumers, at the cost of processor time on the producer and on consumers.

Partitions are how Kafka spreads work, and how a benchmark exposes whether the spread is enough.

Each partition is written by its leader, and a leader handles its partitions using shared broker threads. Too few partitions and some brokers or disks sit idle. Too many and each carries less traffic, batches become thin, and the cluster spends more effort on bookkeeping and replication between partitions. There is a sweet spot, and a benchmark is the way to find it.

Leader balance

Check that leaders are spread evenly. If one broker leads most partitions, it carries most of the write load, and the benchmark finds that one broker’s limit instead of the cluster’s. Look at the distribution before you start, and rebalance leaders if needed.

Performance on a good day is only half of the story. What happens when something breaks?

Real clusters lose brokers, suffer slow disks and drop network packets. A benchmark that only runs in perfect health will not reveal how your setup behaves in trouble. Combine a steady, throttled load with a planned failure and watch what the producer sees.

A few working habits recur in good benchmarks, and a few recur in bad ones.

Patterns that work

Anti-patterns to avoid

A checklist to keep by your side.

Some beliefs sound sensible and lead teams astray.

Modern platforms add layers that can silently change your results.

A complete plan from start to finish, in the order to do things.

You do not need advanced mathematics to avoid fooling yourself. A few simple habits are enough.

Suppose the baseline gives 200, 210 and 205 thousand messages a second, and a variant gives 215, 220 and 212. The variant’s average is about 216 and the baseline’s about 205. Is the variant better? The spread inside each group is about 10 thousand. The gap between groups is about 11 thousand. That is close to the noise, so the honest answer is “probably slightly better, but run more repeats”. If the variant had given 250, 255 and 248, the gap would be far larger than the spread and you could be confident.

When the producer produces faster than the cluster accepts, memory fills, and behaviour changes.

The producer keeps unsent messages in a pool of memory. Messages enter the pool when the application sends them and leave when the broker acknowledges them. If messages arrive faster than they leave, the pool fills. Then the producer applies back-pressure: the send call waits for space, up to a configured maximum blocking time, and then fails. This is a healthy safety valve, but it changes what a benchmark measures.

flowchart LR
  A["Application sends"] --> P["Buffer pool"]
  P --> S["Sender thread"]
  S --> B["Broker"]
  B -->|"acknowledgement frees space"| P
  P -->|"pool full"| W["Send call waits, then may fail"]

A run that completes is not always a run that succeeded. Look for hidden trouble.

When a send fails for a temporary reason, such as a leader change, the producer retries automatically within limits set by its timeouts. Retries hide short problems from the application, but they add delay, and they consume capacity. A benchmark that shows good throughput while retries quietly climb is telling you the system is under strain.

The lines printed during the run are a time series. Reading them in order tells a story the final summary cannot.

While it runs, the tool prints a line for each reporting interval, giving the count, rate and latency for just that stretch. If you copy those lines into a spreadsheet and plot rate and latency against time, patterns appear at once.

When a benchmark disappoints, work through these ideas in order of cost and likelihood.

Good records turn a pile of runs into evidence anyone can check.

flowchart TD
  S["Study folder"] --> P["Plan: question, goal, decision rule"]
  S --> E["Environment notes: versions, machines, network"]
  S --> R["Runs"]
  R --> R1["Run folder: command, settings, raw output"]
  R --> R2["Run folder: monitoring graphs and logs"]
  S --> A["Analysis: tables, charts, conclusions"]

Where to begin your first test runs for common goals. These are starting hypotheses to test, never final answers.

A quick reference for the terms used in this guide.

The whole method in a few lines.

The kafka-producer-perf-test tool sends messages of a chosen size, at a chosen rate, with chosen producer settings, and reports throughput and latency percentiles. On its own it produces numbers. A method turns those numbers into knowledge: ask a clear question, mimic production in message shape, durability and topic layout, warm up and measure steady state, change one variable at a time, repeat to learn the noise, watch every resource to find the bottleneck, and report results with their conditions. Use unlimited runs to find ceilings, and fixed-rate ladders to find the knee where latency turns bad. Remember that send latency is not end-to-end latency, that saturated runs flatter latency, and that safety settings are performance settings. Done this way, a benchmark supports real decisions about sizing, tuning and upgrades.