Batch size tuning is the practice of choosing how many bytes and how much waiting time a Kafka producer uses to group messages before sending them.

Start with the basic words. A message, also called a record, is a small piece of data such as “customer 17 clicked the buy button”. A producer is a program that sends messages to Kafka. A topic is a named log where messages are stored, and each topic is split into partitions, which are separate ordered logs that can live on different servers called brokers.

A batch is a group of messages that a producer sends to one partition in a single unit. Instead of sending one hundred separate messages over the network, the producer packs them into one batch, adds one header, and sends them together. The broker stores the batch as it arrived and later hands it to consumers as a batch as well.

Batch size tuning means adjusting the settings that control batches so that the producer sends the right amount of data at the right times. The main settings are batch.size, the maximum size of a batch in bytes, and linger.ms, how long the producer may wait for more messages before sending. Around them sit compression, memory, request limits and acknowledgment settings, which all interact.

The goal of tuning

Every request has a fixed cost that does not depend on how much data it carries. Batching spreads that cost over many messages.

When a producer sends data to a broker, a lot happens besides moving the message bytes. The producer builds a request with headers. The operating system copies data into network buffers. The network carries packets and confirmations. On the broker, a network thread reads the request, a request handler thread checks it, writes it to the log, and waits for replicas to copy it before acknowledging. All of this work has a fixed part and a part that grows with the number of bytes. Sending ten thousand tiny requests pays the fixed part ten thousand times. Sending one hundred batches pays it a hundred times.

What goes wrong when batches are too small

To tune batching you need a picture of what happens between your code calling send and the broker replying.

A Kafka producer is not a single loop. It is a small pipeline with two kinds of threads. Your application threads call send and return quickly. A separate background sender thread does the network work. Between them sits a shared memory area called the record accumulator, which holds batches waiting to be sent.

flowchart LR
  APP["Application Threads calling send"] --> SER["Serializer"]
  SER --> PART["Partitioner chooses partition"]
  PART --> ACC["Record Accumulator with one batch queue per partition"]
  ACC --> SEND["Sender Thread"]
  SEND --> NET["Network Client with requests per broker"]
  NET --> BR["Kafka Broker"]
  BR --> ACK["Acknowledgment"]
  ACK --> CB["Callback to application"]
  BUF["Buffer Memory Pool"] --> ACC

The stages in plain words

A batch is not just a convenient bundle in memory. It is the same structure that goes over the network and onto the disk.

Kafka uses a single format for a batch of records, called a record batch, in memory, on the wire and in the log files on the broker. This design means the broker can usually write what it receives directly to disk and send it later to consumers without unpacking and repacking it. That is one of the reasons Kafka is so fast.

Why this matters for tuning

batch.size is the maximum number of bytes the producer will put into one batch for one partition.

The setting batch.size is measured in bytes. Its long-standing default is 16,384 bytes, which is 16 kibibytes. When the open batch for a partition reaches this size, the batch is closed and becomes ready to send. A new batch is opened for the next message. The word to notice is maximum. The producer never waits until a batch is full merely because you set a large number. It only uses batch.size as a ceiling.

What batch.size does and does not do

The memory side effect

The producer takes memory from its buffer pool in chunks of batch.size for each new batch. If you set batch.size to one megabyte and write to two hundred partitions, the producer may try to hold two hundred megabytes of open batches, far beyond the default buffer of thirty-two megabytes. When the pool is short of memory, sending slows and calls may block. So a very large batch.size is not free, and it must be balanced against the number of active partitions.

Records larger than batch.size

If a single record is bigger than batch.size, the producer does not refuse it. It creates a batch just big enough for that one record. This means large messages travel alone, and the batching benefit is lost for them. If most of your messages are larger than batch.size, raising batch.size, or reducing message size, is the first thing to consider.

linger.ms is the small delay that gives a batch a chance to fill before it is sent.

The setting linger.ms tells the producer how many milliseconds it may wait for more messages to join a batch before sending it. Older Kafka versions default to zero, which means “send as soon as the sender thread is free”. Newer releases, starting with the Kafka 4.0 line, changed the default to a few milliseconds. Check the documentation of your version.

With a linger of zero, batching still happens, but only by accident. While the sender thread is busy with an earlier request, new messages pile up in the accumulator and are sent together in the next round. Under light traffic, however, each message may go alone, because the sender is idle and finds one message waiting. A small positive linger creates a deliberate window in which messages can gather.

Linger is a maximum delay, not a fixed delay

Linger.ms does not add a fixed pause to every message. If the batch fills up sooner, it is sent immediately. Linger is the longest a message will wait. So at high traffic, where batches fill quickly, a large linger costs nothing, and at low traffic, it caps the extra delay at the linger value.

Several conditions can make a batch ready. Knowing them explains most of the surprises you will see in metrics.

flowchart TD
  A["Message added to partition batch"] --> B{"Is batch full at batch.size"}
  B -->|"Yes"| R["Batch is ready to send"]
  B -->|"No"| C{"Has linger.ms passed since batch was created"}
  C -->|"Yes"| R
  C -->|"No"| D{"Is buffer memory exhausted or flush called"}
  D -->|"Yes"| R
  D -->|"No"| W["Keep waiting for more messages"]
  R --> S["Sender thread includes batch in request to its broker"]

The ride-along effect

When at least one batch for a broker is ready, the sender builds a request for that broker and usually includes waiting batches for other partitions on the same broker, even if they are neither full nor expired. This means that the batch size you observe can be smaller than what linger.ms and batch.size alone would predict. It also means that spreading writes across many partitions on the same broker does not always cause the smallest possible batches, but it does cause smaller ones than concentrating writes would.

Batches are per partition, so how messages spread over partitions decides how easily batches fill.

Suppose a producer sends ten thousand messages per second. If all go to one partition, each batch fills quickly. If they are spread evenly over one hundred partitions, each partition receives only one hundred messages per second, and each batch takes one hundred times longer to fill. The same total traffic yields far smaller batches.

Sticky partitioning for messages without keys

For messages without a key, older producers rotated through partitions message by message, which produced many tiny batches. Since Kafka 2.4, the default strategy is sticky: the producer sends all keyless messages to one partition until its batch is full or the linger time has passed, then switches to another partition. Over time the load is balanced, but each individual batch is larger. Later versions improved this by adapting to broker speed so that slow brokers receive fewer batches.

Compression and batch size are partners. Bigger batches compress better, and compressed batches move faster.

The producer setting compression.type chooses whether and how batches are compressed. The options are none, gzip, snappy, lz4 and zstd. Compression is applied to the whole batch, not message by message. Similar messages placed together share patterns, such as repeated field names in JSON, that the compressor can replace with short references.

How batch size affects compression

A batch containing one message of two hundred bytes compresses badly, because the compressor has nothing to compare it with. A batch of one hundred similar messages compresses very well. This is why two producers with identical data can see very different compression ratios: the one with better batching wins. When you tune batch size upward, expect the compression ratio to improve, up to a point of diminishing returns.

Where the CPU cost lands

The producer compresses batches, using its own CPU. Brokers normally store batches as they are, when the topic compression setting is left at its default of “producer”, so they do not spend CPU recompressing. Consumers decompress batches when reading. Spending CPU at the producer to save network, broker disk and consumer fetch bandwidth is often a good trade for high-volume topics.

Batches wait in memory. When memory runs out, the producer pushes back on your code.

The setting buffer.memory is the total memory the producer may use to hold batches that are waiting to be sent or waiting for acknowledgment. The default is thirty-two mebibytes. If your application produces faster than the brokers can accept, the buffer fills. When it is full, calls to send block for up to max.block.ms, which defaults to sixty seconds, and then fail with a timeout. This blocking is called backpressure, and it protects the producer from running out of memory.

flowchart TD
  A["Application calls send"] --> B{"Is buffer memory available"}
  B -->|"Yes"| C["Message joins a batch in the accumulator"]
  B -->|"No"| D["Call blocks and waits"]
  D --> E{"Space freed before max.block.ms"}
  E -->|"Yes"| C
  E -->|"No"| F["Send fails with a timeout error"]
  C --> G["Sender thread sends the batch and frees memory after acknowledgment"]
  G --> B

Why the buffer fills

Several limits sit around batches. If they disagree, messages are rejected.

How the limits relate

Notice that the broker limit and the topic limit apply to a batch, not to a single record, in modern Kafka versions. A producer batch that is larger than the broker limit is rejected with a “message too large” error. The producer limit max.request.size applies to the whole request, so a request carrying many batches cannot exceed it. To send large messages safely, all three levels must agree: the producer, the broker or topic, and the replication fetch size.

How long a producer waits for the broker changes how many batches can be in motion, and so changes batching behavior.

The setting acks decides when the producer considers a batch delivered. With acks set to zero it does not wait at all. With one it waits for the leader to write. With all it waits until every in-sync replica has the batch. Stronger acknowledgment means the broker takes longer to reply, and the producer holds the batch in memory for longer.

The setting max.in.flight.requests.per.connection, by default five, limits how many unacknowledged requests the producer may have outstanding on one connection to a broker. While the connection is at its limit, new messages keep gathering in the accumulator and form larger batches. Slower acknowledgment therefore tends to produce larger batches by itself: the sender thread has less chance to send early, so more messages wait together.

Every tuning choice moves you along one line: more waiting gives more efficiency, less waiting gives faster individual messages.

Throughput is how many messages, or bytes, the system moves per second. Latency is how long one message takes from creation to safe storage. Batching improves throughput because each request carries more data. It can worsen latency because messages wait for company. The two are in tension, but the tension is gentler than most people expect, because a small amount of waiting often brings most of the benefit.

flowchart LR
  A["Tiny batches and no linger"] --> B["Lowest wait per message"]
  A --> C["Highest request rate and CPU use"]
  D["Moderate batches and small linger"] --> E["Small added wait"]
  D --> F["Much lower request rate and good compression"]
  G["Huge batches and long linger"] --> H["Long wait for early messages"]
  G --> I["Little extra gain and more memory used"]

The shape of the curve

A small calculation shows why linger.ms often matters more than batch.size.

A service sends ten thousand messages per second, each one kilobyte in size, to a topic with ten partitions. Keys are well distributed, so each partition receives about one thousand messages per second, which is one megabyte per second, or roughly one kilobyte per millisecond. The producer uses the default batch.size of sixteen kibibytes and a linger of zero.

With the defaults

Because linger is zero, the sender thread ships whatever is waiting the moment it is free. A request to a broker takes about two milliseconds round trip. During those two milliseconds, each partition collects about two messages. So every batch holds about two kilobytes, far below the sixteen kibibyte limit. The producer sends about five hundred requests per second per broker, each carrying tiny batches.

With a linger of ten milliseconds

Now each partition may collect messages for up to ten milliseconds. At one kilobyte per millisecond, a batch holds about ten kilobytes when the timer expires, still under the size limit. Requests to each broker drop to about one hundred per second, one fifth as many, and each batch is five times bigger. Compression improves and broker work falls. The price is at most ten milliseconds of extra delay.

With a larger batch size and linger of twenty milliseconds

With batch.size raised to sixty-four kibibytes and a linger of twenty milliseconds, a batch would grow to about twenty kilobytes before the timer fires. The size limit is not reached, so raising it made no difference. Linger was the limiting factor. This example shows the general rule: at moderate rates, linger decides the batch size, and batch.size only sets the ceiling.

You cannot tune what you cannot see. The producer publishes exactly the numbers you need.

Read them as a group

Well-tuned producers make brokers calmer. Broker metrics confirm that your tuning helped.

The broker measures how long it takes to handle each type of request and how busy its threads are. After tuning producers, look for these changes on the broker.

Interpreting produce time

The total time of a produce request is made of parts: waiting in the queue, local processing, waiting for followers when acks is all, and sending the response. Bigger batches slightly increase the local and replication parts, since there is more data, but they reduce the queue part, because there are fewer requests. Overall, a well-tuned producer lowers the load, even if a single request takes a little longer.

Change one thing at a time, measure, and stop when the target is met.

flowchart TD
  A["Set targets and measure baseline"] --> B["Change one setting"]
  B --> C["Run realistic load"]
  C --> D{"Targets met and metrics healthy"}
  D -->|"No"| B
  D -->|"Yes"| E["Test failure cases"]
  E --> F["Document settings and set alerts"]

Different jobs need different starting points. These recipes are places to begin, not final answers.

How to use a recipe

Consumers have their own version of batching. It controls how much data one fetch request brings back.

Just as producers batch messages before sending, consumers batch what they receive. A consumer sends a fetch request to a broker, and the broker replies with data from one or more partitions. Several settings shape that reply.

How they resemble producer settings

Raising fetch.min.bytes reduces the number of fetch requests and lowers broker load, at the cost of latency up to fetch.max.wait.ms. This is a good change for consumers that process data in bulk and do not need every message the moment it arrives. On the other hand, a consumer that reads a topic with very large batches must have fetch limits big enough to hold at least one full batch, or it cannot make progress.

You often use producers indirectly. The same settings apply, but you must know where to set them.

The commit interval effect in Kafka Streams

Kafka Streams flushes its producers whenever it commits progress. With exactly-once processing, the commit interval defaults to a tenth of a second, which means batches are forced out at least ten times per second regardless of linger settings. Very small commit intervals therefore limit batch size. If your Streams application shows tiny batches and you can accept a slightly longer delay in results, raising the commit interval is the effective lever.

A simple estimate shows whether your batch settings fit inside the memory you have given the producer.

Memory used by open batches is roughly the number of partitions the producer writes to actively, multiplied by batch.size, multiplied by the number of batches per partition that may be waiting or in flight at once. To this add space for compressed copies and for the requests that are being sent. A generous rule is to allow two to three times batch.size per active partition.

An example

A producer instance writes to 120 partitions with a batch.size of 128 kibibytes. One open batch on each partition takes 120 times 128 kibibytes, which is about 15 mebibytes. Allowing three times that for in-flight batches gives about 45 mebibytes. The default buffer of 32 mebibytes is too small, so the producer would spend time waiting for memory. Either raise the buffer to 64 mebibytes, or reduce the batch size to 64 kibibytes, which brings the estimate to about 22 mebibytes.

Batching is one of the best bargains in Kafka, but it is not free. Know both sides.

Trade-offs in plain words

Batching decides how far one producer, and one cluster, can scale.

Each producer instance has a maximum request rate it can sustain before its sender thread or network becomes the limit. Bigger batches lift that limit because each request carries more work. When you add more producer instances, though, each one sends its own small batches unless it too is tuned. Ten instances with poor batching can produce ten times the request load of one.

Batching interacts with failures. Understand what happens to a batch when things go wrong.

Retries

When a request fails with a temporary error, the producer retries the whole batch. With idempotence enabled, the broker recognizes repeated batches and discards duplicates, so retries are safe. Without idempotence, a retry after a lost acknowledgment may store the batch twice. Large batches make a duplicate larger, which is another reason to keep idempotence on.

Timeouts

The delivery timeout covers everything from the moment a record is added until it is acknowledged, including the time spent waiting in the buffer. If linger, retries and broker delays together exceed that limit, records fail. Keep the delivery timeout larger than linger plus the request timeout.

Batch splitting

If a batch turns out to be too large for the broker to accept, the producer may split it into smaller ones and try again. This wastes work. Metrics for batch splits, when present, are a hint that size limits are not aligned.

Producer crashes

Records still sitting in the buffer are lost if the application dies. Larger linger times mean more records at risk. Applications that cannot lose data should call flush during a controlled shutdown, and should wait for acknowledgments before treating work as done.

flowchart TD
  A["Batch sent to leader"] --> B{"Acknowledged in time"}
  B -->|"Yes"| C["Records marked done"]
  B -->|"No, temporary error"| D["Retry same batch"]
  D --> E{"Idempotence on"}
  E -->|"Yes"| F["Broker drops duplicates"]
  E -->|"No"| G["Possible duplicate batch"]
  D --> H{"Delivery timeout reached"}
  H -->|"Yes"| I["Records fail and error is reported"]

Batch settings look like performance knobs, but they also touch security and fair use.

Turn the metrics of chapter fifteen into dashboards and alerts that catch problems early.

Logging

Producer logs at the warning level report timeouts, expired batches and disconnections. Send them to a central place and search them during incidents. Log the effective configuration at startup, so you can see what values a running instance actually uses, rather than what you believe you set.

When something breaks, batch metrics often show the first clue.

flowchart TD
  A["Producer slow or failing"] --> B{"Send calls blocking"}
  B -->|"Yes"| C["Check buffer and acknowledgment speed"]
  B -->|"No"| D{"Batches expiring"}
  D -->|"Yes"| E["Check broker health and timeouts"]
  D -->|"No"| F{"Batches tiny"}
  F -->|"Yes"| G["Check linger, flush calls, partitions"]
  F -->|"No"| H["Check compression and CPU"]

Measure with realistic tests before changing production.

Kafka ships with a producer performance test tool. It generates messages of a chosen size at a chosen rate and reports throughput and latency percentiles. You pass the same producer settings you plan to use. It is useful for comparing settings side by side on a test cluster.

Patterns that repeatedly work, and mistakes that repeatedly cause trouble.

A short checklist to keep tuning sane.

These are illustrations of common situations, not reports about specific companies.

Plain meanings for the words used in this guide.

Short answers to questions that come up again and again.

Batch size tuning is the craft of balancing waiting time against efficiency, one measured step at a time.

Kafka producers gather records into batches per partition and send them to brokers together. Batch size, linger, compression and memory decide how full those batches are. Good tuning lowers the load on brokers and networks while keeping delay within what the business accepts. It starts with a latency budget, proceeds with careful measurement, and ends with monitoring that keeps the result healthy.