Before we talk about grouping messages, we need to be clear about who is sending them and where they are going.

Kafka in one paragraph

Apache Kafka is a system that stores streams of events, such as clicks on a website, payments, temperature readings or log lines. Kafka was originally built at LinkedIn to move huge amounts of activity data, and it is now used across banking, retail, travel and streaming companies. Programs that write events into Kafka are called producers. Programs that read events out are called consumers. The servers in the middle that store the events are called brokers, and a group of brokers is called a cluster.

Topics and partitions

Events are written into named channels called topics. Each topic is split into smaller pieces called partitions, and each partition lives on one broker at a time, which is called its leader. When a producer sends an event, it must deliver that event to the leader broker of the correct partition. This detail matters a lot for batching, because messages headed for the same partition and the same broker are the ones that can travel together.

Why the cost of one request matters

Every time a producer talks to a broker, it pays a fixed price no matter how small the message is. It has to build a request, send it across the network, wait for the broker to write the data to disk, and wait for an acknowledgement to come back. That fixed price is similar whether the request carries one tiny message or five hundred. Once you notice this, the idea behind batching becomes obvious: pay the fixed price once and carry as much as you sensibly can.

Batching means collecting several messages together and sending them as one unit instead of one at a time.

The plain definition

A batch is a small group of records that are all going to the same topic partition. The producer keeps them together in memory and then ships the whole group to the broker in a single go. The broker writes the group to its log as one chunk and sends back one acknowledgement for the entire chunk. Nothing about the messages themselves changes. Each message keeps its own key, value and timestamp, and consumers still see them as individual events.

Why batching exists

Batching exists because network round trips and disk writes are expensive compared to the size of a typical event. A click event may be only a few hundred bytes. Sending it alone wastes most of the request on headers and waiting. Batching spreads that overhead across many records, which raises the number of events per second the whole system can handle, lowers CPU use on both client and broker, and reduces the number of tiny disk writes.

Where batching is used

Batching is used in every Kafka producer client, whether it is written in Java, Python, Go or another language. It is on by default and you cannot really switch it off completely. What you can do is influence how big the batches get and how long the producer is willing to wait to fill them, and that is where our two settings come in.

A handful of words appear again and again in any discussion of producer batching, so let us define them once.

To understand linger.ms, you must know that a producer is not a single straight pipe. It has a front desk and a delivery team working at the same time.

The two sides of a producer

When your application calls the send method, your own thread does only a small amount of work. It converts the key and value into bytes using serializers, decides which partition the record belongs to, and then drops the record into the record accumulator. After that, your call returns immediately and your code moves on. The record has not necessarily left your machine yet.

Meanwhile a separate background thread, the sender, keeps looking at the accumulator. Whenever it finds batches that are ready, it groups them by the destination broker and sends them out. This split between a fast front desk and a background delivery team is what makes batching possible in the first place, because the front desk can keep collecting while the delivery team is busy on the road.

flowchart LR
  A["Your application thread"] --> B["Serializer"]
  B --> C["Partitioner"]
  C --> D["Record Accumulator: one queue of batches per partition"]
  D --> E["Sender thread"]
  E --> F["Broker 1 leader partitions"]
  E --> G["Broker 2 leader partitions"]
  F --> H["Acknowledgement"]
  G --> H
        

The diagram shows that batches are built in the accumulator, one queue per partition. The sender thread does not send partition by partition. Instead it looks at which broker leads which partitions and packs all ready batches for the same broker into a single network request. That is a second layer of grouping on top of the first, and it further cuts down the number of requests.

The first of the two settings decides how big a batch is allowed to become.

What batch.size really controls

The setting batch.size is measured in bytes, not in the number of messages. Its default value is 16384 bytes, which is 16 KB. It defines the size of the memory block reserved for one batch of one partition. When the records in a batch add up to roughly this size, the batch is considered full and becomes eligible to be sent immediately, without waiting for any timer.

A common misunderstanding

Many beginners think batch.size is a target that the producer always tries to reach, or that the producer will wait until every batch is full. Neither is true. It is only an upper limit. A batch can leave the producer with much less data than this number if some other rule, such as the linger time, says it is time to go. Also, if a single record is larger than batch.size, the producer still sends it, as a batch containing just that one record.

Memory cost per partition

Because a batch buffer is created for each partition that is actively receiving records, the memory used by batching grows with the number of partitions. A producer writing to 500 partitions with a large batch size needs far more memory for its buffers than a producer writing to five partitions. This is one of the main reasons not to raise batch.size blindly.

The second setting decides how patient the producer is.

The definition

The setting linger.ms is the number of milliseconds the producer is willing to wait, after the first record enters a new batch, for more records to join it before sending the batch. Think of it as a small artificial delay that you add on purpose. If the batch fills up to batch.size before the time is over, it leaves early. If the time is over first, it leaves with whatever it has.

The default values

For many years the default for linger.ms was zero, which meant do not wait deliberately. In newer Kafka clients, starting with the 4.0 generation, the default was raised to five milliseconds so that out-of-the-box producers get better batching without extra tuning. Because defaults differ between client versions and between client libraries in other languages, always check the documentation for the exact version you run.

The clock starts with the first record

The linger timer for a batch begins when the first record lands in it, not when the producer starts. This means a quiet topic with one message every few seconds will see each message wait up to linger.ms alone and then be sent by itself. Waiting cannot help if nobody else is coming to join the batch. This is why high linger values make sense for busy streams but are pointless, and slightly harmful, for slow ones.

The two settings are two doors out of the same room, and whichever opens first wins.

The rules for sending a batch

A batch becomes ready to leave the accumulator when any one of several conditions is met. The most common are that the batch is full, meaning it has reached batch.size, or that the batch has been waiting for linger.ms. Other reasons exist too: the application calls flush, the producer is being closed, or the memory pool is running short and the producer needs to free space. The sender thread checks these conditions all the time and picks up whatever is ready.

flowchart TD
  A["New record arrives for a partition"] --> B["Is there an open batch with space?"]
  B -- "Yes" --> C["Add record to existing batch"]
  B -- "No" --> D["Open a new batch and start the linger timer"]
  C --> E{"Batch full or linger.ms expired?"}
  D --> E
  E -- "Yes" --> F["Batch is ready: sender sends it"]
  E -- "No" --> G["Keep collecting more records"]
  G --> E
        

Four situations, four outcomes

This last point is worth remembering. Under heavy load, batches fill up before the timer expires, so linger.ms adds almost no delay at all. Under light load, the timer is the one that fires, and that is when the small extra delay is really felt. The added latency is therefore highest exactly when there is the least benefit, which is something to keep in mind when choosing values.

A worked example with plain numbers

Suppose a producer writes to one partition with batch.size at 16 KB, linger.ms at 20 milliseconds, and each record is about 1 KB. If records arrive at 2000 per second, that is one record every half millisecond. In 20 milliseconds, about 40 records could arrive, but a 16 KB batch holds only around 16 of them. So the batch fills in roughly 8 milliseconds and leaves early, and the linger timer never gets to fire. Now suppose the arrival rate drops to 200 records per second, one every 5 milliseconds. In 20 milliseconds only about four records arrive, so the timer fires and a batch of four leaves. Same settings, very different behaviour, and all because of the arrival rate.

Why the sender is not always idle

There is one more subtle detail. The sender thread can only have a limited number of requests in flight to a broker. While all those slots are busy waiting for answers, new batches simply sit in the accumulator and keep growing until they are full or a slot frees up. This means that on a slow network or a busy broker, batches automatically become bigger even without any linger time. The system adapts by itself, and linger.ms only sets the minimum patience on top of that natural behaviour.

Let us follow one record from the moment your code sends it until you learn that it was safely stored.

sequenceDiagram
  participant App as Application
  participant Acc as Accumulator
  participant Snd as Sender
  participant Brk as Broker
  App->>Acc: send record 1
  Note over Acc: New batch opened, linger timer starts
  App->>Acc: send record 2
  App->>Acc: send record 3
  Note over Acc: linger.ms expires or batch is full
  Snd->>Acc: collect ready batches
  Snd->>Brk: one request with records 1, 2, 3
  Brk-->>Snd: one acknowledgement
  Snd-->>App: callbacks for all three records
        

Compression works on a whole batch at a time, so bigger batches directly mean better compression.

Why compression loves batches

When a producer compresses data, it looks for repeated patterns. A single small message has few repeated patterns, so it compresses poorly. A batch of hundreds of similar messages, such as JSON events with the same field names, has enormous repetition and can shrink dramatically. This is why the compression setting and the batching settings should always be thought of together rather than separately.

Choices of compression type

Since a batch only ever holds records for one partition, the way records are spread across partitions changes how well batching works.

The problem with spreading too thin

Suppose your producer sends 1000 records per second to a topic with 100 partitions, and the records are spread perfectly evenly. Each partition then gets only about ten records per second. With a short linger time, each batch will hold only one or two records when it leaves. Perfect evenness looks fair, but it gives you many tiny batches, which is the opposite of what batching wants.

The sticky approach for records without keys

To fix this, modern Kafka producers use a sticky strategy for records that have no key. Instead of rotating to a new partition for every record, the producer sticks to one partition until the current batch is complete, and then switches to another one. Over time, the load is still spread evenly, but each batch fills up much faster. This change in the default partitioner, introduced in the 2.4 release and refined in later versions, gave many teams better throughput and lower latency without touching a single setting.

Records with keys

When records have keys, the key decides the partition, so the producer has no freedom to be sticky. If the keys are spread across a huge number of partitions, each batch stays small. In this case, a slightly larger linger.ms is the main tool for improving batch sizes. If instead a few keys are extremely popular, their partitions will fill batches instantly, while other partitions stay almost empty. This is called a hot partition problem, and batching cannot fix it on its own.

What changed in newer versions

The sticky idea was refined again in later releases so that the producer also takes the speed of each broker into account. Partitions on brokers that are answering slowly receive somewhat fewer records, and partitions on healthy brokers receive somewhat more. The aim is to keep every batch filling nicely while avoiding piling up work on a struggling broker. You do not need to configure this to benefit from it, but knowing that it exists helps you understand why upgrading client libraries sometimes improves batching results by itself.

Custom partitioners and their side effects

Some teams write their own partitioner to route certain customers or regions to certain partitions. This is a valid technique, but it changes batching in a direct way. If a custom rule sends most records to a handful of partitions, those batches fill instantly and the rest stay nearly empty. If it spreads records across all partitions in strict rotation, each batch stays small. Whenever you change partitioning logic, look at the batch metrics again, because a new partitioner is effectively a new batching setup.

All those waiting batches live in memory, and memory is not unlimited.

The buffer pool

The producer has a total memory budget for holding unsent records, controlled by the setting buffer.memory, whose default is 32 MB. Every open batch takes a piece of this budget. When batches are sent and acknowledged, their memory is returned to the pool and reused. As long as records are sent as fast as your application produces them, the pool never runs empty.

What happens when the pool is full

If your application produces faster than the brokers can accept, batches pile up until the pool is exhausted. At that point, the send call starts to block, waiting for space. It waits for up to the time set in max.block.ms, which defaults to 60 seconds. If space does not free up within that time, the call fails with a timeout error. This blocking is a form of back-pressure, and it is actually a safety feature: it slows down your application instead of letting it run out of memory.

Batching is very powerful, but it is a trade, not a free gift.

The key to using batching well is to know how much delay your use case can accept. A payment confirmation that a customer is staring at needs an answer in tens of milliseconds. A log pipeline feeding a dashboard that refreshes every minute can easily tolerate a few hundred milliseconds of waiting if it saves half the cluster load.

Thinking in terms of a latency budget

A helpful habit is to think of latency as a budget that is spent in several places. Part of it goes to your own code, part to serialization, part to time spent waiting in the accumulator, part to the network trip, part to the broker writing and replicating the data, and part to the answer coming back. The linger time is only one item on that list, and it is the one you control most directly. If your total budget is 200 milliseconds and the rest of the path already uses 150, you have at most about 50 milliseconds to give to linger.ms, and choosing something smaller leaves a healthy safety margin.

Cost is a benefit that is easy to forget

Batching also saves real money. Cloud providers often charge for network traffic between zones, and Kafka replication multiplies that traffic. Compressed and well-filled batches reduce the bytes that cross those boundaries, and they reduce the number of brokers you need to run for the same workload. A team that lifts average batch size from a few hundred bytes to several kilobytes may find it can serve the same traffic with a smaller cluster, which is a benefit that finance teams notice very quickly.

Batching interacts with the settings that keep your data safe, and it is important to know how.

Acknowledgements

The setting acks decides when the broker tells the producer that a write succeeded. With acks set to all, the leader waits until the in-sync replicas have also stored the batch. Newer Kafka clients use all by default. Because the acknowledgement is per request, a larger batch means that one slow acknowledgement covers many records, and one failure affects many records at once.

Retries and timeouts

When a request fails for a temporary reason, the producer retries automatically. Several settings control the limits. The request.timeout.ms setting limits how long the producer waits for a single answer, and delivery.timeout.ms limits the total time from send to final success or failure, with a default of two minutes. The linger time counts toward this total time, so a very large linger.ms leaves less room for retries. As a rule, delivery.timeout.ms must be at least as large as linger.ms plus request.timeout.ms.

Ordering and in-flight requests

Kafka guarantees order only within a partition. The producer may have several requests on the way to the same broker at once, controlled by max.in.flight.requests.per.connection, which defaults to five. Without protection, a retry of an earlier batch could arrive after a later batch, which would break the order. Turning on idempotence, which is the default in recent clients, lets the broker detect duplicates and keep the order intact even with several requests in flight.

The crash window

Records sitting in the accumulator exist only in the memory of your application. If the application crashes or is killed before they are sent, those records are gone. A longer linger time widens this window slightly. For most systems the window is tiny, but if you cannot tolerate any loss, you should call flush at meaningful checkpoints, wait for callbacks before acknowledging upstream work, and shut the producer down cleanly.

Why a failed batch affects many records

Because the broker accepts or rejects a batch as a whole, a problem with the batch affects every record in it. If the request times out, all its records are retried together. If the broker reports that the batch is too large, all its records fail together. This is not a reason to avoid batching, but it is a reason to keep batches at a sensible size, to handle failures in callbacks, and to make downstream processing safe against seeing the same event twice in unusual situations.

Transactions and batching

Producers that use transactions to write to several partitions atomically still use batching underneath. Records inside a transaction are batched in the usual way, and the transaction is completed with a commit step. Very short transactions with only a few records will limit how much batching can happen, so systems that use transactions often group more work into each transaction to keep batches healthy.

Batching does not change your security model, but it does interact with size limits and encrypted connections.

Encryption overhead

When the connection between producer and broker is encrypted with TLS, each request carries extra work for encrypting and decrypting data. Because batching reduces the number of requests and puts more data into each one, the fixed part of this cost is shared across many records. Teams that turn on TLS often see a batching setup help them keep throughput stable.

Size limits you must respect

Three limits work together. The producer setting max.request.size caps the size of one request and defaults to about one megabyte. On the broker side, the setting message.max.bytes, or its per-topic form max.message.bytes, caps the size of a batch the broker will accept. If a compressed batch is larger than the broker limit, the request is rejected with a message-too-large error. When you raise batch.size for higher throughput, always check these other limits so the bigger batches are not refused.

Authorization and quotas

Access control lists decide which producers may write to which topics, and quotas can limit how many bytes per second a producer may send. Batching does not bypass either one. Quotas are measured in bytes and requests, and a well-batched producer usually stays inside them more comfortably because it makes fewer requests for the same amount of data.

You should never tune batching by guessing. The producer publishes metrics that tell you exactly what is going on.

How to read them together

If batch-size-avg is tiny and records-per-request-avg is close to one, you are sending too many small requests, and a higher linger.ms is worth trying. If record-queue-time-avg is already close to your latency budget, you have no room to raise it further. If buffer-available-bytes keeps shrinking, look at memory and broker speed before touching batching values. Reading the metrics as a group tells the whole story, while reading one alone can mislead you.

Logs and dashboards should also track the broker side, such as request rate, produce request time and disk write rate. A successful batching change typically shows up as a drop in requests per second and CPU use on the brokers while message rate stays the same.

Setting alerts that make sense

Useful alerts are based on trends and thresholds tied to your goals. An alert on record-queue-time-avg tells you when the delay from waiting is approaching your latency budget. An alert on buffer-available-bytes tells you the producer is close to blocking. An alert on error and retry rates warns you that batches are failing and being resent, which wastes bandwidth. It is better to have a few alerts that people trust than a long list that everyone ignores.

Comparing before and after fairly

When you judge the effect of a change, compare periods with similar traffic. Comparing a busy Monday morning with a quiet Sunday night will fool you into thinking a setting helped or hurt when only the load changed. Keep notes about what was changed and when, and if possible run the old and new settings side by side on similar instances so the comparison is honest.

There is no single perfect value, but there are well-known starting points for common situations.

A safe method for changing values

Starting values worth remembering

These numbers are starting points for experiments, not rules carved in stone. Your record sizes, partition counts and hardware will move the best values up or down. What matters is that you test, measure and choose with evidence instead of copying a number from a blog post, including this one.

Teams repeat the same mistakes with batching, so it helps to learn them in advance.

The same thinking appears on the reading side of Kafka, which helps the whole idea feel natural.

Consumers also trade a little waiting for efficiency. The setting fetch.min.bytes tells the broker not to answer a fetch request until at least that many bytes are available, and fetch.max.wait.ms limits how long the broker will wait for that amount. The first is like batch.size in spirit and the second is like linger.ms, only from the reader’s point of view.

Seeing the pairing makes it easier to remember. Wherever a system moves data between two machines, you will find the same two questions: how much should we gather, and how long are we willing to wait? Kafka simply exposes both answers as settings that you can change to fit your own workload.

The same two settings produce very different choices depending on the business.

No. It is a maximum wait, counted from the first record in a batch. If the batch fills up sooner, or if the sender is free and other rules trigger a send, records leave earlier. Under heavy traffic, the actual added delay is often close to zero.

Yes. The producer always uses batches. The settings only decide how large they can grow and how long the producer will deliberately wait to fill them.

Bytes. It is a memory size per batch, not a count of records. A batch with many tiny records and a batch with a few large records may therefore both be considered full.

No. A batch is always for one partition. However, batches for different partitions that live on the same broker can be sent together inside a single request.

The producer still sends it, in a batch that contains only that record. It does not reject it because of batch.size, but the request and broker size limits still apply.

Not blindly. Beyond a certain point, batches already fill up quickly and extra waiting adds delay, memory use and crash risk with little extra gain. Increase it step by step while watching the metrics.

Not by itself. Records waiting in memory can be lost if the application crashes before sending, so use acknowledgements, idempotence, callbacks, flush and clean shutdown wherever loss is unacceptable.

Calling flush tells the producer to send everything in the accumulator right now, ignoring the linger timer, and to wait until all of it is acknowledged. It is useful at checkpoints but should not be called after every record, since that would destroy batching.

Yes. The linger time default was zero for a long time and was raised to five milliseconds in the newer client generation. Other language clients may also have their own defaults, so always check the documentation for your exact library and version.

Look at average batch size, records per request and queue time together. If batches are small, requests carry few records and you still have latency room, a higher linger time is likely to help. If queue time is already near your limit, stop raising it.

No. Bigger batches help until the fixed costs are already well shared. After that point, extra size adds memory use, larger retries and possible size-limit errors while giving very little extra speed. The gain flattens out, and the right size is the one where the curve starts to flatten.

A single producer instance has one set of settings, so the usual approach is to run separate producer instances for topics with very different needs, such as one for latency-sensitive events and another for bulk data. This keeps each stream tuned for its own purpose.

Most likely your traffic is light enough that batches are leaving because of the timer rather than because they are full. In that case every record pays the waiting time. Lower the value, or check whether more records could be routed to fewer partitions so batches fill faster.

No. Kafka producers do not ask you to build lists. You send records one at a time and the client library groups them for you behind the scenes, according to the settings you choose. Your application code stays simple while the library handles the grouping.

Batching is the way a Kafka producer avoids paying the full price of a network trip for every single message. It gathers records for the same partition, compresses them together, packs batches for the same broker into one request and receives one acknowledgement for the lot. The two settings that shape this behaviour are batch.size, which limits how big a batch may grow, and linger.ms, which limits how long the producer will wait to fill it. Whichever limit is reached first sends the batch on its way. Used with care, and checked against real metrics, these two settings can raise throughput, cut cost and keep latency well within what your business needs.