Before we look at the setting itself, we need to see why a producer keeps records in memory at all.

Sending is not instant

Apache Kafka is a system that stores streams of events, such as orders, clicks, payments or sensor readings. Programs that write events into Kafka are called producers, and the servers that store the events are called brokers. When your code hands a record to a producer, that record does not fly to the broker straight away. The producer has to turn the record into bytes, decide which partition it belongs to, group it with other records, compress it if needed, open or reuse a network connection, and wait for the broker to answer. All of this takes time, and your application is producing new records the whole time.

The gap between producing and delivering

Because records are created by your code at one speed and delivered over the network at another speed, there is always a gap between the two. Something has to hold the records that have been created but not yet delivered. That something is memory inside the producer. Without it, your application would have to stop after every single record and wait for the network, which would make it painfully slow.

Where the buffer fits in the big picture

The producer’s buffer sits between your application code and the network. Your code writes into it quickly, and a background thread takes records out of it and sends them to the brokers. The size of this waiting room is controlled by one setting, buffer.memory, and that is the subject of this tutorial.

Producer buffer memory is the total amount of memory a producer is allowed to use for holding records that are waiting to be sent.

The plain definition

The setting buffer.memory tells the producer how many bytes it may use, in total, to store records that have been handed over by your code but have not yet been confirmed by the broker. Its default value is 33554432 bytes, which is exactly 32 megabytes. Every record waiting in a batch, every batch waiting to be sent, and every batch that has been sent but not yet acknowledged takes a share of this budget.

A budget, not a bucket you fill on purpose

Many beginners imagine the buffer as a bucket that the producer tries to fill before sending. That is not how it works. The buffer is a limit on how much can be waiting at the same time. In a healthy system the buffer is mostly empty, because records leave almost as fast as they arrive. The buffer only fills up when the producer is creating records faster than the brokers and network can take them.

Why it exists

The buffer exists for two reasons. First, it lets your application keep working while the network catches up, which smooths out short bursts of traffic. Second, it puts a firm upper limit on the memory the producer can consume. Without such a limit, a slow broker could cause the producer to keep swallowing records until the whole program ran out of memory and crashed.

A few words appear again and again when people talk about producer memory, so let us define each of them clearly.

The buffer is one part of a small team of components that work together to move your records to the brokers.

The front desk and the delivery team

When your application calls the send method, its own thread does only a little work. It converts the key and value into bytes with serializers, chooses the partition, and then asks the record accumulator for space to store the record. If the accumulator has an open batch for that partition with room left, the record is simply added to it. If not, the accumulator asks the buffer pool for a new block of memory to start a fresh batch. After that, your call returns and your code carries on.

In the background, a separate thread called the sender keeps checking the accumulator. When batches are ready, because they are full or because their waiting time has passed, the sender collects them, groups them by destination broker and sends them over the network. When a broker acknowledges a batch, the sender tells the pool, and the memory used by that batch is returned for reuse.

flowchart LR
  A["Application thread calls send"] --> B["Serialize and choose partition"]
  B --> C["Record Accumulator"]
  C -- "asks for memory" --> P["Buffer Pool: total limited by buffer.memory"]
  P -- "gives a block" --> C
  C --> D["Sender thread"]
  D --> E["Broker"]
  E -- "acknowledgement" --> D
  D -- "returns memory" --> P
        

The important detail about timing

Notice when the memory is given back. It is not returned when the batch is sent. It is returned when the batch is finished, meaning the broker has answered with success or the producer has given up with an error. A batch that is on the network, waiting for an answer, still occupies its memory. This is why a slow broker or a slow network eats into the buffer even when your application is producing at a modest pace.

The pool is more clever than a simple counter, and knowing how it behaves explains many surprising situations.

Standard blocks that are recycled

Most batches are created with a block of memory equal to batch.size. Creating and destroying memory blocks constantly would create a lot of work for the garbage collector, which is the part of the Java runtime that cleans unused memory. To avoid that, the pool keeps a list of finished standard blocks and hands them out again to new batches. This recycling makes the producer steadier and reduces pauses.

Odd-sized blocks

Sometimes a single record is larger than batch.size. In that case the producer creates a special batch big enough to hold it. Such odd-sized blocks are not kept for recycling. When they finish, their size is simply returned to the total budget, and the memory itself is left for the garbage collector. Systems that send many large records therefore put a bit more pressure on garbage collection than systems that send small ones.

Waiting in line

If the pool has no free memory when a thread asks for a block, that thread is put into a waiting line. When memory is released, waiting threads are served in the order they arrived, so no thread waits forever while newer ones jump ahead. Each waiting thread waits no longer than the time allowed by max.block.ms. If the time runs out, the send call fails with a timeout error.

Why fairness matters

Serving waiting threads in the order they arrived may sound like a small detail, but it has a real effect on behaviour under pressure. Without it, a thread that happened to ask at the right moment could repeatedly win memory while an unlucky thread waited until its time ran out. With an orderly line, the delays are spread evenly, and every caller gets a predictable experience even when the system is stressed.

Garbage collection and the pool

The Java client stores batches in ordinary memory that is managed by the Java runtime. Recycling standard blocks keeps the number of new objects low, so the garbage collector has less to clean. When you set a very large buffer or send many oversized records, the runtime has more to manage, and you may notice longer pauses. If your application is sensitive to pauses, watch garbage collection logs together with producer metrics when you change the buffer size.

This is the situation everyone worries about, and it follows a clear and predictable path.

sequenceDiagram
  participant App as Application thread
  participant Acc as Accumulator
  participant Pool as Buffer Pool
  participant Snd as Sender
  App->>Acc: send record
  Acc->>Pool: need a new block
  Note over Pool: No free memory left
  Pool-->>App: thread waits in line
  Snd->>Pool: batch acknowledged, memory returned
  Pool-->>App: memory granted, thread wakes up
  App->>Acc: record stored, send returns
        

Step one: the send call slows down

When the pool cannot supply memory, the send call does not return immediately. It blocks, which means your application thread simply pauses inside the call. This is a deliberate slowdown that gives the network time to drain the batches already waiting. From your application’s point of view, sending suddenly becomes slow, and threads that produce records may stop making progress.

Step two: the wait ends one of two ways

If space is released within max.block.ms, the thread wakes up, gets its memory and carries on as if nothing happened, only a little later. If the whole waiting time passes without space becoming available, the producer gives up on that record and reports a timeout error saying it could not allocate memory in the allowed time. The record is not stored, and your code must decide what to do about it.

Step three: the consequences spread

A blocked send is not only a problem for the producer. If the thread that is blocked is also a web request thread, the web request will hang. If it is a message consumer thread reading from another system, that consumer will slow down and fall behind. In this way a full producer buffer can create a chain of slowdowns that travels far from the place where the problem started.

A worked example with plain numbers

Imagine a producer with a 32 megabyte buffer that creates records at 8 megabytes per second. In calm conditions, brokers acknowledge batches within about 20 milliseconds, so only a small fraction of a second of data is ever waiting, roughly 160 kilobytes, and the buffer stays almost empty. Now suppose a broker becomes slow and takes two seconds to answer. Suddenly about 16 megabytes of data is waiting for answers at any moment. If the slowness grows to four seconds, the whole 32 megabytes is in use and the buffer is full. The producer did nothing different. Only the time to get an answer changed, and that alone was enough to fill the waiting room.

This example shows why the delivery speed matters more than the raw size of the buffer. Doubling the buffer would only have bought a few more seconds. Fixing the slow broker removes the pressure altogether.

The buffer setting and the blocking-time setting are partners, and you should always think about them together.

What it controls

The setting max.block.ms decides how long certain producer calls may pause your thread. It applies to waiting for buffer memory during send, and it also applies to waiting for topic metadata, which is the information about which broker leads which partition. The default is 60000 milliseconds, or one minute.

Two very different errors that look alike

Because the same time limit covers two different waits, you can see timeout errors that mean very different things. One says memory could not be allocated in time, which points to a full buffer and a slow drain. The other says the topic was not present in the metadata in time, which points to a wrong topic name, missing permissions, or a cluster that cannot be reached. Reading the error text carefully saves hours of looking in the wrong place.

Choosing a value

A long wait is friendly to short slowdowns, because the producer can ride them out. A short wait fails fast, which suits systems that would rather drop a record or divert it elsewhere than hold up a user request. There is no single correct value. A background data pipeline might keep the default or even raise it, while a web endpoint that must answer within two seconds should use a much smaller value and handle the failure explicitly.

Blocking inside frameworks

Many applications do not call the producer directly. They use a framework that calls it on their behalf, such as a web framework, a message listener or a stream processing library. In these cases the blocking happens in threads you did not create and may not think about. A listener thread that blocks for a minute cannot read new messages, and the lag of its topic grows quietly. Knowing which threads reach the producer in your own application is part of understanding what a full buffer will do to it.

Let us follow one block of buffer memory from the moment it is borrowed until it is available again.

One of the most common surprises is that the producer can use more memory than buffer.memory suggests.

A soft target, not a hard wall

The setting buffer.memory limits the memory used for holding records, but it is not a hard limit on everything the producer consumes. Other parts of the producer need memory too. Compression needs working space while it squeezes a batch. Requests that are on the network need their own structures. Metadata about topics and brokers takes a little room. Serializers may create temporary objects. All of these sit on top of the buffer.

What this means for planning

If you tell your container that a service may use 512 megabytes of memory, and the producer inside has a 32 megabyte buffer, the buffer is only a small part of the story. The rest of the application, the runtime itself and the garbage collector all need room. When teams raise buffer.memory to hundreds of megabytes without raising the total memory of the process, they sometimes cause the whole program to be killed by the operating system for using too much memory.

Each producer has its own buffer

Every producer instance owns a separate buffer. If your application creates ten producers, each with a 32 megabyte buffer, the potential memory is ten times as large. This is one more reason to share one producer across threads, as the design allows, instead of creating many.

Memory demand does not come from buffer.memory alone. It is shaped by batching settings and the number of partitions.

Batch size and partition count multiply

Each partition that is actively receiving records can have an open batch, and each open batch holds a block of about batch.size bytes. If your producer writes to 200 active partitions and batch.size is 16 KB, then roughly 3.2 megabytes are needed just for the open batches. Raise batch.size to 256 KB and the same 200 partitions need about 51 megabytes, which is already more than the default buffer. This multiplication catches teams by surprise when they tune for throughput without checking memory.

Linger time keeps memory occupied longer

The setting linger.ms makes batches wait to collect more records. While a batch waits, its memory is in use. A long linger time on a stream with many partitions means many partly filled batches sitting in the buffer at the same time. Combined with a large batch size, it can push the buffer close to its limit during busy periods.

Compression changes the picture

Records are stored uncompressed in the batch while it is being filled, and they are compressed when the batch is sent. So compression saves network and disk space, but it does not shrink the space used in the buffer while the batch is collecting. It also needs extra working memory when it runs. Do not assume that turning on compression will reduce buffer pressure.

There is no magic number, but there is a sensible way of thinking about how much memory you need.

Start from the shape of your traffic

Ask three questions. How many partitions does this producer write to actively? What is the typical batch size it builds? And how many seconds of trouble do you want to be able to absorb before sending starts to block? The first two give you the memory for normal operation. The third gives you the extra room for bad moments.

A simple mental calculation

First, estimate the memory for open batches by multiplying the active partitions by the batch size. Second, estimate how much data your producer creates per second at the busiest time. Third, multiply that rate by the number of seconds of broker slowness you want to survive, for example ten seconds. Add the first and third figures together and then add a comfortable margin. The result is a rough but sensible target for buffer.memory. For instance, a producer creating five megabytes per second that should survive ten seconds of trouble needs about 50 megabytes of waiting space, plus the open batch memory.

Test with realistic slowness

The best way to check your sizing is to test under pressure. In a test environment, slow down a broker or add network delay and watch how the buffer behaves. If it fills within seconds, your margin is too small. If it never gets past a few percent even in these tests, you may be able to reduce it and save memory for other work.

Common sizing situations

The buffer brings clear benefits, and it also brings costs that you should understand before choosing a size.

The trade is between tolerance and honesty. A large buffer tolerates trouble silently, and a small buffer complains early. Neither is wrong in general. What matters is that the choice matches what your business can accept, and that you notice when the buffer is doing more work than usual.

Cost and efficiency angle

Memory is not free, especially in cloud environments where every gigabyte of memory in a container is paid for. Running many service instances each with an oversized buffer adds up to a large bill. On the other hand, a buffer that is too small causes stalls that can cost far more in lost work and unhappy users. The sensible approach is to size the buffer from measured needs, review it after traffic patterns change, and avoid copying values from other teams without checking that their workload looks like yours.

The buffer interacts with the settings that decide whether your records are safe, so we must look at them together.

Records in memory are fragile

Everything inside the buffer lives only in the memory of your application. If the program crashes, is stopped forcefully, or its machine fails, those records are gone. Kafka cannot recover what it never received. A bigger buffer or a longer wait time before sending means more records are exposed to this risk at any moment.

The delivery timeout

The setting delivery.timeout.ms limits the total time from the moment a record is handed to the producer until it is either confirmed or declared failed. Its default is two minutes. If a batch waits in the buffer too long because the brokers are unreachable, it eventually expires. The producer then fails all the records in that batch, reports the failure to your callbacks, and frees the memory. This is an important safety valve, because without it a dead broker would keep the memory locked forever.

Retries and memory

When a request fails for a temporary reason, the producer retries. During retries, the batch stays in memory. If many batches are retrying at once, the buffer fills up rapidly. This is why you may see buffer problems appear right after a broker failure or a leadership change: the retries themselves keep the memory busy.

Shutting down cleanly

When your application stops, records may still be waiting in the buffer. Calling flush waits until everything in the buffer has been sent and acknowledged, and closing the producer does a similar job with a time limit. Skipping these steps during shutdown is one of the most common reasons for mysterious missing records after a deployment.

Transactions and the buffer

Producers that use transactions write records to several partitions and then commit them together. During a transaction the records still pass through the same buffer, and they are held there and in flight until the commit finishes. Long transactions or slow commits keep memory busy for longer, so systems using transactions should pay extra attention to buffer metrics and should keep transactions short enough that memory is returned promptly.

Restarts and rebalancing

When brokers restart or partition leaders move, requests fail for a short time and are retried. During that window the producer keeps accepting records, and the buffer absorbs them. A well-sized buffer lets a routine rolling restart of the cluster pass without any visible trouble to your application, while a tiny one turns the same restart into a burst of errors.

The buffer has no security features of its own, but it sits inside a larger system with rules you must respect.

What is held in memory

Records in the buffer are ordinary bytes in the memory of your process. They are not encrypted while they wait there. Anyone who can read the memory of the process, for example through a memory dump, could see them. Sensitive data should therefore be handled with the same care in the producer as anywhere else in your application, including limiting who can take dumps and how long such dumps are kept.

Encryption and access control on the wire

The connection between the producer and the brokers can be encrypted with TLS and protected with authentication. These features add some work to every request. Because the buffer allows batching, that work is spread across many records. If authentication or authorization fails, records cannot be delivered, and they will pile up in the buffer until timeouts arrive. A sudden permission change on a topic can therefore look like a memory problem on the producer.

Size limits

Two limits are worth remembering. The setting max.request.size limits how large a single request may be, which is about one megabyte by default. The broker also limits the size of a batch it accepts. A record that is too large is rejected at the producer or by the broker, and it does not remain in the buffer. Very large records also create odd-sized blocks that are not recycled, which puts extra pressure on memory management.

You should never learn about a full buffer from an angry customer. The producer publishes metrics that show the state of memory at any moment.

Exact metric names can vary slightly between client versions, so check the documentation of the library you use. What matters is the idea: watch how much memory is free, how many threads are waiting, and how long brokers take to answer.

Reading the metrics together

If available bytes are dropping while request latency is rising, the brokers or the network are the cause, and increasing the buffer will only delay the problem. If available bytes drop while latency is normal, your application is simply producing faster than the current setup can carry, and you should look at batching, compression, partition counts and the number of brokers. If available bytes are low only during short bursts and then recover, the buffer is doing exactly the job it was designed for.

Alerts that people trust

A useful alert is one tied to a clear action. An alert for waiting threads above zero for more than a minute, an alert for available bytes below a fifth of the total, and an alert for send timeouts are usually enough. Too many alerts teach people to ignore all of them.

Building a dashboard that tells the story

A good dashboard puts related numbers next to each other so the story is obvious at a glance. Place available buffer bytes next to request latency and the production rate. Add a panel for waiting threads and a panel for send errors. When something goes wrong, the order in which these panels change tells you what happened first. If latency rises before memory falls, the cause is downstream. If the production rate rises first, your own application is the cause. A dashboard built this way answers the first question of every incident within seconds.

When memory errors appear, a calm and ordered investigation beats random changes to settings.

This is the most important idea in the whole chapter. A full buffer is a symptom that tells you the flow of data is blocked somewhere downstream. Enlarging the buffer treats the fever and leaves the disease untouched. Always ask why records are leaving slowly before asking how to store more of them.

Teams keep repeating the same mistakes with producer memory, so it pays to learn them in advance.

The producer protects itself, but a well-designed application also helps by controlling how fast it produces.

Slow down at the source

The best place to control speed is where the records are created. If your application reads data from a file, a database or another queue, it can slow its reading when the producer reports trouble. This keeps the pressure from building up in the buffer in the first place and keeps the rest of the system healthy.

Use a bounded holding area

Some teams place a small queue with a fixed capacity between their application logic and the producer. When the queue is full, the application either waits briefly, drops low-value records, or diverts them to a different place, such as local storage. Because the queue has a fixed size, memory use stays predictable, and the producer’s blocking never spreads to the rest of the program.

Decide what deserves protection

Not every record has the same value. A payment event must never be dropped, while a debug log line might be. By deciding in advance which records are precious and which are disposable, you can choose different behaviours under pressure, such as retrying the first kind and dropping the second. Making this choice calmly in a design meeting is far better than making it in the middle of an outage.

Testing your fallback behaviour

Designing fallbacks is only half the job. They must also be tested, because code that runs only during an outage is the code most likely to contain mistakes. Simulate a stopped broker in a test environment, watch what your application does when memory timeouts begin, and confirm that records are diverted or dropped exactly as planned. Repeat this test after major changes, and you will be far more relaxed the first time a real incident arrives.

The same setting leads to very different choices depending on what the business needs.

No. It limits the memory for holding waiting records, but the producer also needs extra memory for compression, in-flight requests, metadata and internal objects. Treat it as a close guide, not a strict ceiling for the whole producer.

The default is 33554432 bytes, which is 32 megabytes. Always check the documentation for your client library, since other language clients may choose different defaults.

The buffer fills up. Once it is full, send calls block for up to max.block.ms. If space frees up in time, sending continues. If not, the call fails with a timeout error.

Yes, until they are acknowledged or fail. A batch that is on its way to the broker still occupies its memory, and only after the answer arrives is the memory returned to the pool.

Not while batches are being filled. Records wait uncompressed in the batch and are compressed when the batch is sent. Compression saves network and storage, and it needs a little extra working memory.

Not as a first step. First find out why delivery is slow, since that is usually the real problem. A bigger buffer can hide the cause and make the eventual failure larger and harder to understand.

You can set a very small max.block.ms so that calls fail almost at once instead of waiting, or place a bounded queue in front of the producer. In both cases you must handle the failed records deliberately, as they are not stored.

Each open batch reserves a block of about batch.size bytes, and the pool recycles blocks of this size. More active partitions and a larger batch size therefore mean more of the buffer is in use even in calm times.

No. They exist only in the memory of your process. Use acknowledgements, callbacks, flush at checkpoints and clean shutdown to reduce the risk, and keep the buffer no larger than your business can tolerate.

No. Each producer instance has its own buffer, so total memory grows with the number of instances. Sharing one producer across threads is usually both simpler and lighter.

Both are limited by max.block.ms but they mean different things. A memory timeout says the buffer stayed full, pointing to slow delivery. A metadata timeout says the producer could not learn about the topic, pointing to a wrong name, missing permission or an unreachable cluster.

Watch for waiting threads, a high wait ratio, exhausted-buffer events and available bytes that regularly fall close to zero during normal traffic. If these appear only during rare incidents, the size may be fine.

If it exceeds delivery.timeout.ms, the producer expires it, fails its records through the callbacks and frees the memory. This stops a dead broker from holding memory forever.

Not necessarily. It tolerates slowness for longer, but it also puts more unsent records at risk in a crash. Safety comes from acknowledgements, idempotence, callbacks and clean shutdown, not from a bigger waiting room.

No. Producer settings are read when the producer is created. To use a new value you must create a new producer with the new configuration, which in practice usually means restarting the application or replacing the producer object carefully.

Not by itself. Speed comes from batching, compression and healthy brokers. A bigger buffer only gives the producer more room to absorb bursts. If the brokers are the bottleneck, extra room only delays the moment the buffer fills.

Producer buffer memory is the waiting room where a Kafka producer keeps records that have been created but not yet confirmed by the brokers. It is controlled by buffer.memory, which defaults to 32 megabytes, and it lets your application keep working smoothly while batching, compression and network trips happen in the background. When records are created faster than they are delivered, the waiting room fills up, send calls begin to block, and after max.block.ms they fail. Understanding how batches borrow and return memory, how partition counts and batch sizes multiply the demand, and how to watch the right metrics turns this setting from a mystery into a tool you can control. Most importantly, a full buffer is a signal that something downstream is slow, and finding that cause is more valuable than simply enlarging the buffer.