Before diving into the producer itself, let us see its place in the larger Kafka picture.

The story so far

Apache Kafka is a system for storing and moving streams of events, such as orders, payments, clicks, or sensor readings. Events are grouped into named categories called topics, and each topic is split into slices called partitions. The servers that store the partitions are called brokers, and a team of brokers is called a cluster.

Brokers are patient. They store what they receive and hand out what is requested. They never go out to look for events. Something has to bring events to them. That something is the producer.

The simple definition

A producer is a client, meaning a program or library inside an application, that writes events into Kafka topics. Whenever a customer places an order, a sensor takes a reading, or a user clicks a button, the application can hand that fact to a producer, and the producer makes sure it is safely recorded in Kafka.

A producer sends records. Each record has a few simple parts.

When an application sends one event, a whole chain of small steps happens inside the producer.

flowchart LR
  A["Application"] --> SR["Serializer"]
  SR --> PT["Partitioner"]
  PT --> BF["Batch Buffer per Partition"]
  BF --> SN["Sender Thread"]
  SN --> LB["Leader Broker"]
  LB --> AC["Acknowledgement"]
  AC --> A
        

Kafka does not understand customers, orders, or sensors. It only understands bytes.

Why serialization exists

Your application works with meaningful data, such as an order with a number, a list of items, and a total. Kafka stores raw bytes and treats them as opaque. A serializer is the small translator that turns your data into bytes before sending. Later, a deserializer on the consumer side turns the bytes back. Both sides must agree on the format.

Common formats

Schemas keep teams safe

A schema is a written description of an event’s shape, listing its fields and their types. A schema registry is a separate service that stores these descriptions and checks that new versions stay compatible with old ones. Without such a check, a producer team might rename a field, and every consumer relying on the old name would break the next morning.

Deciding which partition gets each event is one of the producer’s most important jobs.

Three ways of choosing

flowchart TD
  R["Record to send"] --> Q{"Explicit partition given?"}
  Q -->|"Yes"| X["Use that partition"]
  Q -->|"No"| K{"Does it have a key?"}
  K -->|"Yes"| H["Calculate partition from the key"]
  K -->|"No"| S["Spread across partitions to balance"]
        

Why keys protect order

Kafka guarantees order only inside a single partition. If a customer creates an account, changes the address, and then closes the account, those three events must be read in that order. By using the customer number as the key, all three go to the same partition, and their order is preserved. Events with different keys may be read in any relative order, and that is usually fine.

A producer is fast not because it sends events one by one at lightning speed, but because it sends them together.

What is a batch?

A batch is a small group of records headed for the same partition, sent together in one network request. Sending one thousand events in a single request is far cheaper than sending one thousand separate requests, because the overhead of the network trip is shared.

The two dials

The trade-off

Waiting a little for a fuller batch adds a tiny delay to each event but greatly improves throughput and compression. If you want the lowest possible delay, use a very short wait, and accept smaller batches and more network requests. If you want maximum throughput, allow a slightly longer wait and larger batches. Most systems choose a small wait of a few milliseconds as a good balance.

flowchart LR
  R1["Record 1"] --> B["Batch for Partition 0"]
  R2["Record 2"] --> B
  R3["Record 3"] --> B
  B --> T{"Full or linger time reached?"}
  T -->|"Yes"| S["Send batch to leader broker"]
  T -->|"No"| W["Keep waiting for more records"]
        

Since batches contain many similar records, they compress very well.

How it works

The producer can compress each batch before sending it. The broker stores the batch in its compressed form, and the consumer unpacks it. This reduces network traffic, disk space, and sometimes even delay, because less data has to travel and be written. Text-based formats like JSON compress especially well because they contain a lot of repetition.

Choosing a method

The most important producer decision is how many confirmations to wait for before calling a write successful.

The three levels

Why the “all” level is safer

Each partition has a leader and followers on different brokers. The followers that are fully caught up are called in-sync replicas. When the producer asks for the “all” level, the leader confirms only after every in-sync replica has stored the event. If the leader then crashes, a follower with the complete data takes over, and nothing confirmed is lost.

The minimum in-sync rule

Topics also have a minimum number of in-sync replicas required to accept a write. With three copies and a minimum of two, the topic keeps working when one broker is down, because two copies still confirm each event. If two brokers fail, the topic refuses new writes rather than risk keeping only one copy. Kafka chooses safety over availability here, and the producer receives an error it must handle.

sequenceDiagram
  participant P as Producer
  participant L as Leader Broker
  participant F1 as Follower 1
  participant F2 as Follower 2
  P->>L: Send batch
  L->>L: Append to log
  F1->>L: Fetch new data
  L-->>F1: Return data
  F2->>L: Fetch new data
  L-->>F2: Return data
  L-->>P: Confirm after in-sync replicas have it
        

Networks and servers are imperfect. A good producer expects trouble and recovers gracefully.

Temporary trouble

Sometimes a broker is briefly unavailable, a leader is changing after a failure, or a network packet is lost. These problems are temporary, and the producer handles them by trying again. Kafka producers retry automatically, within an overall delivery time limit that you can set. If the event still cannot be delivered within that limit, the producer gives up and reports an error to the application.

The duplicate problem

Retries create a subtle danger. Suppose the producer sends a batch, and the broker stores it, but the confirmation is lost on the way back. The producer, seeing no answer, sends the same batch again. Now the partition holds the same events twice. Without protection, retries can cause duplicates.

The reordering problem

There is a second danger. If the producer allows several requests in flight to the same broker at once, and the first fails and is retried while the second succeeds, the events could end up stored in the wrong order. For applications that need strict order, this must be prevented.

The idempotent producer

Kafka’s answer to both dangers is the idempotent producer. When enabled, each producer receives a unique identity and numbers its batches for each partition. The broker remembers the latest numbers it has seen and quietly ignores repeats, and it also uses the numbers to keep batches in the right order even when retries happen. The result is that retries become safe, with no duplicates and no reordering inside a partition. Modern client versions enable this protection by default, and it works together with the strongest confirmation level.

Producers are designed so that your application does not stand around waiting for each event to be confirmed.

Three ways to send

Why callbacks matter

Because sending is asynchronous, the call that hands over the record returns before the event is actually stored. The real result arrives later. If your application never checks that result, failures disappear without a trace. A callback lets the application log the problem, retry in its own way, or raise an alert.

Sometimes several writes must succeed together or not at all.

The need

Imagine an application that reads a payment event, updates a balance, and writes two new events: one to a “receipts” topic and one to an “audit” topic. If the application crashes after writing the first but before the second, the system is left in an inconsistent state. What is needed is an all-or-nothing guarantee.

How Kafka transactions help

The producer can begin a transaction, write to several partitions and topics, and then either commit or abort the transaction. If it commits, all the writes become visible together. If it aborts or crashes, none of them become visible to readers who ask for committed data only. Consumers choose whether to read everything, or only events from committed transactions.

A dependable application prepares for failure. These are the situations every producer team should understand.

Retriable and non-retriable errors

Some errors are temporary, and trying again is the right move. Others are permanent, and no number of retries will help, such as a message that is too large or an identity without permission. A good producer retries the first kind automatically and reports the second kind immediately to the application.

flowchart TD
  E["Send fails"] --> Q{"Is the error temporary?"}
  Q -->|"Yes"| R["Retry within the delivery time limit"]
  R --> OK{"Success in time?"}
  OK -->|"Yes"| D["Event stored"]
  OK -->|"No"| F["Report failure to application"]
  Q -->|"No"| F
  F --> H["Log, alert, or store in fallback"]
        

There is no single perfect configuration. The right choices depend on how precious your events are and how fast they arrive.

Three typical profiles

The way Kafka producers work has clear strengths and clear costs.

The in-memory risk

Events waiting in the producer’s buffer live only in memory. If the application process crashes or is killed before the buffer is sent, those events are gone. This is why applications should shut down gracefully by flushing pending events, and why critical systems sometimes write events to a database first and then publish them, so nothing is lost between the business action and Kafka.

A producer carries valuable business facts, so it must connect safely and be watched closely.

Connecting safely

Producers can use encrypted connections so that nobody watching the network can read the events. They can prove their identity to brokers through certificates or through a username and secret, which is called authentication. Brokers can then check rules about which identities may write to which topics, which is called authorisation. A well-run company gives each application its own identity, allows it to write only to the topics it needs, and stores passwords and certificates in a proper secrets system rather than inside the program.

What to watch

Also watch the average batch size, which reveals whether batching is working, and the compression ratio, which shows how much space is saved. A steadily full buffer means events are created faster than the cluster accepts them. A rising retry rate points to unstable networks or overloaded brokers. Alert on error rates and delivery failures, not only on averages, so small problems are fixed before customers notice.

Most producer problems repeat across companies. Learn them once and avoid them forever.

A short checklist keeps your producers healthy.

Let us see a producer at work in an online shop during a busy sale.

flowchart LR
  C["Customer Clicks Buy"] --> OS["Order Service"]
  OS --> PR["Shared Producer"]
  PR --> K["Kafka Cluster: orders topic"]
  K --> EM["Email Service"]
  K --> FR["Fraud Service"]
  K --> WH["Warehouse Service"]
  PR --> CB["Callback: log result or alert on failure"]
        

A producer is the client that writes events into Kafka topics, handling serialization, partition choice, batching, delivery, and retries on the application’s behalf.

No. Producers and consumers are completely separate. The producer only writes to a topic, and any number of consumer groups can read it later.

It starts with a few bootstrap broker addresses and asks for the cluster map, called metadata. The map tells it which broker leads each partition, so it sends each batch straight to the right leader.

Yes, if configured carelessly. Using no confirmations, ignoring send results, or crashing before the buffer is flushed can all lose events. With full confirmations, idempotence, replication, and proper result handling, loss becomes very unlikely.

Retries after a lost confirmation can store the same batch twice if idempotence is off. Enable the idempotent producer, and also make consumers safe against seeing an event twice.

The producer spreads events across partitions to balance the load. There is no ordering guarantee between those events.

One for the whole application, or a small number when needed. Creating a producer for every event is slow and wasteful, and it defeats batching.

By default, sending is asynchronous. The call returns once the record is accepted into the buffer, and the real result arrives later through a callback or result object. You may choose to wait for the result, though this reduces throughput.

Use them when several writes, often across topics, must succeed together or not at all, such as in stream processing that reads, transforms, and writes results. For independent events, they are usually unnecessary.

No. It holds events only briefly in memory before sending them. The durable, replicated copy is stored by the brokers.

We began with a courier company that checks labels, sorts parcels into sacks, waits for receipts, and retries when roads are closed. A Kafka producer does the same for your events. It serializes each record, chooses a partition using the key, gathers records into batches, optionally compresses them, sends them to the partition leader, and waits for the confirmations you asked for. Idempotence protects against duplicates and reordering during retries, callbacks report the real outcome, and transactions offer all-or-nothing writes when needed. The producer’s power comes from a handful of choices: how sure you want to be, how fast you want to go, and how carefully you handle failures. Choose them deliberately, monitor them continuously, and your events will travel safely from your application into Kafka.