Producing simply means a program creating a piece of data and handing it to Kafka to be stored and eventually delivered.

In Kafka’s world, any program that creates and sends data is called a producer. A producer might be a website’s checkout service announcing “an order was placed,” a mobile app reporting “the user tapped this button,” or a factory sensor reporting “the temperature just changed.” Whatever the source, the act of producing is the same: package up a small piece of information, called a message or an event, and send it to a specific topic, Kafka’s named, ordered stream for that kind of data.

It is tempting to imagine this as a single, instant action, throw the message and forget it, but Kafka producers are actually careful, deliberate pieces of software. They decide exactly where a message should go, how to describe it, whether to wait for confirmation, and what to do if something goes wrong along the way.

Before a message can be sent, it needs to be assembled from a few distinct pieces, each with its own job.

A topic is usually split into several partitions, and every message needs a clear, predictable way of landing in exactly one of them.

When a message includes a key, Kafka runs that key through a consistent formula, called a hashing function, which always produces the same partition number for the same key. This means every message about, say, “customer 88” will always land in the same partition, keeping that customer’s history in perfect order.

When a message has no key at all, Kafka instead spreads messages across all available partitions in a rotating, balanced pattern, so that no single partition becomes overloaded while others sit idle. This is a reasonable default when strict per-entity ordering does not matter.

flowchart LR
    M1["Message: key=customer-88"] --> H{Hash the Key}
    H --> P1[Partition 1]
    M2["Message: key=customer-42"] --> H2{Hash the Key}
    H2 --> P2[Partition 2]
    M3["Message: no key"] --> R{Round-Robin}
    R --> P0[Partition 0]
    R --> P1
    R --> P2
        

Before a message can travel across a network, it must be converted from a program’s internal data structure into a flat, transportable format.

This conversion process is called serialization. A producer takes a structured piece of data, perhaps an object representing an order with a customer name, items, and a total price, and flattens it into a simple sequence of bytes that any system can transmit and store, regardless of what programming language wrote it. On the receiving end, a consumer reverses this process, called deserialization, turning the bytes back into a usable structure.

From the instant a program decides to send something, to the moment it is durably stored, a message passes through several distinct stages.

sequenceDiagram
    participant App as Application
    participant Prod as Producer
    participant B as Broker (Leader)
    App->>Prod: Build message
    Prod->>Prod: Serialize + choose partition
    Prod->>Prod: Add to batch
    Prod->>B: Send batch
    B->>B: Append to partition log
    B-->>Prod: Acknowledge
    Prod-->>App: Confirm send success
        

Sending one tiny network request per message would be wasteful. Kafka producers are smart enough to bundle messages together first.

Rather than firing off a separate network call for every single message, a producer briefly holds newly created messages in a small local batch, grouped by which partition they are headed to. Once the batch reaches a certain size, or a short amount of time passes, whichever happens first, the entire batch is sent together in one network request.

Once a message is sent, Kafka lets the producer decide exactly how much proof it wants before considering the send truly successful.

This setting is often called acks, short for acknowledgements, and it is one of the most important decisions a producer makes. It controls a direct trade-off between speed and safety: waiting for less confirmation is faster but riskier, while waiting for more confirmation is safer but slightly slower.

Networks glitch, brokers restart, and messages do not always reach their destination on the first try. Producers are built to handle this gracefully.

If a producer does not receive the expected acknowledgement in time, it typically assumes the send may have failed and automatically retries, sending the same batch again. This built-in resilience is one of Kafka’s strengths, but it introduces a subtle new problem: what if the original message actually did succeed, and only the acknowledgement was lost on its way back? The retry could then create a duplicate.

A producer can be told to wait for confirmation before continuing, or to keep working and check on the result later.

In synchronous sending, the application pauses and waits until it receives confirmation that the message was safely written before moving on to its next task. This is simple to reason about but can slow the application down, since every send briefly blocks other work.

In asynchronous sending, the application fires the message off and immediately continues with other work, without waiting. A callback, a small piece of code that runs later, is triggered once the result, success or failure, eventually becomes known. This approach is far more efficient for high-throughput systems.

Producers play a direct role in whether messages end up in the correct order once they are stored.

Because all messages sharing the same key always land in the same partition, and a single partition always writes messages in the exact order it receives them, choosing a consistent key is the single most important decision a producer makes for ordering. Messages sent without a key, spread across multiple partitions, carry no such guarantee relative to each other.

Retries introduce a subtlety here too: if retries are not handled carefully, an older message that needed to be resent could theoretically be written after a newer message that succeeded on its first attempt. Kafka’s idempotent producer mode specifically protects against this, preserving the original sending order even through retries.

Producing messages carefully and correctly is the foundation every downstream Kafka use case is built on.

Producing messages carefully takes some deliberate configuration, but each choice trades off against another.

A producer is often the very first point where sensitive data enters a Kafka system, making it a natural place to enforce security.

A producer that looks fine from the outside can still be quietly struggling. A few key numbers reveal the truth.

A few consistent habits separate producers that behave predictably from ones that cause quiet, hard-to-trace problems later.

Helpful patterns

• Consistent keying — always use the same key for related events, so their order is preserved.
• Idempotent production — enable it whenever accidental duplicates would cause real harm.
• Asynchronous sending with proper callbacks — for high-throughput systems that cannot afford to block.
• Schema agreement — settle on a message format and version it deliberately, rather than changing it silently.

A short checklist for producing messages with confidence.

No, a key is optional. Without one, Kafka spreads messages evenly across all partitions using a rotating pattern instead of routing by key.

Grouping messages together into a batch dramatically reduces the number of network requests needed, which is one of the main reasons Kafka can handle such high volumes of data efficiently.

Acks = all offers the strongest protection, since it waits for every in-sync replica to confirm the message before considering the send successful, at the cost of slightly higher latency.

Yes, this can happen if an acknowledgement is lost and the producer retries a message that actually already succeeded. Enabling idempotent production specifically prevents this from creating a true duplicate.

Not always. It offers much higher throughput, which suits high-volume systems, but synchronous sending can be simpler to reason about for lower-volume or simpler applications.

The producer detects the failure and automatically redirects the send to the newly elected leader, usually with only a brief delay, and without requiring any manual intervention.

It is strongly recommended, and usually enforced through an agreed schema, since consumers rely on a predictable, consistent shape to correctly deserialize and process every message they receive.

We began at a post office counter and ended inside the careful, multi-step journey a message takes through a Kafka producer. Producing a message is never truly as simple as “send and forget”; it involves choosing a partition through a key, serializing data into a transportable form, batching efficiently for performance, deciding how much confirmation to demand, and gracefully handling the retries and duplicates that inevitably happen in any real network. Understanding this full journey is what separates a system that occasionally loses or duplicates data from one that behaves reliably, even under real-world failure conditions.