To appreciate the solution, first watch the problem happen in slow motion.

Why a producer retries at all

A Kafka producer sends a batch of messages to the broker that leads a partition. The broker writes the batch to its log, waits for the replicas it needs, and then sends back an acknowledgement. Networks are unreliable, and brokers restart, so any step of that conversation can fail. A producer that gives up at the first failure would lose data, so by design it retries: it sends the very same batch again.

How a retry becomes a duplicate

Retries are safe when the first attempt truly failed. They are dangerous when the first attempt actually succeeded but the acknowledgement got lost on the way back. From the producer’s point of view, a lost request and a lost reply look identical: it simply hears nothing. If it retries in the second case, the broker stores the same messages twice.

sequenceDiagram
    participant P as Producer
    participant B as Broker (partition leader)
    P->>B: Send batch with 3 messages
    B->>B: Write batch to log (offsets 40 to 42)
    B--xP: Acknowledgement lost on the network
    Note over P: No reply before timeout, assume failure
    P->>B: Retry the same 3 messages
    B->>B: Write again (offsets 43 to 45)
    B-->>P: Acknowledged
    Note over B: The log now holds every message twice
        

Why duplicates hurt

The two old options

Before idempotence, a team had to pick one of two unpleasant trade-offs. Turn retries off, and any failure could lose data. Keep retries on, and some duplicates would appear. Pushing the burden of deduplication onto every consumer was possible, but it meant every downstream team had to build and maintain its own logic. The idempotent producer moves that responsibility to the place where it can be solved cheaply and once: the broker.

Three phrases appear in every messaging discussion. Here is what each one really promises.

Where idempotence sits

The idempotent producer gives you exactly once within a single producer session, for a single partition. That is a smaller promise than “exactly once everywhere”, but it is the essential building block on which larger guarantees, like transactions, are built. Understanding its exact edges is one of the goals of this tutorial, and later chapters return to them in detail.

Each term below is defined once here and then used consistently in the rest of the tutorial.

The whole mechanism rests on identity, numbering, and memory.

Idea one: identity

Before sending anything, the producer asks the cluster for a producer identity. From then on, every batch it sends says “I am producer number 4711, in epoch 0”. This tells the broker whose numbering to look at.

Idea two: numbering

For each partition it writes to, the producer keeps a counter. The first record sent to partition 3 is number 0, the second is number 1, and so on. Every batch states the number of its first record and how many records it holds, so the numbers of a batch are fully known.

Idea three: broker memory

For every producer and partition, the broker remembers the number of the last record it stored, and the details of a few recent batches. When a new batch arrives, the broker compares its numbers with what it remembers, and decides whether it is the expected next batch, a repeat of one already stored, or something unexpected.

flowchart TD
    A["Producer starts"] --> B["Ask cluster for a producer identity"]
    B --> C["Receive PID and epoch"]
    C --> D["Number records per partition from zero"]
    D --> E["Send batch stamped with PID, epoch, base sequence"]
    E --> F{"Broker compares with its memory"}
    F -->|"Next expected"| G["Store and remember the new numbers"]
    F -->|"Already seen"| H["Do not store, reply success with original offset"]
    F -->|"Gap or too old"| I["Reject with an error"]
        

What the user has to do

In modern Kafka versions, almost nothing. The setting that turns on idempotence is enabled by default in current client releases, as long as the other related settings are compatible. Yet knowing what happens under the hood is essential, because the guarantee has boundaries, and some common configurations quietly switch the feature off or weaken it.

Several parts of Kafka cooperate to make idempotence work. Knowing who does what makes debugging much easier.

On the producer side

On the broker side

flowchart LR
    subgraph Client["Producer process"]
        ACC["Record accumulator"]
        SND["Sender thread"]
        TM["Transaction manager: PID, epoch, sequences"]
    end
    subgraph Cluster["Kafka cluster"]
        CTRL["Controller: allocates PID blocks"]
        BR["Broker: identity requests"]
        LEAD["Partition leader: validate, append"]
        FOL["Follower replicas: rebuild producer state"]
    end
    TM --> SND
    ACC --> SND
    SND -->|"identity request"| BR
    BR -->|"needs new block"| CTRL
    SND -->|"batches with PID, epoch, sequence"| LEAD
    LEAD -->|"replicate batches"| FOL
        

The PID is the anchor of the entire design, so it deserves a careful look.

What a PID is

A PID is a 64-bit whole number. That is huge, over nine quintillion possible values, so the cluster will never run out in practice. It is chosen by the cluster, not by the application, which means no two live producers can accidentally pick the same identity.

The initialization handshake

When an idempotent producer starts sending, its first step is to send an initialize producer ID request to a broker. For a plain idempotent producer, any broker can answer. The broker responds with a fresh PID and an epoch, which begins at zero. The producer stores both in memory and stamps every later batch with them.

sequenceDiagram
    participant P as Producer
    participant B as Any broker
    participant C as Cluster controller
    P->>B: Initialize producer ID
    alt Broker has unused IDs in its block
        B->>B: Take next ID from local block
    else Block exhausted
        B->>C: Request a new block of IDs
        C-->>B: Block of unused IDs
        B->>B: Take next ID from new block
    end
    B-->>P: PID and epoch zero
    Note over P: Every future batch carries this PID and epoch
        

Why IDs come in blocks

If every identity request had to reach the central controller, the controller would become a bottleneck, and a controller outage would stop new producers from starting. Instead, each broker receives a block of consecutive IDs at a time. It can then hand them out locally, without any coordination, and only goes back to the controller when its block runs out. The controller records which blocks it has given away, so no ID is ever handed out twice, even after a crash or failover.

A PID is not forever

A plain idempotent producer holds its PID only in memory. If the producer process restarts, it asks for a brand new PID. The broker has no way to link the new identity to the old one. This detail is the source of the most important limitation, explained later: idempotence protects a producer session, not the logical application across restarts.

A PID says who you are. The epoch says which version of you is currently valid.

What the epoch is for

The epoch is a small counter, a 16-bit number, stored right beside the PID. It exists so the cluster can retire an older holder of an identity without inventing a new PID. Whenever a newer holder appears, it receives a higher epoch, and every broker will refuse any batch stamped with a lower epoch for that PID.

Where the epoch changes

flowchart TD
    A["Instance A holds PID 88, epoch 3"] --> B["Instance A freezes during a long pause"]
    B --> C["Instance B starts with same transactional ID"]
    C --> D["Coordinator returns PID 88, epoch 4"]
    D --> E["Instance A wakes up and sends epoch 3"]
    E --> F{"Broker compares epochs"}
    F -->|"3 is lower than 4"| G["Reject with invalid epoch error"]
    G --> H["Instance A is fenced and must stop"]
        

The sequence number is the number written on the tracking slip. Everything else exists to support it.

One counter per partition

A producer that writes to five partitions keeps five independent counters, one for each partition. Each starts at zero and grows by one for every record sent to that partition. Numbering is per PID and per partition, because the broker that leads one partition knows nothing about the others.

Numbering records, stamping batches

Although the counter counts records, the message travels in batches. Each batch carries the sequence number of its first record, called the base sequence, together with the count of records. The broker can therefore compute the last sequence number in the batch, and it knows exactly which range of numbers the batch claims.

The golden rule the broker checks

The broker remembers the last sequence number it stored for this producer and partition. The next batch must begin exactly one higher. If the base sequence is exactly one more than the last stored, the batch is the expected next one. This simple rule catches every kind of problem.

Sequences wrap around

The sequence number is a 32-bit whole number, roughly two billion values before it wraps. The counter is designed so that, when it reaches its maximum, it wraps back to zero, and the broker’s comparison handles this wrap correctly. A producer that writes a very large number of records to one partition in one session simply cycles through its numbers.

Everything idempotence needs is carried in a few fixed fields at the front of each batch.

Kafka stores and transmits records in batches, and each batch begins with a header. In the modern format, three of the header fields are dedicated to producer identity. Because the header is written to disk exactly as sent, the identity information becomes part of the permanent log, and this is precisely what allows a broker to rebuild its memory after a restart.

Why the header is written to disk

Because the producer fields are stored with every batch, the log itself is a complete record of who wrote what and in what order. If a broker loses its in-memory state, it can walk through the log and reconstruct the memory it needs. Followers receive the exact same bytes, header included, so they can build the same memory as well.

flowchart LR
    A["Base offset"] --> B["Length"]
    B --> C["Leader epoch"]
    C --> D["Magic and checksum"]
    D --> E["Attributes"]
    E --> F["Offset delta and timestamps"]
    F --> G["Producer ID"]
    G --> H["Producer epoch"]
    H --> I["Base sequence"]
    I --> J["Record count"]
    J --> K["Records"]
        

Follow one message from the moment the application hands it to the producer until the acknowledgement returns.

sequenceDiagram
    participant App as Application
    participant Acc as Accumulator
    participant Snd as Sender
    participant L as Partition leader
    participant F as Follower
    App->>Acc: send record
    Acc->>Acc: add to batch for partition
    Snd->>Acc: drain ready batch
    Snd->>Snd: stamp PID, epoch, base sequence
    Snd->>L: produce request
    L->>L: validate sequence against memory
    L->>L: append batch and update memory
    F->>L: fetch new batch
    L-->>Snd: acknowledgement with offset
    Snd-->>App: callback success
        

Every batch arriving at a partition leader goes through a small set of questions, always in the same order.

The remembered state

For each producer that has written to a partition, the leader keeps a small entry. It holds the producer’s latest epoch, the last sequence number stored, and compact details of up to five of the most recent batches: their first and last sequence numbers and the offsets where they landed. These details let the broker answer a retry with the correct original offset, rather than a made-up one.

flowchart TD
    A["Batch arrives with PID, epoch, base sequence"] --> B{"Does broker know this PID for this partition?"}
    B -->|"No, and base sequence is zero"| C["Accept as first batch, create entry"]
    B -->|"No, and base sequence not zero"| D["Reject: unknown producer or out of order"]
    B -->|"Yes"| E{"Compare epoch"}
    E -->|"Lower than stored"| F["Reject: producer fenced"]
    E -->|"Higher than stored"| G["Reset entry, accept only if base sequence is zero"]
    E -->|"Equal"| H{"Compare base sequence with expected"}
    H -->|"Equals last plus one"| I["Accept, append, update entry"]
    H -->|"Matches one of last five batches"| J["Duplicate: skip append, return original offset"]
    H -->|"Otherwise"| K["Reject: out of order sequence"]
        

Outcomes at a glance

Why rejecting gaps matters

A gap means some earlier batch never reached the broker. If the broker accepted the later batch anyway, the log would silently miss messages and the order of events would be wrong. By rejecting, the broker forces the producer to resend from the point of the gap, which preserves both completeness and order.

Return to the problem from chapter one and watch the same failures play out with idempotence turned on.

Failure kind one: the request is lost

The batch never reaches the broker. After a timeout, the producer retries with the same numbers. The broker sees a batch with exactly the next expected sequence, and stores it. There is no duplicate, because the first attempt never happened.

Failure kind two: the reply is lost

The batch reaches the broker and is stored, but the acknowledgement vanishes. The producer retries with the same numbers. This time the broker finds that the batch matches one it remembers, and skips the append. It replies success, reporting the original offset. The producer believes its retry worked, which in a sense it did, and the log contains the messages exactly once.

sequenceDiagram
    participant P as Producer
    participant B as Broker
    P->>B: Batch PID 7, epoch 0, base sequence 20, 3 records
    B->>B: Accept, store at offsets 100 to 102, remember sequences 20 to 22
    B--xP: Acknowledgement lost
    Note over P: Timeout, retry with identical numbers
    P->>B: Same batch PID 7, epoch 0, base sequence 20, 3 records
    B->>B: Matches remembered batch, do not append
    B-->>P: Success with offsets 100 to 102
    Note over B: Log holds the three records once
        

Failure kind three: reordering during retries

Suppose two batches are in flight, batch A with sequences 20 to 22 and batch B with sequences 23 to 25. Batch A fails and is retried, but batch B arrives first. Without idempotence, batch B would be stored before batch A, reversing the order. With idempotence, the broker sees that B’s base sequence, 23, does not equal the expected 20. It rejects B with an out-of-order error. The producer resends A and then B, and the log ends up in the correct order.

Why does Kafka insist on a small number, five, when it comes to outstanding requests?

What “in flight” means

To use the network efficiently, a producer does not wait for one request to be acknowledged before sending the next. It keeps several requests to the same broker outstanding at once. A setting controls how many such unacknowledged requests are allowed per connection.

The link with the broker’s memory

The broker remembers the details of only the last five batches per producer and partition. That number is closely tied to the limit on in-flight requests. If a producer never has more than five batches outstanding, then any batch that might be retried is guaranteed to be among the five the broker still remembers. The broker can therefore always tell a retry from a genuinely new batch.

flowchart LR
    subgraph Window["Broker remembers last five batches"]
        B1["Batch 1"]
        B2["Batch 2"]
        B3["Batch 3"]
        B4["Batch 4"]
        B5["Batch 5"]
    end
    N["New batch arrives"] --> Q{"Matches one in the window?"}
    Q -->|"Yes"| D["Duplicate: acknowledge without storing"]
    Q -->|"No, and expected next"| S["Store and slide the window forward"]
    Q -->|"No, and not expected"| R["Reject"]
        

What happens if you go beyond the limit

With more than five batches in flight, a retry of the oldest one could arrive after the broker has already pushed it out of memory. The broker could then no longer recognize it as a repeat. Modern clients therefore refuse to enable idempotence when the setting for in-flight requests is above five, and older clients treat it as a configuration error.

The broker’s notebook must survive restarts and must not grow forever. Both needs shape the design.

Where the state lives

Each partition replica has a producer state manager. It holds a map from PID to that producer’s entry. The map lives in memory for speed, but the truth is always recoverable from the log, because every batch header contains the PID, epoch, and sequence numbers.

Snapshots to speed up recovery

If a broker had to scan the entire log after every restart to rebuild its notebook, startup would be painfully slow for partitions holding many gigabytes. To avoid this, the broker periodically writes a producer state snapshot file next to the partition’s log segments, usually when a log segment rolls over. On restart, the broker loads the latest snapshot and replays only the batches written after it.

Expiry keeps the notebook small

A busy cluster sees a large number of producers come and go. If the broker remembered every producer forever, the memory would grow without limit. Entries are therefore removed after a period of inactivity, controlled by an expiration setting whose default is measured in a day. Entries can also disappear when the log data they describe has been deleted by retention.

flowchart TD
    A["Producer writes to partition"] --> B["Entry created or refreshed with last write time"]
    B --> C{"No writes for longer than expiry period?"}
    C -->|"No"| B
    C -->|"Yes"| D["Entry removed from broker memory"]
    D --> E["Producer returns later with same PID"]
    E --> F{"Sequence is zero?"}
    F -->|"Yes"| G["Treated as a fresh producer"]
    F -->|"No"| H["Unknown producer error, client must recover"]
        

Memory cost

Each entry is small, only a handful of numbers and five compact batch summaries, but the total count is the number of producers multiplied by the number of partitions they touch. A cluster with a large number of short-lived producers writing to many partitions can accumulate a surprising amount of state, which is why controlling producer churn is a recurring operational theme.

The guarantee must survive the death of the very broker that holds the notebook. Here is how.

Followers keep their own notebooks

Follower replicas copy batches from the leader byte for byte, including the header that holds the PID, epoch, and sequence numbers. As each batch is appended to a follower’s log, the follower also updates its own producer state. So at any moment, every in-sync replica holds a nearly identical notebook, differing only by the last few batches still being copied.

What happens at failover

Suppose the leader dies just after storing a batch and just before acknowledging it. A follower that had already copied the batch becomes the new leader. The producer, having no acknowledgement, retries. The retry arrives at the new leader, which recognizes the numbers from its own notebook, and skips the append. There is no duplicate, even though the broker that first stored the batch is gone.

sequenceDiagram
    participant P as Producer
    participant L1 as Old leader
    participant L2 as Follower who becomes leader
    P->>L1: Batch with sequences 50 to 54
    L1->>L1: Append batch, update notebook
    L2->>L1: Fetch, copies batch and updates own notebook
    L1--xP: Leader crashes before acknowledging
    Note over L2: Elected new leader
    P->>L2: Retry same batch, sequences 50 to 54
    L2->>L2: Notebook shows this batch already stored
    L2-->>P: Success with original offsets
        

The acknowledgement setting matters

The story above assumes the follower had received the batch before the leader failed. Idempotence therefore requires the strongest acknowledgement setting, in which the leader answers success only after all in-sync replicas hold the batch. With weaker settings, the leader might acknowledge a batch that no follower has yet copied. If that leader then dies, the batch is lost, and the producer, which thinks it succeeded, will have sent sequence numbers that the new leader has never seen. This mismatch can surface as an out-of-order error or as silent data loss.

Unclean leader election

If the cluster allows a replica that was out of sync to become leader, it may lack batches that the previous leader had acknowledged. The producer’s next batch then has a base sequence that is higher than that leader expects, and the leader rejects it. Recent clients treat this as a recoverable problem in many cases, but it is a sign that data has been lost, and it deserves investigation.

Knowing the names of the errors, and what each one means, makes log messages far easier to read.

How the client reacts

flowchart TD
    A["Produce request fails"] --> B{"Error kind"}
    B -->|"Temporary network or leader change"| C["Retry with identical numbers"]
    B -->|"Out of order sequence"| D["Resend earlier batches in order, or bump epoch if state was lost"]
    B -->|"Unknown producer"| E["Request new epoch and restart numbering"]
    B -->|"Invalid epoch, fenced"| F["Stop the producer, a newer instance owns the identity"]
    B -->|"Delivery time limit exceeded"| G["Fail the batch to the application, bump epoch if needed"]
    C --> H["Success or eventually one of the other branches"]
        

A handful of settings decide whether idempotence is on and how it behaves.

What happens with conflicting settings

If you explicitly request idempotence together with an incompatible value, such as a weaker acknowledgement level, the client raises a configuration error at startup. If you leave idempotence unset and choose a conflicting value, recent clients quietly fall back to non-idempotent behavior rather than fail. This silent fallback is a classic trap, because an application may believe it has protection when it does not. Always verify the effective configuration in the startup logs of your producers.

Older client versions

In older releases idempotence was off by default and had to be enabled by hand. Applications upgraded to newer clients may therefore change their behavior after the upgrade, gaining protection they did not previously have. Read release notes when moving between major versions, and re-test producer behavior under failure.

A guarantee is only useful when you know its boundaries. These are the most important ones.

Transactions build on the same three ideas: identity, epoch, and sequence numbers.

What transactions add

A transactional producer receives a name, the transactional ID, chosen by the application. A dedicated broker role, the transaction coordinator, maps that name to a PID and keeps the mapping in an internal log. When the application restarts with the same name, it gets the same PID and a higher epoch. This gives a stable identity across restarts, and it fences older instances.

flowchart TD
    A["Idempotence: PID and sequence numbers"] --> B["Add a named identity: transactional ID"]
    B --> C["Stable PID, epoch increases on each restart"]
    C --> D["Zombie instances are fenced"]
    C --> E["Group writes into transactions"]
    E --> F["Commit or abort markers written to each partition"]
    F --> G["Consumers can read only committed data"]
        

Idempotence is always inside

A transactional producer cannot work without idempotence. Every write inside a transaction still carries a PID, epoch, and sequence number, and the broker still validates them in exactly the way described earlier. Transactions simply add markers and a coordinator on top. Understanding the idempotent producer thoroughly is therefore the best preparation for understanding transactions.

Abstract rules become easy once you watch the numbers move through a small, concrete story.

The setting

A payment service uses one idempotent producer. The cluster hands it PID 5001 with epoch 0. The service writes to a topic with three partitions, so the producer keeps three independent counters, all starting at zero. The leader of partition 1 is broker A, and broker B holds a follower replica of partition 1.

Round one: a normal batch

The producer collects four records for partition 1. It stamps the batch with PID 5001, epoch 0, base sequence 0, and a count of four. Broker A sees an unknown producer with base sequence zero, which is the legal way to begin. It appends the batch at log offsets 900 to 903 and creates a notebook entry that says: producer 5001, epoch 0, last sequence 3, last offset 903. Broker B copies the batch and builds the same entry.

What the table teaches

Step three shows deduplication: the broker recognized the batch, did not add it again, and returned the offsets of the first copy. Step four shows gap detection: a later batch arrived early and was refused. Step five shows the missing batch arriving, and step six shows the earlier rejected batch succeeding once its predecessor landed. The final log holds every record exactly once and in the intended order, despite one lost reply and one overtaking batch.

sequenceDiagram
    participant P as Producer PID 5001
    participant A as Broker A (leader)
    P->>A: Base 0, 4 records
    A-->>P: Stored at 900 to 903
    P->>A: Base 4, 2 records
    A->>A: Stored at 904 and 905
    A--xP: Reply lost
    P->>A: Base 9, 3 records (sent while waiting)
    A-->>P: Rejected, expected base 6
    P->>A: Retry base 4, 2 records
    A-->>P: Duplicate, success with 904 and 905
    P->>A: Base 6, 3 records
    A-->>P: Stored at 906 to 908
    P->>A: Retry base 9, 3 records
    A-->>P: Stored at 909 to 911
        

Now add a failover

Imagine broker A crashes right after step five, before acknowledging it. Broker B, which had copied that batch, becomes leader. Its notebook says the last sequence is 8. The producer retries the batch with base 6, and broker B recognizes it, returning success without a second copy. Next the producer sends base 9, which is exactly the expected next value, and the story continues as if nothing happened.

Now add a producer restart

Suppose the payment service restarts. The new producer asks for identity and receives PID 5002. Its counters start at zero. If it now resends a payment event that the old producer already wrote, the broker sees producer 5002 with base sequence zero, which is perfectly valid, and stores it. Two copies exist. This is the boundary described earlier, made visible in numbers: the notebook has entries for 5001 and 5002, and nothing links them.

Inside the producer library, a few carefully arranged rules keep the sequence numbers honest.

Numbers are assigned late

Records sit in the accumulator for a short time before they are drained into a request. Sequence numbers are assigned when a batch is drained for its first send, not when the record is first handed to the producer. This ordering matters: the number a batch receives reflects the order in which batches actually leave for the partition, which is the order the broker will validate.

Keeping retries in order

When a batch must be retried, the producer places it back at the front of the queue for its partition. Batches with earlier sequence numbers are always sent before batches with later ones. If it did not do this, its own retries would keep colliding with the broker’s gap checks. The sender also tracks, per partition, the highest sequence number that has been acknowledged, so it knows which in-flight batches are safe to forget.

flowchart TD
    A["Batch drained from accumulator"] --> B["Assign base sequence from partition counter"]
    B --> C["Send to leader, remember as in flight"]
    C --> D{"Reply"}
    D -->|"Success"| E["Remove from in flight, advance acknowledged sequence"]
    D -->|"Retriable error"| F["Put batch back at front of partition queue, keep same numbers"]
    F --> C
    D -->|"Delivery time limit reached"| G["Fail batch, decide about epoch bump"]
    G --> H["Later batches for this partition are reset with new epoch and fresh numbers"]
        

When a batch cannot be confirmed

The trickiest moment is when a batch has used up its delivery time and the producer still does not know whether the broker stored it. The producer cannot simply forget the batch and move on, because later batches have already been numbered on the assumption that this one exists. If the broker never received it, those later batches would now show a gap. The producer therefore has two choices: keep trying until it learns the answer, or bump the epoch and renumber everything that follows from zero. Recent clients choose the second path, since it lets the producer continue, and they accept a small, understood risk in exchange.

Partition changes and numbering

The counters are per partition, so when a topic gains partitions, the producer simply begins a new counter for each new partition it writes to. Changing which partition a given key maps to does not disturb existing counters. The producer’s memory of counters grows with the number of distinct partitions it writes to, which is another reason to keep the count of partitions per producer reasonable.

Idempotent producers are one tool among several. Knowing the others helps you pick the right mix.

Choosing a combination

Where to put the effort

The closer to the source you remove a duplicate, the cheaper it is, because fewer systems ever see it. That is the argument for idempotent producers: they stop the most common duplicates before they enter the log. Yet duplicates from application logic, restarts, or consumer reprocessing sit outside the producer’s view, so a prudent design adds a second, downstream line of defense for the data that truly matters, such as money.

Knowing the history explains many quirks you will meet in older clusters and older documentation.

What changed for you as a user

Does this safety come at a price? The honest answer is yes, but the price is very small.

Costs on the producer

The producer keeps a counter per partition and a little bookkeeping for batches that are in flight. It must hold on to each unacknowledged batch until confirmed, so that retries can use the identical numbers. These items use a small amount of memory and negligible processing time.

Costs on the wire and on disk

Each batch header carries fourteen extra bytes for the PID, epoch, and base sequence. Since a batch typically holds many records, the added weight per record is tiny. These fields are present in the modern format whether or not idempotence is turned on, so switching the feature on adds nothing further to the stored data.

Costs on the broker

The real performance trap: producer churn

The overwhelming majority of idempotence-related production problems come from one habit: creating a new producer object for every request, or for every small unit of work. Each new producer asks for a new PID, then creates fresh state on every partition it writes to. If a service creates thousands of such producers per second, the cluster accumulates a heap of short-lived identities. Brokers spend time answering identity requests and storing state that will expire unused, and memory usage climbs.

flowchart LR
    subgraph Bad["Producer per request"]
        R1["Request 1 creates producer"] --> P1["New PID"]
        R2["Request 2 creates producer"] --> P2["New PID"]
        R3["Request 3 creates producer"] --> P3["New PID"]
        P1 --> S["Thousands of short-lived state entries on brokers"]
        P2 --> S
        P3 --> S
    end
    subgraph Good["Shared long-lived producer"]
        R4["Requests 1 to N"] --> P4["One producer, one PID"]
        P4 --> T["Small stable state on brokers"]
    end
        

Throughput effects of the in-flight limit

Because idempotence caps in-flight requests at five, a very high-latency link between producer and broker can limit throughput. In such cases, larger batches and moderate delays to fill them are more effective than trying to push more requests in flight.

A back-of-envelope sizing exercise

Numbers make the memory cost of producer state easy to feel. Suppose a partition replica has entries for producers that wrote to it during the last expiry period. Each entry holds a producer id, an epoch, a last sequence, a last offset, timestamps, and up to five small batch summaries. Even with generous rounding, an entry is a few hundred bytes. Now multiply by the number of distinct producers and partitions.

The lesson from the table is not that idempotence is heavy. The first two rows show it is almost free. The third row shows how one bad habit multiplies a tiny cost by a giant factor. Bounding the count of producers is therefore a better lever than trying to shave bytes from each entry.

Identity systems attract attention from attackers and from accidents alike.

Can someone forge a PID?

Producer identities are assigned by the cluster, and the client cannot choose one. A malicious client could try to send batches stamped with a PID it observed elsewhere, but in a secured cluster every connection is authenticated, and every write is authorized against the topic. The PID adds no permission by itself. An attacker who already has write permission to a topic can write to it with or without a PID, so the identity mechanism does not widen access.

When idempotence works, nothing visible happens. That is why you need signals that show it working and warn when it is not.

On the producer

On the broker

Tools that ship with Kafka

flowchart TD
    A["Alert: rising out-of-order or unknown producer errors"] --> B{"Recent broker restarts or leader changes?"}
    B -->|"Yes"| C["Expected in small numbers, check they settle"]
    B -->|"No"| D{"Producer count growing fast?"}
    D -->|"Yes"| E["Find the client creating many short-lived producers"]
    D -->|"No"| F{"Producers idle for longer than expiry?"}
    F -->|"Yes"| G["Producers resume after expiry, allow client recovery or shorten idle gaps"]
    F -->|"No"| H["Check for unclean elections and lost data on the partition"]
        

Building a small lab to prove it works

Trust in a guarantee grows fastest when you have watched it survive abuse. A small test environment with three brokers is enough. The plan is simple: run a producer that sends a large, known number of unique records, each carrying a serial number in its value. While it runs, cause trouble: restart the leader of the partition, pause and resume a broker’s process, or add artificial network delay and packet loss between producer and broker. When the run ends, read the partition from the beginning and check two things: that every serial number from the run appears, and that none appears more than once.

Habits that repeatedly work well, and habits that repeatedly cause trouble.

Patterns worth copying

Anti-patterns to avoid

A checklist you can carry into design reviews and code reviews.

A short design review checklist

A tuning cheat sheet for common workloads

Different workloads stress idempotence in different ways. The guidance below is a starting point, not a law, and each row should be confirmed against measurements from your own system.

Reading the retry rate correctly

A nonzero retry rate is not a defect. It means the machinery is doing its job: transient trouble happened and was absorbed. What matters is the trend. A flat, low retry rate with zero final errors is healthy. A retry rate that climbs steadily points to a slowing broker, a saturated network, or an overloaded partition, and the idempotent producer is quietly hiding the symptoms from your application. Treat the retry metric as an early warning light, because it often rises hours before any user-visible failure appears.

Choosing delivery time limits with care

The delivery time limit is the total budget for one batch, including waiting in the buffer, every network attempt, and every retry pause. If it is too short, healthy batches will expire during a brief leader election, forcing epoch bumps and uncertainty for no reason. If it is too long, a genuinely stuck send hangs around for minutes, holding memory and blocking later batches for the same partition. A good starting point is a value comfortably longer than the slowest leader election you have observed, and shorter than the time after which your business would rather see a failure than wait.

The same mechanism shows up in very different businesses, each with its own reasons to care.

No. Idempotence removes duplicates caused by the producer’s own retries within one producer session and one partition. Exactly-once across restarts, several partitions, and consumers needs transactions and careful consumer design.

Yes. Without a small memory of recent sequence numbers per producer and partition, the broker would have no way of telling a retry from a new batch.

The cluster chooses it. The client asks for an identity, and receives one from a block of unused values held by a broker.

A plain idempotent producer gets a brand new PID and starts every partition’s counter at zero. The old numbers are irrelevant, because the broker treats the new PID as an unrelated producer.

It keeps broker memory small, and it matches the cap of five requests in flight. Any batch that could be retried is always still within the remembered window.

Only slightly, if at all. The extra work is a few integer comparisons and a small header, and the cap of five in-flight requests is rarely a limit in practice.

The producer epoch versions a producer identity and fences older holders of that identity. The leader epoch versions the leadership of a partition and fences older leaders. They serve similar purposes for different actors.

Not at the same time in a healthy cluster. For plain idempotent producers, each instance gets its own PID. For transactional producers, two instances with the same transactional ID share a PID, but only one, the one with the higher epoch, is allowed to write.

It can be rebuilt from the batch headers in the log, using the last snapshot as a shortcut. Followers hold their own copies too, so a leader change does not lose the state.

The broker discards state for producers that have been silent longer than the expiration period. When the producer wrote again, the broker no longer recognized its identity. Recent clients recover by asking for a new epoch.

Usually not. In current client versions it is enabled by default. You mainly need to avoid configuration settings that conflict with it, and confirm the effective settings at startup.

No. Deduplication looks at the batch header fields, not the compressed record contents, so it works the same with any compression choice.

So that every acknowledged batch is present on all in-sync replicas. Then a new leader after failover already knows the sequence numbers, and retries are handled correctly.

Yes, through the log dump tool, which prints batch headers from segment files. Regular consumers do not receive these fields as part of the records they read.

No. Such a job also needs its consumed positions tied to its produced output. Transactions do this, by committing the output and the input offsets together as one unit.

An older instance of a transactional producer that is still running after a newer instance has taken over, perhaps after a long pause. Its lower epoch means brokers refuse its writes, so it cannot cause harm.

Yes, and it is the simplest case. The producer keeps one counter for that partition, and the leader keeps one notebook entry per producer. Everything works exactly as described, just with fewer counters.

Existing counters continue unchanged, and the producer starts fresh counters for any new partitions it writes to. The guarantee stays intact, although the mapping of keys to partitions may change, which affects ordering by key rather than deduplication.

No. A batch is accepted or treated as a repeat as a whole. The broker compares the full range of sequence numbers claimed by the batch, so there is no state in which half of a batch is stored and half is recognized as a repeat.

Indirectly. The state itself is not copied as a separate object. Instead, each follower rebuilds the same state from the batches it copies, because the batch headers carry all the needed numbers. Snapshots are local to each replica.

A few are normal while leadership moves and producers retry. If they continue for a long time, check for lost data on the partition, unclean leader elections, or producers whose state expired while they were idle.

Not in the record objects they receive. The values live in the batch header and are used by brokers, and by tools that read log files directly. Consumers only see the effect: no repeated retries in the data.

It reduces memory, but makes it more likely that a producer returning after a quiet gap finds its entry gone. Recent clients recover, yet each recovery involves an epoch bump and some uncertainty about the last unconfirmed batch. Shorten it only if your producers write frequently and your memory truly requires it.

For anything important, yes. Idempotent producers close the retry gap, but duplicates can still arrive from application resends, producer restarts, or consumer reprocessing. A business key check downstream is a cheap and durable second line of defense.

The idempotent producer solves a specific, common problem: a retry that follows a lost acknowledgement must not leave a second copy in the log. It does so with three simple ingredients. A producer ID identifies who is writing, an epoch says which version of that identity is valid, and per-partition sequence numbers, carried in the header of every batch, say exactly which records a batch contains. The partition leader keeps a small notebook of recent numbers for each producer, rebuilds it from the log after restarts, and shares it with followers through ordinary replication, so failovers keep the protection. Understanding the precise edges of this guarantee lets you use it confidently, and add transactions or consumer-side keys where a larger promise is needed.