Idempotence In Depth in Kafka
Networks fail, messages get resent, and duplicates sneak in. Idempotence is the idea that makes a repeated action safe. This guide takes you from the basic meaning to the exact machinery inside Kafka producers and brokers, then shows where the protection ends and what you must build yourself.
Think about pressing the button for a lift. You press it once, and the lift is called. You press it again out of impatience, and the lift is still simply called. Nothing bad happens, and the second press changes nothing. Now think about a vending machine where every press of “buy” drops one more snack and takes one more coin. Pressing twice means paying twice. The lift button is idempotent. The vending machine button is not.
Messaging systems face this puzzle all day. A program sends a payment message, the network hiccups, and the program never hears back. Did the message arrive or not? Sending it again might create a second payment. Not sending it again might lose the payment. Both choices feel wrong. The way out is to make the act of sending safe to repeat, so that trying again can never do harm.
Apache Kafka provides exactly this safety inside its producer, the part of your program that writes messages into Kafka. Since 2017 it has been available, and in recent versions it is switched on by default. Yet many engineers use it without knowing how it works, what it covers and, most importantly, what it does not cover. This guide fills that gap step by step, using simple pictures and everyday comparisons before it dives into the details.
1What Does “Idempotent” Mean?
The word sounds heavy, but the idea is light. Once you see it in daily life, you will spot it everywhere.
An operation is idempotent when doing it many times has the same effect as doing it once. The first time changes something. Every repeat after that changes nothing further. The result after ten attempts is identical to the result after one.
Idempotent or not? A quick tour
| Action | Idempotent? | Why |
|---|---|---|
| Set the thermostat to 22 degrees | Yes | Setting it again leaves it at 22 |
| Turn the thermostat up by 1 degree | No | Each press adds another degree |
| Save a file with the same name and content | Yes | The file ends up the same |
| Add 100 rupees to an account | No | Each attempt adds another 100 |
| Set the account balance to 500 rupees | Yes | Repeating leaves it at 500 |
| Delete a photo by its unique name | Yes | After the first delete, it is already gone |
| Send an email | No | Each send delivers another copy |
A light switch marked “on” is idempotent: flipping it to “on” ten times leaves the light on. A dimmer knob that you turn one notch brighter each time is not, because ten turns give a very different result from one. Engineers try to design important systems more like the switch and less like the knob.
The mathematical hint
Mathematicians describe it this way: applying a function twice gives the same answer as applying it once. Taking the absolute value of a number is a classic case. The absolute value of minus five is five, and the absolute value of five is still five. You will not need any formulas in this guide, but the picture helps: repeating changes nothing after the first time.
Why it matters so much for messages
In distributed systems, messages travel over networks that lose, delay and repeat packets. The sender often cannot know whether a message arrived. The safe response is to send again, which means the receiver may see the same message twice. If the receiving side is idempotent, the second copy is harmless. If it is not, you get double charges, double emails and double stock deductions.
When you meet a new operation, ask: “What happens if this runs twice?” If the honest answer is “something bad”, you have found a place where idempotence needs to be designed in.
Where the word comes from
The term was coined by the mathematician Benjamin Peirce in the nineteenth century, from Latin roots meaning “the same power”. It described quantities that stay the same when multiplied by themselves. Software engineers later borrowed the word for operations that stay the same when repeated. The borrowing stuck because the idea is so useful: if repeating an action is harmless, you can be far less nervous about failures.
The idea in web design
Anyone who has worked with web services has met the idea already. The rules of the web describe some request types as idempotent: reading a page, replacing a resource with new content and deleting a resource. They describe another type, the one used to create new items, as not idempotent, because sending it twice may create two items. This is why online shops warn you not to press “pay” twice, and why good ones protect you even if you do.
The double-click problem
A customer clicks “place order” and the page seems to freeze. The customer clicks again. Without protection, two orders arrive. With an idempotency key created when the page loaded and sent with both clicks, the server sees the same key twice and processes only the first. The customer sees one order, and nobody needs to explain a refund.
2Why Duplicates Happen in Kafka
Before fixing a problem, we should see exactly how it arises. Duplicates in Kafka are rarely a bug. They are a natural side effect of trying to be reliable.
A quick look at how producing works
A producer is the part of an application that sends records to Kafka. It gathers records into small bundles called batches and sends each batch to the broker (a Kafka server) that leads the target partition (one ordered slice of a topic). The broker writes the batch to its log, copies it to follower brokers, and then replies with an acknowledgement, a short “got it” message.
The lost acknowledgement
Now imagine the reply goes missing. The broker did write the batch, but the acknowledgement was lost in the network, or the broker was slow, or the connection dropped. The producer waits, hits its timeout and concludes that the write may have failed. To be safe it sends the same batch again. Kafka now holds the same records twice.
sequenceDiagram
participant P as Producer
participant B as Broker
P->>B: Send batch with records A and B
B->>B: Write batch to log
B--xP: Acknowledgement lost in the network
Note over P: No reply, so assume failure
P->>B: Retry the same batch
B->>B: Write batch to log again
B->>P: Acknowledgement received
Note over B: Log now holds A, B, A, B
You post a cheque to a friend and never hear back. Worried, you post an identical cheque. Both were actually delivered, and your friend now holds two. The post office did nothing wrong, and neither did you. The problem is that neither of you could tell whether the first cheque made it.
Other ways duplicates appear
Timeouts and resets
Slow links or dropped connections make the producer unsure, so it retries records that were already stored.
Leader changes
If the leader broker changes while a batch is in flight, the producer may resend to the new leader without knowing whether the old one saved it.
Application restarts
A program that crashes and restarts may resend records it had already sent before dying.
Reprocessing
A consumer that crashes before saving its position re-reads and reprocesses messages, which can duplicate the effects downstream.
Duplicates are the price of safe retrying. The fewer retries you allow, the fewer duplicates you get, and the more messages you may lose. Idempotence lets you keep retries and still avoid the duplicates.
Why not simply avoid retries?
It is tempting to say: if retries cause duplicates, turn retries off. That fixes duplicates and creates a worse problem. Every network has moments of trouble, and a busy system sees them constantly. A cluster handling one million records per minute might see a few thousand fail on the first attempt because of a brief timeout or a leader change. With retries off, all of those records vanish without a trace. Most businesses would rather have a rare duplicate than a steady trickle of missing orders.
| Choice | What you gain | What you risk |
|---|---|---|
| No retries | No duplicates | Lost records during any hiccup |
| Retries without protection | No lost records | Duplicates and possible reordering |
| Retries with idempotence | No losses and no retry-caused duplicates | Only the edges covered in chapter 10 |
Consider a lifeguard who is told never to repeat a rescue attempt in case the swimmer is grabbed twice. That rule makes no sense: a failed rescue must be tried again. What the lifeguard needs is a way to see clearly whether the swimmer is already safe. Sequence numbers are the lifeguard’s clear view.
3The Delivery Guarantee Ladder
Messaging systems describe their promises using three standard phrases. Knowing them makes idempotence easy to place.
Never twice, maybe never
The producer sends once and never retries. If something fails, the message is lost. No duplicates, but no safety against loss.
Never lost, maybe twice
The producer retries until it is sure the message arrived. Nothing is lost, but duplicates can appear.
Never lost, never twice
Every message takes effect once, even through failures and retries. The hardest to achieve, and it needs care from end to end.
Think of ringing a doorbell. At most once means pressing it one time and leaving, even if nobody answered. At least once means pressing until someone answers, which may make the bell ring several times inside. Exactly once means the household hears exactly one ring no matter how many times you press, because the bell is clever enough to ignore repeat presses.
Where Kafka’s producer sits
Without idempotence, a Kafka producer with retries turned on provides at-least-once delivery to a partition. With idempotence turned on, the same producer provides exactly-once delivery to each partition for as long as it stays alive. That is a strong claim, and the rest of this guide explains precisely how far it goes.
| Setup | Loss possible? | Duplicates possible? | Order kept with retries? |
|---|---|---|---|
| No retries | Yes | No | Yes |
| Retries, no idempotence | No | Yes | Not always |
| Retries with idempotence | No | No, within one producer session and one partition | Yes |
| Transactions on top | No | No, across partitions and restarts | Yes |
Three separate places to think about
Delivery guarantees are not one single thing. There are three hand-offs in a Kafka pipeline, and each has its own rules.
Producer to broker
Idempotence protects this hop. It stops retries from duplicating records in a partition.
Broker to consumer
Consumers track their position by saving offsets. If a consumer fails before saving, it re-reads. This hop is at-least-once unless you add more.
Consumer to the outside world
When a consumer writes to a database or calls a service, Kafka cannot guarantee anything. Your own design must make the effect safe to repeat.
Choosing a guarantee for real workloads
Different jobs tolerate different mistakes. The table below shows how teams often match a workload to a guarantee.
| Workload | Cost of a lost record | Cost of a duplicate | Usual choice |
|---|---|---|---|
| Debug logs | Low | Low | At most once or at least once |
| Website click tracking | Low to medium | Low | At least once |
| Order and payment events | High | High | Idempotent producer plus idempotent consumers |
| Inventory adjustments | High | High | Transactions or key-based sets |
| Financial ledger entries | Very high | Very high | Transactions plus business identifiers plus audits |
Ask the business two questions for each event type: “What if we lose one?” and “What if we see one twice?” The honest answers tell you which guarantee to build.
4The Idempotent Producer: The Big Idea
The idea behind Kafka’s solution is beautifully simple: give every record a serial number, and let the broker ignore numbers it has already seen.
Two labels on every batch
When idempotence is on, the producer attaches two extra labels to everything it sends:
- a producer ID, a unique number that identifies this producer, handed out by the cluster when the producer starts;
- a sequence number, a counter that goes up by one for every record sent to a particular partition, starting from zero.
Because the counter is kept separately for each partition, the pair “producer ID plus partition” has its own private numbered line of records.
A busy cafe hands out numbered order tickets. The kitchen keeps track of the highest ticket number it has cooked. If a waiter sneaks in and hands over ticket 57 again, the kitchen sees that 57 was already cooked and quietly ignores it. The customer gets one meal, however many times the ticket was passed through the hatch.
The broker’s part
The broker keeps a small record, for each producer and each partition, of the latest sequence numbers it has accepted. When a new batch arrives, the broker compares the batch’s number with what it expects next.
sequenceDiagram
participant P as Producer with ID 7
participant B as Broker
P->>B: Batch with sequence 0 to 1
B->>B: Expected 0, accept and write
B--xP: Acknowledgement lost
P->>B: Retry batch with sequence 0 to 1
B->>B: Already seen 0 to 1, do not write again
B->>P: Acknowledgement sent
Note over B: Log holds each record exactly once
What the producer sees
Nothing unusual happens from the program’s point of view. The application still calls the send method, gets back a confirmation and moves on. The producer library assigns the numbers and handles retries behind the scenes. The application never sees the sequence numbers and does not need to manage them.
A little history
This feature was designed in a Kafka improvement proposal known as KIP-98 and released with Kafka 0.11 in 2017, alongside transactions. For years it was optional. In Kafka 3.0, released in 2021, the producer started turning on idempotence by default, together with the strongest acknowledgement setting. Many teams therefore already use it without having chosen to.
Older client versions and older tutorials still describe idempotence as an optional extra. If your clients are recent, check the effective settings before assuming either way, because a setting written explicitly in your configuration overrides the default.
Sequence numbers for batches and records
To be precise, sequence numbers count records, but they travel in batches. A batch header stores the sequence number of its first record, called the base sequence, and the batch also states how many records it holds. The broker can therefore work out the full range covered. If the base sequence is 200 and the batch holds 50 records, the batch covers numbers 200 to 249, and the next batch should start at 250.
| Batch | Base sequence | Records | Covers | Next expected |
|---|---|---|---|---|
| First | 0 | 50 | 0 to 49 | 50 |
| Second | 50 | 50 | 50 to 99 | 100 |
| Retry of second | 50 | 50 | 50 to 99 | Ignored as already seen |
| Third | 100 | 20 | 100 to 119 | 120 |
Page numbers in a book work in the same way. A chapter that starts on page 50 and runs for 50 pages ends on page 99, and the next chapter starts on page 100. If a copy of the second chapter turns up again in the pile, the page numbers make it obvious it is a repeat.
Why counting records matters
Counting individual records rather than batches means the producer can rebuild a batch differently on a retry, for example by combining the original records with newer ones or by splitting a large batch, while still keeping the numbers consistent. The rule for the producer is to keep the numbering of records the same as when they were first assigned.
The sequence counter is a whole number that wraps around after it reaches a very large value. In practice this is never seen, because it would take billions of records to a single partition from a single producer session.
5Inside the Broker: How Duplicates Are Detected
The detection rule is short. Understanding where the broker keeps its memory explains why the feature survives failures.
The three possible cases
For each producer and partition, the broker remembers the latest accepted sequence number. When a new batch arrives, exactly one of three things is true.
Accept
The batch starts exactly where the last one ended. The broker writes it and moves its counter forward.
Duplicate
The batch’s numbers are at or below what the broker already accepted. It is not written again, yet the broker replies with success so the producer can stop retrying.
Out of order
The batch starts after a hole, meaning some earlier numbers never arrived. The broker rejects it with an out-of-order-sequence error.
flowchart TB
A["Batch arrives with producer ID and sequence"] --> B{"Compare with last accepted sequence"}
B -- "Exactly next" --> C["Write to log and update counter"]
B -- "Already accepted" --> D["Skip writing and send success"]
B -- "Gap in the numbers" --> E["Reject with out-of-order error"]
C --> F["Send acknowledgement"]
Where does the memory live?
The clever part is that the numbers are stored inside the log itself. Since the record batch format introduced in Kafka 0.11, every batch on disk carries a header holding the producer ID, a version counter for that producer called the epoch, and the base sequence number. Followers copy batches exactly as they are, so they receive these headers too.
This has a big consequence. If the leader fails and a follower takes over, the new leader can work out the producer state from the log it already has. There is no separate database to keep in step, and no special hand-over. Brokers also write periodic snapshots of producer state to disk so that a restart does not need to read the whole log again.
Imagine a librarian who stamps each borrowed book’s card with the borrower’s number and a running count. Because the stamp is on the card itself, any other librarian can pick up the shift and know at once which requests were already recorded. Nobody needs a separate notebook, and nothing is lost when the librarians swap.
How much does the broker remember?
The broker does not store every sequence number ever seen. For each producer and partition it keeps details of only the last five batches. That is enough to recognize a retry of any of the most recent five in-flight requests, which matches the limit on how many unacknowledged requests the producer may have at once. Older information is not needed, because the producer will not retry something older than the window it is allowed to keep open.
Advantages of this design
- State follows the log, so leader changes are safe.
- Very little memory per producer and partition.
- No extra network calls at write time.
- The application code stays unchanged.
Limits of this design
- Memory grows with the number of producers times partitions.
- Producer state expires after a period of inactivity.
- Protection covers one producer identity, not the whole application.
- It cannot detect duplicates that use different producer IDs.
A peek at the batch header
Every batch stored in a Kafka log begins with a small header. Reading it shows exactly what the broker knows about a batch, and where idempotence fits among the other details.
| Header part | What it holds | Used for |
|---|---|---|
| Base offset | Position of the first record in the partition | Locating data for consumers |
| Leader epoch | Which leader term wrote the batch | Detecting stale leaders after failover |
| Attributes | Compression type, timestamp type and flags for transactional or control batches | Reading and filtering correctly |
| First and maximum timestamp | Times of the earliest and latest records | Retention and time-based lookups |
| Producer ID | The identity of the writing producer | Duplicate detection |
| Producer epoch | The version of that identity | Fencing old instances |
| Base sequence | Sequence number of the first record | Duplicate and gap detection |
| Record count | Number of records inside | Working out the sequence range |
A parcel label carries a destination, a weight, a handling note and a tracking number. The tracking number does nothing for the parcel’s contents, but it lets the depot know at a glance whether it has handled this exact parcel before. The producer ID and base sequence are the tracking number of a batch.
Note that these headers add up to a few dozen bytes per batch. When a batch holds hundreds of records, the cost per record is close to nothing, which is one more reason batching is worth tuning.
6Producer IDs, Epochs and Sessions
The protection is tied to an identity. Understanding how that identity is created, and when it changes, explains almost every surprise you will meet in production.
How a producer gets its identity
When an idempotent producer starts, it sends a short request to the cluster asking for an identity. The cluster answers with a unique producer ID and an epoch, which starts at zero. From then on, every batch the producer sends carries these two numbers plus its sequence numbers.
Start
The producer asks the cluster for an identity and receives a producer ID and an epoch.
Send
Each batch to each partition carries the ID, the epoch and the next sequence numbers for that partition.
Retry safely
If a batch must be retried, it is sent again with exactly the same numbers, so the broker can recognize it.
Stop
When the producer closes or crashes, its identity is simply abandoned. The next producer gets a fresh one.
The session idea
Because a new producer always receives a brand-new identity, the protection lasts for a session: the life of one producer object. Inside a session, the broker can spot repeated batches. Across sessions, it has no way to know that the new producer is a continuation of the old one.
flowchart LR
S1["Session 1: producer ID 7"] --> C["Application crashes"]
C --> S2["Session 2: producer ID 12"]
S1 --> L["Partition log with records from ID 7"]
S2 --> L
L --> Q["Broker cannot link ID 7 and ID 12"]
Visitors to an office building receive a numbered badge for the day. The guard can tell if the same badge is scanned twice in a row. If the visitor leaves, comes back after lunch and receives a new badge, the guard has no idea it is the same person. The badge, not the person, is the identity.
What the epoch is for
The epoch is a version number attached to an identity. Its most important role is in transactional producers, where a fixed name chosen by you lets the cluster recognize a restarted program and raise the epoch to shut out the old copy, an idea covered in chapter 11. For plain idempotent producers, newer Kafka versions can also raise the epoch to recover from certain error states, such as when the broker has forgotten an idle producer, and then continue the session cleanly.
Idle producers are forgotten
Brokers do not keep producer state forever. If a producer writes nothing to a partition for a long time, its details are removed to save memory. The default waiting period is one day, controlled by a broker setting named producer.id.expiration.ms. Producers that send records regularly never notice. A producer that sleeps for longer than the limit and then wakes up may receive an “unknown producer” error, and modern clients handle this by starting a fresh sequence.
Keep long-lived producer objects for the life of your application and share them across threads. Creating a new producer for every message wastes identities and defeats the idea of a session.
Sharing one producer across threads
The Kafka producer is designed to be shared. Several threads in one application can call send on the same producer object, and the library gathers their records into batches for each partition. Sharing has two benefits. It keeps the number of identities and connections small, and it gives better batches, because more records arrive at the same buffer in the same moment.
Because the sequence counters are kept per partition inside the producer, the library assigns numbers in the order records are added to a partition’s buffer. Even if ten threads add records at once, each partition still ends up with one clean numbered line.
One producer per application
Create it at startup, share it, and close it at shutdown so buffered records are flushed.
Flush on shutdown
Closing the producer sends whatever is still in the buffer and waits for acknowledgements.
Producer per request
Wasteful and defeats session-level protection, as described in chapter 17.
Kill without closing
Terminating the process abruptly loses buffered records and abandons the session, so a restart begins with a new identity.
A shared post box in an office is more efficient than each employee walking to the post office. One collector empties the box, bundles letters by destination and hands them over. Ten people can drop letters in at the same time without any of them getting mixed up.
Many instances of the same service
Modern services often run as dozens of identical copies behind a load balancer. Each copy has its own producer and therefore its own identity. This is fine and normal. It does mean, once more, that Kafka’s duplicate detection works separately for each copy, and that a request retried by a load balancer onto a different copy will produce two records, one from each. Handle that with a business identifier.
7The Settings That Matter
A handful of producer settings control whether idempotence works. Some are required, and some are just sensible companions.
| Setting | What it does | Rule for idempotence |
|---|---|---|
| enable.idempotence | Turns the feature on or off | On by default in recent client versions |
| acks | How many brokers must confirm a write before it is acknowledged | Must be “all” |
| retries | How many times a failed send is retried | Must be greater than zero; the default is very large |
| max.in.flight.requests.per.connection | How many unconfirmed requests can be open at once | Must be five or fewer; the default is five |
| delivery.timeout.ms | The total time allowed for a record to be sent, including retries | Default is two minutes; the producer gives up after this |
| request.timeout.ms | How long to wait for one reply | Must fit inside the delivery timeout |
| transactional.id | A fixed name that enables transactions | Optional; implies idempotence |
Sending a parcel with tracking works only if a few rules hold. You must ask for signed delivery (acks all), be willing to resend if it goes missing (retries), and not have too many parcels on the road at once to keep track of (in-flight limit). Take away any one rule, and the tracking system can no longer promise a clean record.
Why each rule exists
Confirmed by the group
The leader waits until the required copies exist on other brokers before replying. Without this, a leader failure could lose a record the producer believed was safe, breaking the sequence story.
Retrying is the point
Idempotence exists to make retries safe. If retries were disabled, there would be nothing to protect against.
Matches the broker’s memory
The broker remembers the last five batches per producer and partition. The producer must not have more than five unconfirmed at once, or the broker could not recognize a retry.
A deadline, not a promise
After this time the producer stops trying and reports failure, even if the record may have been stored. Chapter 9 explains the trap hidden here.
Conflicting settings
If you explicitly turn idempotence on and also set a conflicting value, such as a single acknowledgement or zero retries, the client refuses to start and reports a configuration error. In some versions, if you never mentioned idempotence and set a conflicting value, the client quietly disables idempotence instead. This silent case is dangerous, because your application seems healthy while the protection is gone. Read the startup log lines that list the effective configuration.
Durability partners on the broker side
Two broker or topic settings work alongside the producer. The replication factor is how many copies of each partition exist, commonly three. The min.insync.replicas setting says how many of those copies must be up to date before a write with acks set to all is accepted, commonly two. Together with idempotence, a common recipe is three copies, at least two in sync, acks set to all and idempotence enabled.
Idempotence protects against duplicates, not against a cluster with too few healthy copies. With a minimum in-sync count of one, a leader failure at the wrong moment can still lose acknowledged data.
Three sensible starting profiles
Teams often begin from one of three profiles and adjust after measuring. All three keep the safety settings intact and differ only in how they trade time for volume.
| Profile | Batching | Waiting before sending | Compression | Best for |
|---|---|---|---|---|
| Balanced | Default batch size | Small | Optional | Most business services |
| High throughput | Larger batches | Tens of milliseconds | Recommended | Logs, clicks, telemetry |
| Low latency | Small batches | Near zero | Usually off | Interactive or alerting events |
A delivery van can leave the depot the instant one parcel is loaded, or wait until it is full. Leaving at once is fast for that parcel but wasteful overall. Waiting fills the van and lowers the cost per parcel. The safety rules, such as signed delivery and tracking numbers, apply either way.
8Ordering: The Second Gift of Idempotence
Idempotence does more than remove duplicates. It also keeps records in the right order when retries happen, something older setups could not do.
The old problem
To go fast, a producer sends several batches without waiting for each answer. Suppose batch one and batch two are both in flight. Batch one fails because of a brief network error, but batch two succeeds. The producer now retries batch one. It lands after batch two, and the partition holds the records in the wrong order: two before one.
sequenceDiagram
participant P as Producer without idempotence
participant B as Broker
P->>B: Batch 1
P->>B: Batch 2
B--xP: Batch 1 fails in the network
B->>P: Batch 2 succeeds
P->>B: Retry batch 1
Note over B: Log order is now 2 then 1
Two friends each write a page of a story and post them separately. Page one gets lost and is re-posted, arriving after page two. The reader gets the pages in the wrong order and the plot makes no sense. Numbering the pages would let the reader put them back in sequence.
The old workaround
Before idempotence, the safe way to keep order was to allow only one request in flight at a time. This worked but was slow, because the producer had to wait for every answer before sending the next batch, wasting the network.
How sequence numbers fix it
With idempotence, every batch has a sequence number, and the broker refuses to accept a batch that would leave a gap. The producer library also remembers the order of batches it has in flight and, when it must retry, resends the older ones first. The result is that the partition always contains records in the order the producer sent them, even when retries occur, and even with up to five requests in flight.
What you gain
- Correct order within a partition, even during retries.
- High throughput, because several requests can stay in flight.
- No need for the old single-request workaround.
What still holds
- Order is guaranteed only within one partition, never across partitions.
- Order depends on records sharing a key, so they go to the same partition.
- Records from different producers can interleave in any way.
If the order of events for one customer or account matters, use that identifier as the record key. Idempotence then keeps each customer’s events in order and free of duplicates within the partition.
9Failure Scenarios Walk-Through
The best way to trust a feature is to watch it face trouble. Here are the common failures and what idempotence does in each.
| What goes wrong | What happens | Result |
|---|---|---|
| Acknowledgement lost in the network | Producer retries with the same sequence numbers; broker sees them as already accepted | One copy stored |
| Leader broker crashes after writing | A follower becomes leader with the same log and producer state; the retry is recognized | One copy stored |
| Leader crashes before copying to followers | The write was never acknowledged with acks set to all, so the producer retries to the new leader | One copy stored |
| Data lost through an unclean leader election | The new leader lacks earlier batches, so a later batch shows a gap | Out-of-order-sequence error surfaces the problem |
| Producer application crashes and restarts | New producer ID; the broker cannot link it to the old one | Duplicates possible if records are resent |
| Delivery timeout expires | Producer reports failure; the batch may or may not have been stored | Uncertain; resending in application code can duplicate |
| Producer idle beyond the state expiry | Broker forgot the producer; newer clients start a fresh sequence | Continues safely in recent versions |
The dangerous gap: the delivery timeout
The last case deserves attention. When the delivery timeout passes, the producer stops retrying and tells your code the send failed. But “failed” here means “I could not confirm success”. The record may in fact be sitting in the log. If your application code catches the error and calls send again with the same data, it starts a new sequence for that record, and the broker will store it a second time.
What people do
They wrap the send in their own retry loop, resending the same data after every reported failure, believing idempotence will make it safe.
Why it hurts
Idempotence only recognizes retries made by the producer library itself. A fresh send call is a brand-new record with a brand-new sequence number, so the broker accepts it even if the earlier attempt succeeded.
What to do instead
Let the library do the retrying and set the delivery timeout to fit your needs. If a final failure is reported, design the rest of the system so a rare duplicate is harmless, for example by including a unique identifier in the record for the consumer to check.
A courier tries three times to deliver a parcel, then reports “delivery failed” and returns it. In truth, a neighbour signed for it on the second attempt and the courier missed the note. If the sender now ships an identical parcel, the family receives two. The courier’s careful tracking numbers could not help, because the second shipment carries a new number.
Two longer stories
The payment service that double-charged after a restart
A payment service publishes a “payment requested” event for every order. One night a deployment restarts the service while it has a few hundred events in its buffer. On startup, the service reads a “pending” table and re-publishes anything not marked as sent. Some of those events had reached Kafka before the restart. The new producer has a new identity, so the broker stores them again. A downstream worker that charges customers on every “payment requested” event then charges some customers twice.
The fix had three parts. Each event now carries the order number as a unique identifier. The charging worker records identifiers it has processed and skips repeats. The service also marks events as sent only after a successful acknowledgement, so fewer are re-published. Idempotence was on the whole time, yet it could not help, because the duplicates were created across sessions.
The delivery timeout that looked like a failure
An inventory service sends stock updates. During a brief broker slowdown, some sends exceed the delivery timeout and are reported as failed. The service’s error handler puts the updates back on its internal queue and sends them again. Many of the first attempts had in fact been stored. Stock counts drift upward by the size of the duplicates.
The team removed the manual retry loop, raised the delivery timeout to match the longest slowdown they had seen, and changed the stock update from “add 5” to “set the count to 105 as of version 42”. Repeats became harmless, and the drift disappeared.
Both stories are like a builder who lays a brick, is unsure whether it set, and lays another brick in the same spot. The wall ends up with an extra brick. The fix is either to check the wall before laying (an inbox check) or to design the job so that laying the same brick twice leaves the same wall (a set operation).
10What Idempotence Does Not Protect
Knowing the edges of a feature is as important as knowing its strengths. Most duplicate incidents in idempotent setups happen just outside these edges.
New session, new identity
If your application restarts and resends records, the broker treats them as new. Idempotence does not survive the loss of the producer object.
Different IDs
Two application instances sending the same event produce two records, since each has its own producer ID.
No atomic writes
Sending to several partitions or topics can succeed on some and fail on others. Idempotence does not make the group all-or-nothing.
Your own retry loop
A manual second send call is a new record. Only the library’s internal retries are recognized.
Reading is separate
A consumer that crashes before saving its position will re-read and may repeat its own side effects.
Databases and services
Writing to a database or calling a web service after reading from Kafka is outside Kafka’s control.
A seatbelt protects you in a car crash but does nothing about a flat tyre or a wrong turn. It has a clear job, and it does that job well. Treating idempotence as a general cure for every duplicate is the same kind of mistake as expecting a seatbelt to fix your route.
The same event from two places
A very common surprise involves two copies of a service both reacting to the same trigger, such as a message from a queue or a scheduled job that fires on two machines. Each copy produces the same business event. Both are accepted, because they come from different producers. This kind of duplicate has to be handled by giving events a stable business identifier and letting consumers or storage ignore repeats.
Copies between clusters
Tools that copy data between Kafka clusters, such as mirroring tools, read from one cluster and produce to another. The copy has its own producer, so identities and sequence numbers do not travel. If a mirroring task restarts, some records can be copied twice. Downstream consumers should still expect the occasional duplicate.
Saying “we have idempotence turned on, so we have no duplicates” is one of the most common false comforts in Kafka projects. It is true only for retries inside a single producer session writing to a single partition.
11From Idempotence to Transactions
When a session-level guarantee is not enough, Kafka offers transactions. They build directly on idempotence and extend it in two important ways.
What transactions add
A stable name
You choose a fixed transactional.id. When the application restarts with the same name, the cluster recognizes it as the same logical producer and raises the epoch, so the old instance can no longer write. This is called fencing.
All-or-nothing writes
A group of writes to several partitions, plus the consumer’s saved position, can be committed as one unit. Either every part becomes visible or none does.
Careful readers
Consumers set to read only committed data ignore records from open or aborted transactions.
flowchart TB
IN["Read records from input topic"] --> BEGIN["Begin transaction"]
BEGIN --> W1["Write results to output topic A"]
BEGIN --> W2["Write results to output topic B"]
BEGIN --> OFF["Include consumer position in the transaction"]
W1 --> C{"All steps done?"}
W2 --> C
OFF --> C
C -- "yes" --> COMMIT["Commit: everything becomes visible together"]
C -- "no" --> ABORT["Abort: none of it becomes visible"]
Moving house is a good picture. Packing boxes and unloading them at the new home are many small steps, but you only consider the move complete when everything has arrived. If the van breaks down halfway, you do not want half your furniture declared moved. A transaction is the rule that says “the move either finishes entirely or is treated as never started”.
How it works under the hood
A component in the cluster called the transaction coordinator keeps the state of each transaction in an internal topic. When a transaction commits, the coordinator writes special marker records into every partition that took part. Consumers reading in the committed mode use these markers, together with a position called the last stable offset, to hide anything not yet committed.
| Feature | Idempotent producer | Transactional producer |
|---|---|---|
| Removes duplicates from retries | Yes | Yes |
| Survives application restarts | No | Yes, with a stable transactional id |
| Atomic writes to many partitions | No | Yes |
| Ties consumer position to output | No | Yes |
| Extra cost | Very small | Higher: coordinator work and commit markers |
| Stops old instances (zombies) | No | Yes, through epoch fencing |
Every transactional producer is also an idempotent producer. Idempotence is the foundation, and transactions are the extra floors built on top of it.
The zombie problem and fencing
A zombie is an old copy of a program that the system believes is dead but which is still running. Perhaps its machine froze for a minute, the cluster replaced it, and then it woke up and started writing again. Two copies now believe they own the same work, which is a recipe for duplicates and corrupt results.
Transactions handle zombies with the epoch. When the replacement starts and registers under the same transactional name, the coordinator raises the epoch by one. From then on, any request from the old copy carries the old epoch, and the cluster rejects it. The old copy learns that it has been fenced and must stop.
Copy A works
It writes with the transactional name “billing-1” and epoch 3.
Copy A freezes
A long pause makes the cluster think it has died, and the work is moved.
Copy B takes over
It registers as “billing-1”, and the epoch rises to 4.
Copy A wakes up
It tries to write with epoch 3 and is rejected with an invalid-epoch error, so its stale work never lands.
A security guard changes the door code whenever a new shift starts. A guard from the previous shift who falls asleep and wakes up later can still use the old code, but the door does not open. The new code is the raised epoch.
The transactional name must be stable and unique for each logical unit of work, for example one name per input partition being processed. Two active instances sharing a name will keep fencing each other.
12Idempotence in Kafka Streams and Kafka Connect
Higher-level Kafka tools sit on top of the producer, so they inherit its protection and add some of their own.
Kafka Streams
Kafka Streams is a library for processing events, and inside it every step that writes back to Kafka uses a producer. Its default processing mode is at-least-once: nothing is lost, but after a crash some input may be processed again, which can repeat outputs or double count in totals. Switching the processing guarantee setting to the exactly-once option turns on transactions. Then the output records, the changes to the local state and the saved input position are committed together as a single unit.
The important lesson is that plain idempotence is not enough for a stream processor. A processor both reads and writes. To avoid duplicates it must tie its read position to its output, and only transactions do that. Idempotence remains the layer underneath, making retries of individual batches safe.
A bank clerk moves money between two accounts and also ticks the transfer off a to-do list. If the clerk collapses after moving the money but before ticking the list, a colleague will repeat the transfer. Doing the move and the tick as one inseparable action prevents that. Exactly-once processing in Streams is the inseparable action.
Kafka Connect
Connect moves data between Kafka and other systems. Its source connectors use producers to write into Kafka, so in recent versions they benefit from idempotent settings by default. Newer Kafka versions, starting with 3.3, also let certain source connectors offer exactly-once delivery by using transactions for the records and the connector’s progress markers.
For sink connectors, which write out to databases, search indexes and storage, the concern shifts to the target. A sink reads at least once, so it may write a record twice. The target must therefore accept repeated writes safely. Well-designed sinks do this through upsert modes, record keys used as document identifiers and file names built from offsets.
Upsert mode
Insert the row if new, update it if it exists. Writing the same record again leaves the same row.
Key as document ID
Using the record key as the document identifier means a repeated write replaces the same document.
Deterministic file names
Naming files from topic, partition and offset range means a repeat overwrites the same file instead of creating a copy.
When choosing a sink connector, read how it handles repeated writes. A sink that can only append rows is a duplicate factory waiting for a restart.
Choosing a processing guarantee in Streams
| Mode | After a crash | Cost | Good for |
|---|---|---|---|
| At-least-once | Some input may be processed again, so outputs can repeat and counts can rise | Lowest | Dashboards, approximate metrics, idempotent outputs |
| Exactly-once | Results are as if each input was processed once | More coordinator work and commit markers | Money, inventory, billing, any total that must be exact |
Exactly-once inside Kafka Streams covers what happens inside Kafka. If your Streams application also calls an outside service or writes to a database, those effects sit outside the transaction and still need to be repeat-safe.
13Designing Idempotent Consumers
Kafka cannot make the outside world safe to repeat. That last step is yours, and a small set of patterns handles nearly every case.
Why consumers see duplicates
A consumer saves its position, called an offset, periodically. If it crashes after doing the work but before saving the offset, it restarts from the older offset and does the work again. Rebalancing, which moves partitions between consumers, can cause the same effect. This means every consumer that does something non-trivial should expect to see a record more than once.
The core patterns
Set, do not add
Design the effect as “set the balance to this value” rather than “add this amount”. Repeating a set changes nothing.
Upsert by natural key
Write with a unique business key so a repeat replaces the same row.
Processed-events table
Store each event’s unique identifier in a table with a uniqueness rule. If the insert fails, the event was already handled and can be skipped. This is often called the inbox pattern.
Version checks
Give each entity a version or timestamp. Apply an update only if it is newer than what is stored, so old repeats are ignored.
One transaction for both
Save the result and the consumer’s position in the same database transaction, so they succeed or fail together.
Idempotency keys in events
Have the producer place a unique identifier inside every business event. Consumers use it to detect repeats.
flowchart TB
R["Consumer reads a record with a unique event id"] --> CH{"Is this id in the processed table?"}
CH -- "yes" --> SK["Skip: already handled"]
CH -- "no" --> TX["Start database transaction"]
TX --> WK["Apply the business change"]
WK --> MK["Record the event id as processed"]
MK --> CM["Commit the database transaction"]
CM --> OF["Save the consumer offset"]
SK --> OF
A hotel front desk checks a booking reference before making a reservation. If the reference is already in the book, the clerk says “we have that already” and does nothing. If it is new, the clerk writes the reservation and the reference together. Guests can phone ten times about the same booking and only one room is held.
How long to remember
A table of processed identifiers cannot grow forever. Decide how far back a duplicate could realistically come from. If your consumers can be down for at most a day and Kafka retains data for seven days, remember identifiers for at least seven days, or use a method that needs no memory at all, such as upserts by key. Clean up old entries on a schedule.
Memoryless methods
- Upserts and “set” operations need no extra table.
- Nothing to clean up or size.
- Work with any amount of history.
Memory-based methods
- Needed when the effect cannot be made a “set”.
- Require a table, an index and a retention plan.
- Add a lookup to every event, which costs time.
Sending an email, charging a card or calling a partner’s service cannot be undone by a database rule. For such effects, pass along an idempotency key so the other side can ignore repeats, or record the intention first and perform it from a separate worker.
A worked example: the inventory consumer
Suppose a warehouse service listens to “item shipped” events and reduces the stock count. The naive version reads an event and subtracts one from the count. After a crash and replay of five events, the count is five too low. Let us improve it in three steps.
Add an identifier
Every “item shipped” event carries a unique shipment number created when the shipment was made.
Record what was applied
The service stores each shipment number in a table with a uniqueness rule. Trying to insert one that already exists fails, which signals a repeat.
Do both in one transaction
Subtracting the stock and inserting the shipment number happen together. If either fails, both are undone, so the count and the table never disagree.
Now, after a crash and replay of the same five events, the insert fails for each of them, the subtraction is rolled back, and the count stays correct. The service does slightly more work per event, and in return, replays are harmless.
What this design gives
- Correct counts after any number of replays.
- Clear audit trail of every shipment applied.
- Simple recovery: just restart and reprocess.
What it costs
- An extra table and an index to maintain.
- An extra write per event.
- A retention plan for old identifiers.
A warehouse worker who ticks each shipment number off a list before updating the shelf count never removes stock twice, even if the supervisor reads the same delivery note again. The list is the memory, and the tick and the count change together.
14Idempotence Beyond Kafka
The same idea shows up all over software. Looking at other systems shows what Kafka’s approach shares with them and where it differs.
| Where | Idempotent by design? | How duplicates are handled |
|---|---|---|
| Web requests that replace a resource | Yes | Replacing with the same content leaves the same result |
| Web requests that delete a resource | Yes | After the first delete, later ones find nothing to remove |
| Web requests that create something new | Not by default | Each call may create another item unless a key is supplied |
| Stripe payment requests | With a key | An idempotency key sent by the client makes repeated requests return the first result |
| Amazon SQS FIFO queues | Within a window | A deduplication identifier makes repeats within five minutes be ignored |
| Database upserts | Yes | A unique key decides between insert and update |
| Kafka idempotent producer | Within a session | Producer ID plus sequence numbers, checked by the broker |
Client-supplied keys
Payment companies such as Stripe have long documented an “idempotency key” approach. The client generates a unique key for each logical operation and sends it with the request. If the client retries after a timeout, the server sees the same key and returns the original result rather than charging again. Stripe’s public documentation says stored keys are eventually removed, which reminds us that every such memory needs a retention period.
A restaurant gives each table a reservation number. If you phone twice to confirm the same booking and quote the same number, the host tells you “already booked” rather than reserving a second table. The number is your idempotency key, and the reservation book is the server’s memory.
Kafka’s twist
Kafka’s producer does not ask you for a key. It creates and manages its own identity and sequence numbers, and the broker checks them at write time. This works well for the producer-to-broker hop but says nothing about your business meaning. A business identifier chosen by you, such as an order number, is more powerful because it survives restarts and works across systems. The two approaches complement each other.
A helpful mental split: Kafka’s idempotence handles technical duplicates caused by retries, while business identifiers handle logical duplicates caused by people, schedulers and restarts.
Designing good idempotency keys
A key is only useful if it is chosen well. A good key identifies one logical operation, not one attempt. Two attempts to do the same thing must carry the same key, and two different things must carry different keys.
Created with the operation
Generate the key when the user action or business event first occurs, then keep it through every retry and hand-off.
Derived from business facts
An order number combined with a step name, such as payment requested, is naturally unique and stable.
Generated at send time
A new random value on each attempt makes every retry look like a new operation, which defeats the purpose.
Reused for different work
Using one key for two different payloads makes the second silently vanish or return the wrong result.
A cloakroom ticket must match one coat. If the attendant gives the same number to two coats, someone leaves with the wrong one. If the attendant gives a fresh ticket every time you ask about your coat, the system cannot tell that it is one coat being asked about repeatedly.
Decide the scope of a key up front: unique for the whole system, or only unique per customer. Write it down, because different teams guess differently and end up with collisions or gaps.
15Performance and Scalability
A common worry is that a safety feature must be slow. For idempotence, the cost is small, and the safest configuration is usually fast enough.
Where the cost comes from
A few extra bytes
Each batch carries a producer ID, an epoch and a sequence number in its header. Spread over a batch of many records, this is a negligible amount.
Small per-producer state
The broker holds details of the last few batches per producer and partition. That is small for each entry, though it adds up when very many producers write to very many partitions.
Waiting for copies
Most of the visible cost comes from acks set to all, which waits for other brokers, not from idempotence itself.
Brief checks
The sequence comparison is a tiny in-memory check made once per batch.
A stamp on a passport takes a second and a little ink. The real delay at a border is the queue, not the stamp. Idempotence is the stamp, and waiting for replicas is the queue.
A worked example in numbers
Imagine one producer sending to one partition, with batches of 100 records. Batch one carries sequence numbers 0 to 99, batch two carries 100 to 199, and so on. The broker’s remembered position after batch two is 199. Suppose the acknowledgement for batch two is lost and the producer retries it. The broker sees a batch starting at 100 while its position is already 199, so it knows those numbers are old. It skips the write and replies with success. If instead a batch arrived starting at 300 while the position was 199, the broker would notice the missing range 200 to 299 and complain.
Scaling considerations
- Many producers, many partitions: in very large fleets, count producers times partitions to estimate broker memory for producer state. Sharing one producer object across threads keeps this number small.
- Throughput tuning: larger batches and a short waiting time before sending raise throughput, and compression reduces network use. These tunings work the same with or without idempotence.
- Transactions cost more: commits add coordinator work and marker records. Group many records into each transaction instead of committing after every record.
- Latency: waiting for replicas dominates. A healthy cluster in one region typically adds only a few milliseconds.
Measure before deciding that safety is too expensive. Run the same load with idempotence on and off and compare throughput and latency. Most teams find the difference is small enough to ignore.
Getting more speed without touching safety
When throughput needs to rise, resist the urge to loosen the safety settings. Several other levers deliver far more speed with no risk to correctness.
Bigger batches
Allow more data per batch so each request carries more records and the fixed per-request cost is shared.
A short waiting time
Let the producer wait a few milliseconds so batches can fill before they are sent.
Compression
Compress batches to reduce network use and broker disk. Text-heavy records shrink dramatically.
More partitions
Spread load across more partitions so more brokers share the work, remembering that the count is hard to change later.
Healthy brokers
Fast disks, enough network and balanced leaders often matter more than any client setting.
To move more people through a train station, you add wider gates and more trains, not remove the ticket check. Ticket checks are quick and prevent chaos. Idempotence is the ticket check; batching and partitions are the gates and the trains.
16Monitoring, Logging and Security
Idempotence works quietly, so you need a few signals to know it is healthy and to spot the moments when it is stretched.
What to watch on producers
| Signal | What it tells you |
|---|---|
| Record retry rate | How often sends are being repeated; a rising rate points to network or broker stress |
| Record error rate | How many sends fail for good after retries |
| Request latency | Whether acknowledgements are getting slower |
| Batch size and records per request | Whether batching is working as expected |
| Time spent waiting for metadata or buffer space | Whether the producer is being blocked |
Errors worth knowing by name
Out-of-order sequence
The broker saw a gap. Often caused by data loss on the broker side, such as an unclean leader election, or by a producer whose state was forgotten.
Unknown producer
The broker no longer remembers this producer, typically after a long idle period. Modern clients recover.
Invalid producer epoch
A newer instance with the same transactional name has taken over, and this older one is now shut out.
Delivery timeout
The producer could not confirm the write in time. The record may or may not be stored.
A smoke alarm is silent for years, then very loud for a few seconds. You do not watch it daily, but you do check its battery. Retry and error rates are the battery check for idempotence: a quiet reading means the feature is doing its job.
Logging
At startup, clients log their effective settings. Keep these logs and check that idempotence, acknowledgements and retries appear as intended. When errors occur, log the topic, partition, producer ID and sequence values if the client exposes them, because these make it possible to trace a duplicate or a gap to a specific moment.
Security
- Permissions: older cluster versions required a special permission called idempotent write on the cluster for idempotent producers. Newer versions accept the ordinary write permission on the topic. Transactional producers additionally need permission on their transactional name.
- Encryption and sign-in: producer identities travel in normal traffic, so protect connections with encryption and authentication like any other Kafka client.
- Deduplication stores: tables of processed event identifiers may reveal business activity. Give them the same access controls and retention rules as the data they describe.
- Client separation: give each application its own credentials, so a misbehaving producer can be identified and limited.
Producer IDs are labels, not secrets. They do not prove who a client is. Rely on authentication and permissions for identity, and treat sequence numbers purely as duplicate-detection tools.
A runbook for a duplicate incident
When a user reports double charges or inflated totals, a calm sequence of checks finds the source faster than guessing.
Confirm where the duplicate lives
Check the Kafka topic itself. Are there really two records, or is the duplicate created later by a consumer or an outside system?
Compare producer identities
If the two records came from the same producer session, idempotence should have prevented it, so suspect a configuration problem. If they came from different sessions, suspect a restart or two instances.
Check the timeline
Look for deployments, crashes, rebalances or broker events around the time of the duplicate.
Inspect the effective configuration
Verify idempotence, acknowledgements and retries in the startup logs of the producing service.
Look for manual retries
Search the application code for its own retry or resend logic around the send call.
Fix the consumer too
Even after finding the cause, make the consumer repeat-safe so the next unknown cause does no harm.
Keep event identifiers in your logs and traces. With one identifier you can follow an event from the original action, through the producer and the topic, to every consumer effect, and see exactly where the second copy appeared.
17Design Patterns and Anti-Patterns
A few habits make idempotent systems reliable. A few shortcuts undermine them.
Patterns that work
Safe defaults
Keep idempotence on, use acknowledgement of all, and pair it with a sensible in-sync replica count.
Key by entity
Use the customer or account identifier as the record key, so related events stay together and in order.
Business event IDs
Stamp every business event with a unique identifier at the moment it is created, before it ever reaches Kafka.
Outbox
Write the event to a table in the same database transaction as the business change, and let a separate process publish it. This avoids losing or inventing events.
Inbox
Record processed event identifiers on the consuming side so repeats are skipped.
Transactions for read-process-write
When a service reads from Kafka and writes back to Kafka, use transactions to make the round trip atomic.
Anti-patterns to avoid
What people do
They assume idempotence covers application restarts, and they resend recent records on startup “just in case”.
Why it hurts
A restarted producer has a new identity, so resent records are accepted as new. Every restart produces a small batch of duplicates.
What to do instead
Use a stable transactional name, or add business identifiers and make consumers ignore repeats.
What people do
They create a new producer object for every request or every message.
Why it hurts
Each producer asks for a new identity, opens new connections and starts a new sequence. Overhead climbs, broker state grows, and any protection tied to a session is lost each time.
What to do instead
Create one producer per application and share it across threads.
What people do
They lower the acknowledgement level, or set retries to zero, to gain speed while still expecting exactly-once behaviour.
Why it hurts
These settings conflict with idempotence. Depending on the version, the client either refuses to start or silently drops the protection, and messages can then be lost or duplicated.
What to do instead
Tune batch size, waiting time and compression for speed, and leave the safety settings alone.
What people do
They build a table of processed event identifiers and never delete anything from it.
Why it hurts
The table grows without limit, lookups slow down, and storage costs rise until the consumer becomes the slowest part of the system.
What to do instead
Choose a retention period that covers the longest realistic duplicate window and delete older entries on a schedule, or use memoryless methods such as upserts.
18Best Practices and Common Mistakes
These points come up in nearly every team that runs Kafka in production.
Do
- Keep idempotence enabled and check the effective settings in startup logs.
- Use acknowledgement of all, with a replication factor of three and at least two in-sync replicas.
- Create one long-lived producer per application and share it across threads.
- Choose record keys that group related events, such as a customer or account.
- Put a unique business identifier inside every important event.
- Make consumers idempotent with upserts, “set” operations or an inbox table.
- Use transactions when you read from Kafka and write back to Kafka.
- Alert on producer retry and error rates.
Do not
- Assume idempotence survives application restarts.
- Wrap sends in your own endless retry loop.
- Lower acknowledgements or disable retries for speed.
- Create a producer per message.
- Let two instances emit the same event and expect Kafka to merge them.
- Increment counters in downstream systems without a duplicate check.
- Keep processed-event tables forever.
- Ignore out-of-order-sequence errors as harmless noise.
The mistakes beginners meet first
“We turned it on, so we can never see duplicates”
Duplicates still appear after restarts, after manual resends and when several instances emit the same event.
Conflicting settings
An old configuration file sets one acknowledgement or zero retries, quietly disabling the protection.
Trusting the consumer to be safe
A downstream service adds to a total on every message, so any reprocessing inflates it.
Treating a timeout as a definite failure
The application retries after a delivery timeout, but the first attempt had actually succeeded.
Forgetting the mirrored copy
Data copied to another cluster is written by a different producer and may contain duplicates the original never had.
Mixing up order and duplicates
Idempotence keeps order only inside a partition. Events with different keys can land on different partitions and interleave freely.
A good fire safety plan has smoke alarms, marked exits and a practised drill. No single item is enough, and each covers a different failure. Reliable messaging needs the same layers: idempotent producers, stable business identifiers, idempotent consumers and monitoring.
A quick review checklist
Producer settings
Confirm idempotence, acknowledgement of all and sensible timeouts in the effective configuration.
Identity
Decide whether a plain idempotent producer or a transactional producer with a stable name is required.
Business identifiers
Ensure every important event has a unique, stable identifier created before publishing.
Consumers
Confirm every consumer effect is a set, an upsert or guarded by an inbox check.
Monitoring
Alert on retry rates, errors, lag and duplicate-skip counts.
Testing for duplicates on purpose
You cannot trust a design until you have watched it fail safely. Happy-path tests never reveal duplicate bugs, because duplicates only appear when something goes wrong. Good teams therefore break things deliberately in a test environment.
Inject delays and drops
Use a fault-injection proxy, such as Toxiproxy, to slow, cut or delay traffic between producers and brokers, and check that no duplicates appear.
Restart leaders
Stop the broker that leads a busy partition in the middle of a load test and compare record counts.
Kill and restart
Terminate producer and consumer processes abruptly, restart them, and verify that totals remain correct.
Count identifiers
Send records with known unique identifiers, then check the destination for identifiers that appear more than once or not at all.
A fire drill tests what people do when the alarm rings, not what they do on a quiet day. It exposes the blocked exit and the confusing sign long before a real fire. Fault-injection tests are the fire drills of messaging systems.
What to record during a test
- Total records sent, and total distinct identifiers seen at the destination.
- Number of retries and errors on the producer during the fault.
- Time to recover after each fault is removed.
- Whether any consumer effect, such as a counter or an email, happened more than once.
Run these tests again whenever you change client versions, cluster versions or key settings, since defaults and behaviour improve between releases and can also shift in subtle ways.
19Real-World Stories in One Place
Public documentation and talks from well-known projects and companies show the same ideas in different clothes.
Confluent and the design of exactly-once
Engineers at Confluent, the company founded by Kafka’s creators, led the design of idempotent producers and transactions, published as KIP-98 and released in Kafka 0.11. They wrote openly that the goal was to make retries safe and to let stream processors avoid double counting, and that end-to-end safety still depends on how outside systems are written to.
Stripe and idempotency keys
Stripe’s public API documentation explains how clients attach an idempotency key to payment requests so that a retry after a timeout returns the original outcome instead of charging twice. This is the same principle as Kafka’s sequence numbers, applied at the business level and controlled by the client.
Amazon SQS FIFO queues
Amazon documents a deduplication identifier for FIFO queues. Messages sent with the same identifier within a five-minute window are treated as one. It shows a common design choice: remember identifiers only for a limited time to keep the memory small.
Debezium and at-least-once delivery
Debezium, the open-source change-data-capture project, documents that its connectors deliver each change at least once, so a restart can repeat some events. It advises consumers to be prepared for duplicates, using the event’s source position or key to detect them.
Search indexes with document identifiers
Search systems such as Elasticsearch replace a document when the same identifier is written again. Sink connectors that use the record key as the document identifier therefore turn repeated writes into harmless overwrites, a simple and popular idempotent design.
LinkedIn and the early days of Kafka
Kafka began at LinkedIn as a high-volume log with at-least-once delivery, and consumers were expected to handle the occasional repeat. Idempotent producers and transactions arrived years later, after many companies asked for stronger guarantees.
20Frequently Asked Questions
Short answers to the questions beginners and experienced engineers ask most often.
21Summary and Key Takeaways
If you keep only one page of this guide, keep this one.
Idempotence turns a dangerous action, the retry, into a safe one. Kafka’s idempotent producer does this at the producer-to-broker hop with a producer ID and sequence numbers. Everything beyond that hop, from restarts to consumers to outside systems, is a design job that you must finish yourself with stable identifiers and repeat-safe operations.
Key Takeaways
- Idempotent means safe to repeat. Doing an action many times has the same effect as doing it once.
- Duplicates come from safe retrying. A lost acknowledgement makes the producer resend a batch that was already stored.
- Kafka uses IDs and sequence numbers. The broker compares each batch with the last accepted number, writes the next one, ignores the old ones and rejects gaps.
- The memory lives in the log. Producer details travel in batch headers, so leader changes do not lose the protection.
- The protection has clear limits. It covers one producer session and one partition, and it does not survive restarts, manual resends or multiple emitters.
- Transactions build on it. A stable transactional name, atomic multi-partition writes and consumer read-committed mode extend the guarantee.
- Finish the job downstream. Use business identifiers, upserts and inbox tables so consumers and outside systems can safely see the same event twice.