Idempotent Producers in Kafka: Sending Every Message Exactly Once
Networks fail, acknowledgements get lost, and producers retry. This guide explains how Kafka's idempotent producer quietly removes the duplicates that retries would otherwise create, how it works inside the broker, and where its protection ends.
Imagine you press the button to call a lift. Nothing lights up, so you press it again, and then a third time for good measure. The lift still arrives only once, because the lift system does not care how many times you pressed. It only cares that someone wants to go up. That behaviour has a name in computer science: idempotence.
Now imagine a different system, where every press of the button sends one more lift to your floor. Three presses, three lifts, and a very confused building. For a long time, a Kafka producer behaved a bit like that second system whenever the network misbehaved. It would retry a message, and sometimes the retry would create a second copy.
The idempotent producer is Kafka’s answer. It teaches the broker to recognise a message it has already stored, so that retries become safe. In this tutorial we will start from zero, build up every idea step by step, and finish with production advice you can use at work.
1The Problem: Why Duplicate Messages Happen
Before learning the cure, we need to feel the pain clearly.
A message that gets sent twice
A Kafka producer is a program that sends messages to Kafka. Each message travels over a network to a machine called a broker, and the broker writes it to disk. After writing, the broker sends back a small reply that says “I have it safely”. This reply is called an acknowledgement.
Networks are not perfect. A cable can be unplugged, a router can be overloaded, a broker can restart at the worst possible moment. Any of these can make a message or its acknowledgement disappear on the way.
Here is the tricky part. If the acknowledgement is lost, the producer sees silence. From the producer’s point of view, two very different things could have happened: the broker never received the message, or the broker stored it but the reply got lost. The producer cannot tell these two situations apart.
You post a parcel and ask the shop to text you when it is delivered. No text arrives for two days. Was the parcel lost, or was only the text lost? If you send a second parcel to be safe, and the first one actually arrived, your friend now owns two identical gifts.
The lost acknowledgement, drawn as a picture
The picture below shows exactly how a duplicate is born when idempotence is switched off. Read it from top to bottom.
sequenceDiagram
participant P as Producer
participant B as Broker Leader
P->>B: Send message M1
B->>B: Append M1 to the log
B--x P: Acknowledgement is lost
Note over P: No reply arrives. Timeout. Retry.
P->>B: Send message M1 again
B->>B: Append M1 again
B->>P: Acknowledgement
Note over B: The log now holds M1 twice
The broker did nothing wrong. It received two requests, and it obeyed both. It had no way to know that the second request was a repeat of the first, because both looked like brand new messages.
Two bad choices
Before idempotence, a producer developer had to pick between two uncomfortable options.
Choice 1: Never retry
If the producer never retries, a lost message stays lost. This is called at-most-once delivery. It is simple, but you can silently lose payments, orders or sensor readings whenever the network hiccups.
Choice 2: Always retry
If the producer retries on every failure, no message is lost, but some are written twice. This is called at-least-once delivery. It protects data, but it can double charge a customer or count one click as two.
Duplicates are dangerous because they look perfectly normal. Nothing crashes and no alarm rings. The wrong numbers simply appear later in a report, a balance or an invoice, and finding the cause can take days.
2What Is an Idempotent Producer?
Idempotent is a long word for a very simple idea.
Idempotence in everyday life
An action is idempotent if doing it once or doing it ten times leaves the world in the same final state. Setting a room thermostat to 22 degrees is idempotent, because the tenth press leaves the room at 22 degrees just like the first press. Switching a light on is idempotent too.
Adding 100 rupees to a bank balance is not idempotent. Doing it twice gives a different result from doing it once. Sending a message to a Kafka topic looks like this second kind of action, because every send adds a new entry at the end of the log.
Think of a cinema ticket with a printed seat number. If two people show a ticket for seat G12, the usher lets only one of them in. The seat number turns “enter the cinema” from a repeatable action into one that happens at most once per seat.
The Kafka meaning
An idempotent producer is a Kafka producer that gives every message a hidden identity. The broker uses that identity to notice repeats and ignore them. Because of this, the producer can retry as often as it needs, and the log still ends up with exactly one copy of each message.
You do not have to change how you write your sending code. You switch the feature on with one configuration setting, and the client library and the broker do the extra bookkeeping for you. The idea was introduced in Kafka 0.11, and in modern Kafka versions it is switched on by default.
The word “producer” matters here. The idempotent producer protects the path from your application into a Kafka partition. It does not protect the consumer side, and it does not protect duplicates that your own application logic creates. We will look at these limits carefully in a later chapter.
3Kafka Basics Before We Go Deeper
A few simple words carry the whole story, so let us define each one.
Broker
A server that stores Kafka data and answers requests from producers and consumers. A Kafka cluster is a group of brokers working together.
Topic
A named stream of messages, such as “orders” or “payments”. Think of it as a labelled category where related messages live.
Partition
A topic is split into partitions. Each partition is an ordered log where messages are only ever added at the end.
Offset
The position number of a message inside a partition. The first message has offset 0, the next has offset 1, and so on.
Leader and Followers
Each partition has copies on several brokers. One copy is the leader and takes all writes. The others are followers that copy the leader.
Producer
The client program that writes messages to topics. Your order service or your click tracker acts as a producer.
Batch
Producers group several messages for the same partition into one batch and send them in a single request. This is faster than sending one by one.
Acknowledgement (acks)
The reply from the broker. A setting called acks decides how many copies must exist before the broker replies.
A topic is like a library section called “Novels”. The partitions are separate shelves inside that section. Each shelf only grows to the right, new books go at the end, and each book’s position on the shelf is its offset. The leader is the main librarian of a shelf, and followers keep spare copies of that shelf in other buildings.
How a producer sends data
When your code asks the producer to send a message, the producer does not fire it off instantly. It first decides which partition the message belongs to, then places it in a small buffer for that partition. After a short wait, or when enough messages are collected, the whole buffer is sent as one batch.
The batch travels to the leader broker of that partition. The leader writes it, waits for the followers if needed, and then replies. If no reply arrives within the allowed time, the producer sends the same batch again. This automatic resend is what we call a retry.
What acks really means
The acks setting controls how careful the broker is before it says “done”. With acks set to 0, the broker does not reply at all, so the producer never learns about failures. With acks set to 1, the leader replies as soon as it has written the batch itself, even if followers have not copied it yet. With acks set to all, the leader waits until every in-sync follower has a copy.
This choice is important for idempotence, because the idempotent producer needs the strongest setting, acks equal to all. Without it, a message acknowledged by a leader that then crashes could vanish, and the whole promise would break.
Messages inside one partition keep the order in which they were written. Kafka does not promise any order across different partitions. This detail explains why the idempotent producer keeps a separate counter for every partition.
How a message finds its partition
Every message sent to Kafka can carry an optional key, such as a restaurant ID or a customer ID. If a key is present, the producer runs it through a hashing function to consistently choose the same partition for that key every time. Messages with no key are spread across partitions in a round-robin-like pattern to balance load evenly.
This detail connects directly to idempotence. Because all messages sharing a key always land on the same partition, and because that partition keeps one strict order, giving related events a shared key is what lets you rely on both ordering and duplicate-free delivery for that group of events, such as every update belonging to a single order.
A hospital assigns every patient a file number, and all of that patient’s records always go into the same filing cabinet drawer, in the order they were created. Two different patients might use different drawers entirely, and nobody expects drawer three to stay in step with drawer seven.
4Delivery Guarantees: At Most Once, At Least Once, Exactly Once
Every messaging system makes a promise about duplicates and losses. Kafka can make three different promises.
At most once
Each message is delivered zero or one time. It is never repeated, but it may be lost. This is what you get if you never retry.
At least once
Each message is delivered one or more times. It is never lost, but it may be repeated. This is what you get with retries but without idempotence.
Exactly once
Each message has its effect one time only. It is neither lost nor repeated. This is the most valuable promise and also the hardest, because the network cannot tell you what really happened.
| Guarantee | Lost messages? | Duplicate messages? | Typical use |
|---|---|---|---|
| At most once | Possible | Never | Non-critical metrics where a gap is acceptable |
| At least once | Never | Possible | Logs and events that can be cleaned up later |
| Exactly once (to a partition) | Never | Never | Payments, orders, inventory, billing |
At most once is a teacher who calls out a homework question a single time, whether or not you heard it. At least once is a teacher who keeps repeating it until someone answers, which is safe but noisy. Exactly once is a teacher with a class register who ticks your name after your first answer and ignores any repeat.
Where the idempotent producer fits
The idempotent producer upgrades the producer-to-broker step from at-least-once to exactly-once for each partition within one producer session. It gets rid of the duplicates caused by retries, while keeping the safety of retries.
Notice the careful words: for each partition and within one session. These words are not decoration. They describe the exact size of the promise, and later chapters explain what falls outside it.
Many beginners hear “idempotent producer” and think their whole pipeline is now exactly-once. It is not. It is one link in the chain, an important link, but the consumer, the processing logic and the database at the end all need their own thought.
5The Three Secret Ingredients: Producer ID, Epoch and Sequence Number
The whole trick rests on three small numbers stamped on every batch.
Producer ID (PID)
A unique number that the cluster gives to a producer when it starts. It answers the question “who is sending this?”.
Producer Epoch
A small version number attached to the PID. It answers “which generation of this producer is this?” and helps the cluster handle recovery.
Sequence Number
A counter that grows by one for every message sent to a given partition. It answers “which message in the line is this?”.
Picture a courier company. The producer ID is your customer account number. The epoch is the version of your account card, in case the company reissues it. The sequence number is the running number on each parcel label, so the sorting office can say “parcel 7 from customer 42 has already been handled”.
Why the counter is per partition
The producer keeps a separate sequence counter for each topic partition it writes to. If a producer sends to three partitions, it manages three independent counters. Each partition is its own ordered log, so each one needs its own line of numbers.
The counter starts at zero and increases by the number of messages in each batch. A batch carrying five messages that begins at sequence 10 covers sequence numbers 10 to 14, and the next batch starts at 15. The batch header records the first number and the count, so the broker can work out the rest.
Where these numbers live
The producer ID, the epoch and the first sequence number are written into the header of every record batch, right next to the messages. They are stored on disk as part of the log itself. This is a clever design choice, because anything stored in the log is also copied to follower replicas.
As a result, the followers learn the same producer history as the leader. If the leader crashes and a follower takes over, the new leader already holds the information it needs to spot duplicates. We will return to this powerful point in the chapter on failures.
When the producer starts, it asks the cluster for an identity. The cluster hands out a unique producer ID and an epoch. You never choose these values yourself, and you never see them in your application code. They are internal plumbing.
The idea in one sentence
Every batch says, “I am producer 7, generation 0, and this batch starts at position 15 for this partition.” If a broker sees exactly that stamp a second time, it knows this is a repeat, and it does not write it again.
6Internal Working: The Life of One Message
Let us follow a single message from your application all the way into the log.
The producer starts and asks for an identity
On startup, the client contacts the cluster and receives a producer ID and an epoch. From now on, every batch this producer sends will carry them.
Your code sends a message
The application hands a message to the producer. The producer picks the partition, and puts the message in that partition’s buffer.
The batch receives its sequence number
When the buffer is ready, the producer stamps the batch with the producer ID, the epoch and the next sequence number for that partition.
The batch travels to the leader
The request goes to the broker that leads the partition. The producer keeps the batch in memory until it gets a final answer.
The broker checks the stamp
The leader compares the sequence number with what it remembers for this producer and partition. If it is the expected next number, the batch is appended to the log and the memory is updated.
Replication and acknowledgement
The followers copy the batch. Once the required copies exist, the leader sends the acknowledgement back to the producer.
If the reply is lost, the producer retries the same batch
The retry carries the very same stamp as the first attempt. The broker sees a number it has already stored, skips the write, and simply replies “success” again.
The same story as a picture
This picture repeats the failure from Chapter 1, but now with the stamp in place. Compare the two pictures carefully. The only difference is the broker’s ability to recognise the retry.
sequenceDiagram
participant P as Producer with PID 7
participant B as Broker Leader
P->>B: Batch with PID 7 and sequence 0
B->>B: Expected 0. Append and remember 0
B--x P: Acknowledgement is lost
Note over P: Timeout. Retry the same batch.
P->>B: Same batch with PID 7 and sequence 0
B->>B: Already stored 0. Do not append
B->>P: Acknowledgement success
Note over B: The log holds the message once
A bank cashier processes a cheque with serial number 000123. Later someone presents the same cheque again. The cashier does not need to remember your face. The serial number alone shows it was already paid, so the cashier politely says “done” without paying twice.
When the broker detects a duplicate, it does not raise an error to your application. It answers with success, because from the producer’s viewpoint the goal has been reached: the message is safely in the log. Your code stays clean, and the duplicate simply disappears.
7Handling Failure: Brokers Restart, Leaders Change
A single crash should never be allowed to break the promise. Here is why it does not.
Where the memory actually lives
You might worry that the “expected next sequence number” is just sitting in the leader broker’s memory, and that a crash would wipe it out. Kafka avoids this trap by writing the producer state as part of the log itself, and by taking periodic snapshots of that state to disk, alongside the log segments.
Because the state travels inside the replicated log, every in-sync follower already has the same information the leader has. When a leader broker fails, Kafka promotes one of the up-to-date followers to be the new leader. That new leader rebuilds its producer state from the log and the snapshot, and it is ready to keep checking sequence numbers correctly, without any gap.
flowchart TD
A[Leader broker fails] --> B[Kafka controller picks an in-sync follower]
B --> C[Follower becomes the new leader]
C --> D[New leader loads producer state from log and snapshot]
D --> E[Producer IDs, epochs and sequence numbers restored]
E --> F[New leader continues detecting duplicates correctly]
A relay race team keeps a shared logbook of every baton handover, and a copy of that logbook travels with every runner. If the lead runner twists an ankle, the next runner already has the full history and knows exactly which handovers happened. Nothing about the race has to be guessed again.
What happens to an in-flight batch during a crash
If the leader crashes after writing a batch but before replying, the producer will time out and retry. The new leader already has that batch, recognises the sequence number, skips the write, and answers “success”. The producer never learns that anything went wrong underneath it.
The epoch number protects against a stranger case: an old, delayed request from a producer that has since restarted and been given a fresh epoch. The broker compares epochs too, and a batch from an outdated epoch is rejected rather than treated as a legitimate retry. This stops a “zombie” producer instance from writing stale data after a newer instance has taken over.
8Out-of-Order Batches and the Five-Batch Memory
Retries do not always arrive back to back. Kafka is ready for that too.
Why batches can arrive out of order
A producer can have several batches in flight at once for speed. If batch 2 fails and retries while batch 3 has already succeeded, the retried batch 2 might reach the broker after batch 3 has been written. The broker needs a way to accept a late but valid retry, not just the very next number.
The rolling memory of recent batches
To handle this, each broker keeps a short memory, by default the metadata for the last five batches, for every producer and partition pair. If an incoming batch matches one already recorded in that memory, it is recognised as a duplicate and safely skipped, even if it is not the very latest one.
Expected next number
The batch continues the sequence exactly. It is appended normally, and the memory is updated.
A number already seen recently
The batch matches one of the last five remembered batches. It is treated as a duplicate and dropped, with success returned.
A number that skips ahead unexpectedly
The batch is neither the expected next number nor a recent duplicate. This signals a gap, and the broker raises an out-of-order sequence error.
A receptionist keeps the last five visitor badge numbers on a sticky note. If someone tries to check in again with a badge number from that sticky note, the receptionist recognises them instantly, even if two other visitors checked in between the two attempts.
9Configuring the Idempotent Producer
Turning this feature on is simple, but a few related settings deserve attention.
| Setting | Recommended value | Why |
|---|---|---|
| enable.idempotence | true | Switches on producer IDs, epochs and sequence numbers. True by default on modern Kafka. |
| acks | all | Required for idempotence. A message must be fully replicated before it is considered safe. |
| retries | a high number, effectively unlimited | The whole point of idempotence is that retries are now safe, so the producer should keep trying. |
| max.in.flight.requests.per.connection | up to 5 | Idempotence keeps ordering correct even with several batches in flight, up to this limit. |
Older guides warn that idempotence needs max.in.flight.requests.per.connection set to 1. Modern Kafka lifted this limit to 5 because the broker can now check sequence numbers correctly even when several batches are in flight, thanks to the five-batch memory from the previous chapter.
What if you set conflicting values?
If idempotence is enabled but you try to set acks to something weaker than all, or retries to zero, the client library will refuse to start, or will override the value and log a warning, depending on the Kafka version. This protection exists because a weaker setting would quietly break the exactly-once promise while looking like it still works.
The simplest correct setup
For most applications, leaving enable.idempotence at its default of true, together with acks set to all, gives you safe, ordered, duplicate-free writing into each partition with almost no extra thought.
Two settings that shape batching, not correctness
Two other producer settings often come up alongside idempotence, even though they do not affect its correctness. The linger.ms setting controls how long the producer waits before sending a batch, in the hope that more messages arrive and fill it. The batch.size setting caps how large a single batch can grow. Raising linger.ms slightly usually improves throughput, because larger batches are more efficient to send and replicate, at the cost of a small amount of extra delay before a message leaves the producer.
These two settings are worth tuning for performance, but they do not change whether duplicates can occur. That question is answered entirely by enable.idempotence and acks, which is why it helps to think of batching settings and idempotence settings as two separate knobs solving two separate problems.
10Idempotent Producer vs Kafka Transactions
These two features are often confused, but they solve different sized problems.
The idempotent producer guarantees exactly-once writing to a single partition, during one producer session. Kafka transactions build on top of idempotence and extend the guarantee across multiple partitions and multiple topics, and they can also tie a “read, process, write” cycle together atomically, which matters for stream processing applications.
Idempotent Producer
- Scope: one partition, one producer session
- Fixes: duplicates caused by retries
- Setup: one configuration flag
- Overhead: very small
Kafka Transactions
- Scope: many partitions and topics together
- Fixes: all-or-nothing writes across partitions, and exactly-once stream processing
- Setup: transactional ID, begin and commit calls in your code
- Overhead: higher, due to coordination and commit markers
The idempotent producer is like a cashier who never rings up the same receipt twice. A transaction is like a shopper buying five items from five different counters in one mall, where either the entire shopping trip is confirmed together, or none of it counts, even though each counter is independently careful about duplicates.
Transactions do not replace idempotence, they require it. Every transactional producer is automatically an idempotent producer underneath. You can think of transactions as idempotence plus coordination across partitions.
11What Idempotence Does Not Protect Against
This is the most important chapter for avoiding false confidence.
Problem
A team assumes that because idempotence is on, no duplicate can ever appear anywhere in their pipeline.
Why It’s Harmful
Idempotence only removes duplicates created by the producer retrying the same batch. It does nothing about your application calling send twice for two separate reasons, a consumer reprocessing a message after a crash, or two different producer sessions both writing the same logical event.
Correct Approach
Treat idempotence as one safety layer among several. For duplicates from your own application logic, give each logical event a business key, and check for that key before acting, or design the downstream write to be naturally idempotent, such as an upsert by ID rather than an insert.
A new producer session
If your application restarts, it usually gets a new producer ID. Any message it re-sends after restart is treated as new, not as a duplicate of what the old session sent.
Application-level double sends
If your own code accidentally calls send with the same business event twice, both calls look like two different, valid messages to the producer.
Consumer-side duplicates
A consumer that processes a message, then crashes before saving its position, will reread that message after restart. Idempotence on the producer side has no effect here.
Cross-partition ordering
Idempotence keeps order correct within a single partition. It says nothing about the relative order of messages sent to two different partitions.
12Advantages and Trade-offs
Every design decision carries a cost. Here is the honest balance sheet.
Advantages
- Removes duplicate messages caused by producer retries automatically
- Requires only a configuration change, no rewrite of sending code
- Preserves message order within a partition even with several batches in flight
- Survives leader failure because state travels inside the replicated log
- Adds very little latency or throughput cost in most workloads
- Forms the foundation that Kafka transactions are built on
Disadvantages / Trade-offs
- Only protects one producer session; a restart resets the producer identity
- Only protects a single partition, not a group of partitions
- Requires acks set to all, which is slightly slower than weaker acks settings
- Does not remove duplicates created by application logic or consumers
- Adds a small amount of broker-side bookkeeping per producer per partition
13Monitoring an Idempotent Producer
A feature you cannot observe is a feature you cannot trust in production.
record-retry-rate
How often the producer is retrying batches. A rising trend often points to network trouble or an overloaded broker.
record-error-rate
Counts failed sends that could not be resolved through retries. This should normally stay near zero.
request-latency-avg
Average time for a produce request to complete. A sudden increase can signal a struggling broker or network path.
batch-size-avg
Shows how well messages are batching together, which affects overall throughput and efficiency.
Reading the logs
Out-of-order sequence errors, or messages about a producer being fenced by a newer epoch, are worth alerting on. They usually mean either a genuine data-loss risk further upstream, such as an unclean leader election, or a case of two producer instances running with the same identity, such as during a rolling restart that overlapped.
If a topic ever undergoes an unclean leader election, meaning an out-of-sync replica is forced to become leader to restore availability, it may be missing some already-acknowledged messages. In that rare situation, the idempotence guarantee can be broken, because the new leader does not have the full producer history. Treat unclean leader elections as a serious event, not routine background noise.
14Design Patterns and Anti-Patterns
Some habits work beautifully with idempotence. Others quietly undo its benefit.
Pattern: Let the client library manage identity
Never try to hand-roll your own producer ID or sequence tracking. The whole design assumes the broker and the official client are the only two parties managing these numbers.
Pattern: Pair idempotence with a stable message key
Give each logical event a consistent key, such as an order ID, so that related messages land on the same partition in the correct order, which idempotence then keeps duplicate-free.
Pattern: Use transactions when you write to several topics for one event
If one business action produces messages on more than one topic or partition, reach for Kafka transactions rather than trying to coordinate several idempotent producers by hand.
Problem
Running two instances of the same logical producer, using the same transactional or client identity, at the same time, such as during a botched deployment.
Why It’s Harmful
The newer instance fences off the older one using the epoch mechanism, so the older instance starts failing its sends in confusing ways, and messages it was in the middle of sending may be rejected.
Correct Approach
Ensure deployments fully stop the old producer instance before starting a new one, and avoid sharing a single fixed transactional ID across multiple running copies of a service.
Problem
Assuming idempotence means you can remove all business-level deduplication from downstream consumers.
Why It’s Harmful
As covered in Chapter 11, plenty of duplicate sources sit outside the producer-to-broker path, so removing all downstream safety nets reopens exactly the risk idempotence was meant to close.
Correct Approach
Keep a lightweight idempotency check at the consumer or database layer, such as an upsert keyed by a business ID, even when the producer is idempotent.
15Real-World and Industry Examples
Seeing where this matters in practice makes the idea stick.
Payments and banking
A payment event written twice could debit a customer twice. Idempotent producers remove the network-retry source of that risk before it ever reaches the ledger.
E-commerce order pipelines
An order confirmation duplicated by a flaky network could trigger two shipments. Retail platforms rely on this feature to keep order counts trustworthy.
Ride-hailing and logistics
Location and trip-state events are sent constantly under unstable mobile networks, where retries are common and duplicate handling matters for accurate billing and matching.
Streaming analytics
Companies building real-time dashboards and stream processing pipelines rely on idempotent producers as the base layer beneath exactly-once stream processing.
Kafka itself is used at large scale by companies such as LinkedIn, where it originated, along with many banks, telecom operators and e-commerce platforms worldwide. The idempotent producer became especially important as these companies pushed financial and transactional workloads onto Kafka, where a silent duplicate carries real cost.
16Common Mistakes to Avoid
A short checklist of the errors that show up most often in real projects.
Weakening acks by mistake
Setting acks to 1 alongside idempotence quietly weakens the guarantee, because an unreplicated, acknowledged message can be lost if the leader fails.
Assuming a restart continues the same guarantee
Each new producer session usually starts a fresh identity, so protection does not carry over automatically across an application restart.
Ignoring out-of-order sequence errors
These errors are Kafka telling you something unusual happened, such as data loss upstream or overlapping producer instances. They deserve investigation, not silencing.
Confusing idempotence with full end-to-end exactly-once
As Chapter 11 explained, the guarantee stops at the partition boundary. Downstream systems need their own thinking.
Running duplicate producer instances during deployments
Overlapping instances with the same transactional identity cause fencing errors that are confusing to debug under pressure.
17Idempotent Producers Compared to Other Systems
Seeing how other messaging systems solve the same problem sharpens the picture.
| System | How it avoids duplicate writes |
|---|---|
| Kafka idempotent producer | Producer ID, epoch and per-partition sequence numbers checked by the broker |
| RabbitMQ | Publisher confirms tell you a message was received, but detecting a duplicate is left to the application, often via a message ID |
| Amazon SQS standard queue | At-least-once delivery by design; deduplication is the consumer’s responsibility |
| Amazon SQS FIFO queue | Deduplication ID supplied by the sender, checked by the service for a short time window |
Kafka’s approach is like a numbered queue ticket system built into the counter itself. Some other systems hand you a receipt and trust you to notice if you are holding two receipts for the same purchase.
By moving deduplication into the broker rather than leaving it entirely to application code, Kafka reduces the amount of custom logic every team has to write and test correctly on their own.
18A Complete Walkthrough Scenario
Let us tie every idea together with one end-to-end story.
A food delivery company runs an order service that publishes an OrderPlaced event to a Kafka topic called orders, partitioned by restaurant ID. During a lunchtime rush, the network between the order service and the Kafka cluster becomes briefly congested.
The order service starts
It receives a producer ID and epoch from the cluster when it connects, as covered in Chapter 6.
An order event is sent
The event for order 88123 is batched and stamped with a sequence number for the restaurant’s partition.
The acknowledgement is delayed by congestion
The batch was actually written successfully, but the reply is slow enough that the producer’s timeout fires first.
The producer retries automatically
The same stamped batch is sent again, exactly as described in Chapter 6’s picture.
The broker recognises the retry
It sees the same producer ID, epoch and sequence number already stored, skips the write, and replies success.
Order 88123 is written to Kafka exactly once
The kitchen display and billing systems downstream never see a phantom second order, protecting both the customer and the restaurant.
No part of this story required the order service’s own code to check for duplicates. The safety came entirely from acks set to all and idempotence switched on, both of which are configuration, not custom logic.
19Frequently Asked Questions
Quick, direct answers to the questions that come up most often.
Yes, in modern Kafka client versions, enable.idempotence defaults to true, and acks defaults to all when idempotence is on. Older client versions required turning it on explicitly.
The overhead is small. The main cost is that acks must be all, which is somewhat slower than weaker acks settings, but this is usually a worthwhile trade for correctness in important data.
Not by itself. A restarted producer typically receives a new producer ID, so duplicates spanning a restart are not covered. Kafka transactions with a fixed transactional ID address this by recovering the previous identity.
Yes, a single idempotent producer safely tracks separate sequence numbers for every partition it writes to. The guarantee simply applies to each partition individually.
You may see an out-of-order sequence number error, or a producer fenced error if an epoch mismatch occurs. Both are worth investigating rather than ignoring.
No changes are required on the consumer side for idempotent producer duplicates, since the duplicate is removed before it ever reaches the log. Consumers still need their own care for consumer-side duplicates, as explained in Chapter 11.
Yes. The protection is based on the producer ID, epoch and sequence number attached to each batch, not on whether a message key is present. Keyless messages are still tracked per partition, exactly like keyed ones.
For the vast majority of applications, no. The overhead is small, it is the default behaviour in modern Kafka, and the correctness benefit is substantial. Very latency-sensitive, non-critical data streams, such as best-effort metrics where an occasional duplicate truly does not matter, are the rare case where a team might consciously choose weaker settings for a small throughput gain.
20How Long Does the Broker Remember a Producer?
Broker memory is not infinite, so Kafka has rules for when a producer’s history can be safely forgotten.
Producer state is not kept forever
Every producer identity that a broker tracks takes up a small amount of memory and disk space through its recorded state. If Kafka kept every producer’s history permanently, a cluster serving thousands of short-lived producers over months or years would slowly accumulate a huge amount of stale bookkeeping.
To avoid this, brokers expire producer state after a configurable period of inactivity, controlled by a broker setting usually called something like transactional.id.expiration.ms for transactional producers, alongside a related idea for plain idempotent producers. If a producer identity has not sent anything for that long, the broker is free to drop its remembered sequence numbers.
What happens if an old identity reappears
If a producer somehow tries to use a producer ID whose state has already expired and been forgotten, the broker treats it as if it were new. Any built-in duplicate protection tied to the old history is gone, though in practice this is rarely a problem, because a producer that has been silent long enough for expiry to trigger is usually a producer that was already replaced or restarted, and thus received a fresh identity anyway.
A hotel keeps a guest’s key card active only while they are checked in, plus a short grace period. Long after checkout, the same card number can safely be reissued to someone new, because the hotel is confident the earlier guest is not still wandering the halls.
Clusters with a very large number of short-lived producer clients, such as serverless functions that start a fresh producer on every invocation, can put real pressure on broker memory through accumulated producer state. Expiry settings, and application patterns that reuse a long-lived producer instance instead of creating a new one per request, both help keep this manageable.
Snapshots keep recovery fast
Rather than replaying an entire partition’s history from the beginning after every restart, brokers periodically write a snapshot of the current producer state to disk alongside the log segments. On startup, or when a follower becomes a new leader, the broker loads the most recent snapshot and then replays only the small amount of log written after it, which keeps recovery quick even for partitions with a long history.
This same snapshot mechanism is also what makes log retention and idempotence work well together. When old log segments are deleted because they have aged past the topic’s retention period, the relevant producer state has usually already been captured in a snapshot, so removing old data does not silently erase the bookkeeping that newer, still-relevant producers depend on.
21Summary and Key Takeaways
The idempotent producer solves a narrow but painful problem: retries that turn a safe recovery mechanism into a source of duplicate data. It does this with three quiet numbers on every batch, a small amount of broker memory, and one configuration flag on your side. It does not solve every duplicate problem in a system, but it removes one of the most common and hardest to notice, and it forms the base that Kafka’s stronger transactional guarantees are built on.
Key Takeaways
- The core problem — a lost acknowledgement makes a producer retry, and without protection, that retry can create a duplicate message.
- The fix — every batch carries a producer ID, an epoch and a sequence number, so the broker can recognise and skip a repeat.
- The scope — the guarantee covers one partition within one producer session, not the whole pipeline.
- The required setting — enable.idempotence set to true, paired with acks set to all.
- Survives failure — producer state is replicated with the log, so a new leader can keep detecting duplicates correctly.
- Not a full solution — application-level double sends and consumer-side duplicates still need their own handling.
- The natural next step — Kafka transactions extend this same idea across multiple partitions and topics.