An Introduction to Exactly-Once Processing in Kafka
A beginner-friendly guide to the strongest promise Kafka can make about your data: that every event is counted once, and only once, no matter what goes wrong along the way.
Imagine a ticket booth at a busy stadium, where a brief power flicker makes the clerk unsure whether a ticket they just sold was actually recorded. If they record it again just in case, someone might be charged twice or a seat could be double-booked. If they skip recording it to be safe, that sale vanishes from the books entirely. What everyone actually wants is a system where the clerk can always know, with total confidence, that every single ticket sale is recorded exactly once, no matter how many power flickers happen. This is precisely the kind of guarantee Kafka’s exactly-once processing is built to provide. In this guide we will build up the whole picture slowly, defining every new word the first time it appears.
1Why Getting This Exactly Right Is Hard
Before defining exactly-once processing, it helps to understand why it is considered one of the harder problems in distributed systems.
Networks and machines are never perfectly reliable
In any real system built from many independent machines, called a distributed system, things occasionally go wrong. A network connection might briefly drop, a machine might crash and restart, or a message might be delayed unexpectedly. A well-designed system has to keep working correctly even when these kinds of failures happen, not just when everything runs smoothly.
The tricky part is that when something fails partway through an operation, it is often genuinely unclear whether that operation actually completed or not. If a producer, a program sending events into Kafka, sends an event and then the connection drops before it receives a confirmation, did the event actually get saved, or not? The producer cannot always be sure.
Imagine posting a letter through a mail slot and the door closing on your hand right at that exact moment. Did the letter actually make it through, or did it get caught and fall back out? Without looking, you genuinely cannot be sure, and guessing wrong in either direction causes a real problem, either sending a duplicate letter or losing the message entirely.
Why “just retry” is not automatically safe
A natural instinct when unsure whether something worked is to simply try again. But if the original attempt actually did succeed, and you retry anyway, you risk creating a duplicate. This single tension, between not wanting to lose anything and not wanting to duplicate anything, sits at the very heart of why exactly-once processing is genuinely difficult to achieve.
2A Quick Recap: The Three Delivery Promises
Exactly-once is the strongest of three possible promises Kafka can make.
At-most-once, at-least-once, and exactly-once
At-most-once delivery means an event might occasionally be lost, but it will never be processed more than once. At-least-once delivery means an event is never lost, but it might occasionally be processed more than once if something goes wrong and a retry happens. Exactly-once delivery means an event is neither lost nor duplicated, combining the strengths of the other two without either of their weaknesses.
| Guarantee | Can lose an event? | Can duplicate an event? |
|---|---|---|
| At most once | Yes, occasionally | No, never |
| At least once | No, never | Yes, occasionally |
| Exactly once | No, never | No, never |
Achieving exactly-once processing generally requires more coordination and effort than the other two guarantees, which is exactly why Kafka provides specific, dedicated features to make it achievable, rather than leaving every application to solve this hard problem entirely on its own.
3Where Duplicates Actually Come From
Understanding the exact source of duplicates helps make sense of how Kafka prevents them.
A classic duplicate-causing scenario
Imagine a producer sends an event to Kafka. The event is actually saved successfully, but the confirmation message travelling back to the producer gets lost due to a brief network hiccup. From the producer’s point of view, it never received confirmation, so, following ordinary retry logic, it sends the very same event again. Kafka now has two copies of what was meant to be a single event.
sequenceDiagram
participant P as Producer
participant B as Broker
P->>B: Send event
B->>B: Event saved successfully
B--xP: Confirmation lost in transit
P->>B: Retry: send event again
B->>B: Event saved a second time (duplicate)
This is a genuinely common situation
This scenario is not some rare, exotic edge case; it is a completely ordinary consequence of how networks work. Confirmations get lost sometimes, machines restart sometimes, and any system built to handle real-world traffic at scale will eventually run into this exact situation many times over.
Beginners sometimes assume duplicates only happen due to bugs or unusual mistakes. In reality, this kind of duplication is a natural, expected consequence of retrying after an uncertain outcome, and any distributed system without special protection will experience it.
4The Idempotent Producer: Kafka’s First Line of Defence
A feature that solves exactly this kind of accidental duplication, at the producer level.
What “idempotent” means here
Idempotent, in this context, describes an operation that produces the exact same result no matter how many times it is repeated. An idempotent producer is a Kafka producer configured so that even if it retries sending the same event multiple times, Kafka’s brokers will recognise the repeats and only actually save the event once.
How this recognition actually works
When idempotence is enabled, Kafka assigns each producer a unique identifier and attaches an increasing sequence number to every event that producer sends. If a broker receives an event with a sequence number it has already seen from that exact producer, it recognises this as a retry of something already saved, and simply discards the duplicate rather than saving it again.
Think of a numbered raffle ticket. If someone tries to hand in ticket number 47 a second time, the organiser can immediately recognise it has already been recorded and politely decline to count it again, without needing to guess or investigate.
sequenceDiagram
participant P as Idempotent Producer
participant B as Broker
P->>B: Send event, sequence number 47
B->>B: Save event 47
B--xP: Confirmation lost in transit
P->>B: Retry: send event, sequence number 47
B->>B: Recognise sequence 47 already saved, discard duplicate
B->>P: Confirm success
The idempotent producer solves duplication caused specifically by producer retries to a single partition. On its own, it does not yet solve every kind of duplication that can happen across a more complex, multi-step process, which is where transactions come in.
5Beyond a Single Write: The Read-Process-Write Pattern
A very common shape of application that introduces a harder version of the exactly-once problem.
A pattern seen constantly in real systems
Many Kafka applications follow a pattern often called read-process-write: reading events from one or more input topics, doing some processing or transformation on them, and then writing the results out to one or more output topics. This pattern also frequently includes committing an offset, a bookmark of reading progress, to record that the input events have been successfully handled.
Where the real difficulty lies
The idempotent producer alone protects a single write to a single partition from being duplicated by a retry. But a read-process-write application actually needs to coordinate several separate actions together, writing results to possibly multiple output topics and committing an offset, all treated as a single, all-or-nothing unit. If only some of these actions succeed and others fail, the system can end up in an inconsistent state, even with idempotence protecting each individual write.
Think of a bank transferring money between two accounts. Deducting money from one account and adding it to another must happen together, as one all-or-nothing unit. If only the deduction happens, and a crash prevents the addition, money has effectively disappeared, even though the deduction step itself was perfectly correct on its own.
6Kafka Transactions: Grouping Actions Together
Kafka’s answer to coordinating several actions as one reliable unit.
What a transaction actually is
A Kafka transaction lets a producer group several actions together, such as writing to multiple output topics and committing an offset, so that either all of them are made visible together, or none of them are, if something goes wrong partway through. This all-or-nothing behaviour is often described using the word atomic, meaning the group of actions behaves as a single, indivisible unit.
Beginning, working within, and ending a transaction
A producer using transactions first marks the start of a transaction, then performs whatever writes and offset commits belong together as part of that same logical unit of work, and finally either commits the transaction, making everything within it visible together, or aborts it, discarding everything within it as if none of it had ever happened.
flowchart LR
Start["Begin transaction"] --> W1["Write to output topic A"]
W1 --> W2["Write to output topic B"]
W2 --> C["Commit input offset"]
C --> End["Commit transaction"]
If a producer crashes partway through a transaction, before it has been committed, Kafka ensures that transaction’s incomplete writes are never made visible to consumers, protecting the all-or-nothing guarantee even in the face of a genuine failure.
7How Consumers See Transactional Data
A transaction is only meaningful if readers can tell the difference between committed and uncommitted data.
Two ways a consumer can choose to read
Kafka lets a consumer choose its isolation level, a setting controlling which events it is willing to see from transactional writes. Reading uncommitted means the consumer sees every event as soon as it is written, including events belonging to transactions that have not yet been committed, or that might later be aborted entirely. Reading committed means the consumer only ever sees events belonging to transactions that have actually been successfully committed, never anything from an incomplete or aborted transaction.
Why reading committed matters for exactly-once
For exactly-once processing to work meaningfully end to end, downstream consumers generally need to use the read committed isolation level, ensuring they never accidentally process events from a transaction that was later aborted, which would otherwise reintroduce exactly the kind of inconsistency transactions are meant to prevent.
| Isolation Level | What a consumer sees | Fit for exactly-once |
|---|---|---|
| Read uncommitted | All events, including uncommitted or later-aborted ones | Not suitable on its own |
| Read committed | Only events from successfully committed transactions | Required for reliable exactly-once behaviour |
8Combining the Pieces: Exactly-Once End to End
Seeing how idempotence, transactions, and read committed work together as a complete solution.
Events are read from an input topic
An application reads events, tracking exactly which offsets it is about to process as part of this unit of work.
A transaction begins
The application marks the start of a transaction, grouping together everything it is about to do as part of processing this batch.
Results are written idempotently
Processed results are written to one or more output topics, protected against accidental duplication thanks to the idempotent producer working underneath.
The input offset is committed within the same transaction
The application commits its reading progress as part of the very same transaction, rather than as a completely separate action.
The transaction is committed
Everything within the transaction, the output writes and the offset commit together, becomes visible as one atomic unit, or none of it does if something failed first.
Downstream consumers read with read committed
Any application reading the output topics only ever sees fully committed results, never partial or later-aborted work.
9Kafka Streams and Exactly-Once
Where this all becomes especially convenient for real applications.
Built-in support, without manual wiring
Kafka Streams, a library that lets applications process data flowing through Kafka, provides built-in support for exactly-once processing, letting an application simply turn on this guarantee through a configuration setting, rather than needing to manually manage transactions, sequence numbers, and isolation levels by hand.
Why this matters for adoption
Because Kafka Streams applications very commonly follow exactly the read-process-write pattern described earlier, having exactly-once support built directly into the library removes a significant amount of complexity that would otherwise fall on every individual application developer to get right themselves.
Think of a car with built-in anti-lock brakes, rather than a driver needing to manually pump the brake pedal correctly during an emergency stop. The safety mechanism is simply part of the vehicle, ready to work correctly without the driver needing to master a difficult technique themselves.
10Where Exactly-Once Guarantees Actually Apply
An important boundary worth understanding clearly.
Within Kafka, the guarantee is strong
Kafka’s exactly-once guarantees apply reliably to actions happening entirely within Kafka itself, such as reading from a topic, processing, and writing back to another Kafka topic, all coordinated through Kafka’s own transaction mechanism.
Beyond Kafka, extra care is needed
If a system’s final action involves something outside Kafka entirely, such as calling an external web service or updating a database that Kafka has no direct coordination with, achieving true exactly-once behaviour for that final external step generally requires additional design work on the application’s part, since Kafka’s transactions alone cannot coordinate actions happening in a completely separate, external system.
A common and costly mistake is assuming exactly-once automatically extends to every external system an application touches. Careful thought is still required whenever a process reaches beyond Kafka’s own transactional boundary.
11Advantages and Trade-offs
Exactly-once is powerful, but it is not automatically the right choice for every situation.
Advantages
- Removes the need for applications to handle duplicate processing themselves
- Provides strong, precise correctness for counting and financial-style calculations
- Built directly into Kafka and Kafka Streams, rather than requiring custom solutions
- Coordinates reads, writes, and offset commits as one reliable, atomic unit
Disadvantages / Trade-offs
- Adds some additional coordination overhead compared to simpler delivery guarantees
- Only applies fully within Kafka’s own transactional boundary, not automatically to external systems
- Requires downstream consumers to correctly use the read committed isolation level
- Adds conceptual complexity that is unnecessary for data where occasional duplicates are harmless
12Real-World Uses of Exactly-Once Processing
Seeing where this level of precision genuinely matters brings the idea to life.
Financial Transaction Processing
Systems calculating account balances or transferring funds need exactly-once guarantees, since even a single accidental duplicate could mean money being counted twice.
Billing and Invoicing
A billing system tallying usage-based charges needs precise, exactly-once counting to avoid overcharging customers due to accidental duplicate processing.
Inventory and Stock Management
A system tracking exact stock levels needs to avoid both losing updates, which would overstate stock, and duplicating them, which would understate it.
Multi-Stage Stream Processing Pipelines
A chain of Kafka Streams applications, each reading one topic and writing to another, benefits from exactly-once at every stage to avoid errors compounding as data flows through the whole pipeline.
13Best Practices and Common Mistakes
Practical lessons for using exactly-once processing well.
Best practices
Reserve exactly-once for data where it truly matters
Apply this guarantee where duplicates or loss would cause real harm, rather than by default for every single topic.
Always pair transactions with read committed consumers
Ensure downstream applications use the read committed isolation level, or the full benefit of transactions is lost.
Prefer Kafka Streams’ built-in support when applicable
Take advantage of exactly-once being available as a simple configuration setting, rather than manually implementing transactional logic by hand.
Plan carefully for steps outside Kafka
Give special design attention to any point where a process writes to an external system Kafka’s transactions cannot directly coordinate with.
Test failure scenarios deliberately
Verify that a system truly behaves correctly during crashes and retries, rather than only under normal, uninterrupted operation.
Mistakes beginners often make
The idempotent producer alone protects individual writes from duplication, but does not coordinate multiple actions, such as writes and offset commits, together as a single unit. That coordination requires transactions.
Using transactions on the producer side without configuring consumers to read committed data undermines the entire guarantee, since consumers might still see uncommitted or later-aborted data.
Believing exactly-once automatically covers a final write to a database or external service outside Kafka overlooks the real, additional design work needed to coordinate that step safely.
14Frequently Asked Questions
Not always. Many systems work perfectly well with at-least-once delivery, especially when duplicates can be safely handled by the application. Exactly-once is best reserved for situations where duplication or loss would cause genuine harm.
An idempotent producer prevents a single event from being accidentally duplicated due to a retry. A transaction goes further, coordinating multiple actions, such as several writes and an offset commit, together as one atomic, all-or-nothing unit.
Yes. Consumers generally need to be configured with the read committed isolation level, ensuring they only ever see data from transactions that were fully and successfully committed.
It adds some additional coordination overhead compared to simpler delivery guarantees, since more work is required to track sequence numbers and manage transactions, but Kafka is designed to handle this efficiently for the workloads that genuinely need it.
Not automatically. Kafka’s transactions coordinate actions within Kafka itself. Extending exactly-once behaviour to an external system generally requires additional, deliberate design work specific to that system.
No. Applications can use Kafka’s producer and consumer transaction features directly, without Kafka Streams, though Kafka Streams offers this capability in a much simpler, more convenient, built-in way for applications that fit its read-process-write model.
The transaction remains uncommitted, and consumers using read committed will never see its incomplete writes. Depending on the situation, the transaction may later be explicitly aborted, and its writes are treated as if they never happened.
15Summary and Key Takeaways
Exactly-once processing is Kafka’s strongest delivery guarantee, ensuring every event is counted once, and only once, even in the face of network hiccups, retries, and crashes. It is built from two complementary pieces: the idempotent producer, which prevents a single retried write from becoming a duplicate, and transactions, which let multiple actions, such as writes to several topics and an offset commit, be grouped together as one atomic, all-or-nothing unit. For this to work end to end, downstream consumers need to read using the read committed isolation level, and Kafka Streams offers this entire capability as a simple, built-in configuration option. While powerful, exactly-once is best reserved for situations where duplication or loss would cause genuine harm, and its guarantees apply fully only within Kafka’s own transactional boundary, requiring extra care wherever a process reaches beyond it.
Key Takeaways
- Exactly-once is the strongest of Kafka’s delivery guarantees — combining no loss with no duplication.
- Duplicates naturally arise from uncertain retries — a normal, expected consequence of real-world network behaviour.
- The idempotent producer prevents single-write duplication — recognising and discarding retried events using sequence numbers.
- Transactions group multiple actions atomically — coordinating writes and offset commits as one all-or-nothing unit.
- Read committed consumers complete the guarantee — ensuring only fully committed data is ever processed downstream.
- Kafka Streams offers this built in — turned on with a simple configuration setting, rather than manual coordination.
- The guarantee has real boundaries — applying fully within Kafka, but needing extra care for external systems.