Exactly-Once in Kafka Streams (EOS v2)
How a streaming application can read, calculate and write again and again, crash in the middle, and still leave every result in Kafka counted exactly one time, and what that promise does and does not cover.
Imagine a wallet app that adds up payments. A payment of 500 rupees arrives, the application updates the balance, publishes the new balance for other systems, and then the machine it runs on loses power. When the application starts again, it has to answer an awkward question: did I finish that payment or not? If it guesses “not” and processes the payment again, the customer is charged twice. If it guesses “yes” but had not finished, the payment vanishes. Neither answer is acceptable when the numbers are money.
Kafka Streams has a setting that removes the guesswork. When you turn it on, the library makes sure that the reading of the input, the update of the local state, the writing of the output and the saving of the progress marker all succeed together or not at all. The setting is called exactly-once, and its modern form is known as EOS v2. This guide explains it from the very beginning. We start with what Kafka Streams is, walk through the small building blocks that make the promise possible, look closely at how zombies are fenced off, and finish with configuration, monitoring, failure handling and a worked example.
1What Is Exactly-Once in Kafka Streams?
The plain-English idea first. The machinery follows in later chapters.
Kafka Streams is a library that lets an ordinary Java application read records from Kafka topics, transform them, keep running totals, and write results back to Kafka. Because applications and machines can crash at any moment, a streaming job needs a rule for what happens to a record that was being handled when the crash struck. The rule is called a processing guarantee.
Exactly-once semantics, often shortened to EOS, means that the visible effect of processing each input record appears in the output exactly one time, even if the application crashed, restarted, rebalanced or retried along the way. It does not mean that your code runs only once. Your code may well run twice on the same record after a failure. What EOS guarantees is that only one of those runs leaves a permanent mark in Kafka. The other run is cancelled as if it never happened.
Think of a cashier who writes each sale on a scratch pad, then copies the total into the shop’s official ledger and ticks the customer’s name off a list. If the lights go out halfway through, a careless shop might have written the sale in the ledger but not ticked the name, so the sale gets entered again tomorrow. A careful shop uses a rule: nothing on the scratch pad counts until the ledger entry and the tick are made together in one single stroke. If the lights go out before that stroke, the whole scratch pad is thrown away and the sale is done again from the start. EOS is that rule.
What “v2” means
Kafka Streams first offered exactly-once in version 0.11 with a design that used one transactional producer for every task. That design worked but became heavy on large applications. Since Kafka 2.5, a lighter design based on KIP-447 uses one producer for every stream thread, and it is now the recommended one. In the configuration it is called exactly_once_v2. The old settings are deprecated, and recent major Kafka versions have removed them, so new applications should start with v2.
Exactly-once in Kafka Streams means that reading, processing, state changes, writing and progress tracking are committed as one atomic unit, and EOS v2 does it with fewer producers and a smarter way of fencing off old instances.
2Kafka Streams Basics You Need First
A short tour of the parts that exactly-once has to keep in step.
A Kafka Streams program is described as a topology: a graph of steps that read from topics, transform records, join streams, group and count them, and write to other topics. You start the same program on several machines, and Kafka splits the work between them automatically. Every program has a name called the application ID, and all copies that share the name form one logical application.
Task
The smallest piece of work. One task handles one partition of the input, and all the steps of its part of the topology. Tasks never share their state.
Stream thread
A thread that runs one or more tasks. An application instance may run several threads to use several processor cores.
State store
A local key-value store, usually RocksDB, that remembers running totals, join data or windows for a task.
Changelog topic
Every change to a state store is also written to a Kafka topic. If the machine is lost, the store is rebuilt by replaying it.
Repartition topic
An internal topic used when records must be re-keyed and sent to a different partition, for example before a count by customer.
Standby replica
An optional warm copy of a state store kept on another instance, so that a failed task can restart quickly.
Committed offset
A bookmark stored in Kafka that says how far the application has read in each input partition.
Consumer group
Streams uses a consumer group behind the scenes to share partitions among threads, and to move them when a thread leaves.
flowchart LR IN["Input Topic
partitions 0 to 3"] --> T0 IN --> T1 subgraph APP["Application Instance"] subgraph TH["Stream Thread 1"] T0["Task 0
partition 0"] T1["Task 1
partition 1"] end S0[("State Store 0")] S1[("State Store 1")] T0 --> S0 T1 --> S1 end S0 -.->|"changes copied"| CL["Changelog Topic"] S1 -.->|"changes copied"| CL T0 --> OUT["Output Topic"] T1 --> OUT
Why the picture matters for exactly-once
Look at Figure 1 and count the places where the application changes something permanent. It writes to the output topic. It writes to the changelog topic. It also stores its progress as a committed offset. That is three separate writes, and they go to different partitions and possibly to different brokers. If a crash strikes between them, the system is left in a half-finished state. The whole purpose of exactly-once is to make those separate writes behave as one.
3Processing Guarantees: At-Most-Once, At-Least-Once, Exactly-Once
Three promises, each with a different price and a different failure mode.
Every messaging system must decide what to do when it does not know whether a record was handled. There are three honest answers, and each one appears in real systems.
| Guarantee | What happens after a crash | Risk | Typical use |
|---|---|---|---|
| At-most-once | The bookmark is saved first, then the record is processed. A crash in between skips the record. | Lost records | Rough metrics where a gap is harmless |
| At-least-once | The record is processed first, then the bookmark is saved. A crash in between repeats the record. | Duplicate results | Idempotent or tolerant pipelines |
| Exactly-once | Processing, output and bookmark are saved together or not at all. | Extra latency and complexity | Counters, balances, billing, inventory |
Kafka Streams uses at-least-once as its default. It is fast and simple, and for many jobs duplicates are harmless. For example, writing the same latest value for the same key twice does no damage. But for anything that adds, subtracts or counts, a duplicate changes the answer. A duplicated payment doubles a balance, and a duplicated click inflates a report.
Picture a post office clerk who sorts letters. At-most-once is a clerk who puts a “done” stamp on a letter before delivering it, and if the clerk faints, that letter is lost. At-least-once is a clerk who delivers first and stamps afterwards, so a fainting spell may cause the same letter to be delivered a second time. Exactly-once is a clerk who does both the delivery and the stamp inside one sealed step that cannot be seen half finished.
Where duplicates really come from
It helps to name the three separate sources of repeated or lost work in a stream job, because EOS attacks each one with a different tool.
- Producer retries. A network glitch can make the producer resend a record that the broker had already stored, creating a duplicate in the output topic.
- Reprocessing after a crash. The application restarts from its last saved bookmark and processes records whose results were already published.
- Zombie instances. An old instance that the group thinks is dead may still be running and writing, while a new instance has taken over the same work.
The idempotent producer removes duplicates caused by retries. Transactions remove the half-finished state caused by crashes. Fencing removes the damage done by zombies. Exactly-once in Kafka Streams is these three ideas, working together and configured for you.
4The Read-Process-Write Cycle and Where It Breaks
A careful look at the small window in which a crash causes trouble.
A Kafka Streams task repeats one loop again and again. It reads a batch of records from its input partitions. It processes them, which may update state stores. It writes the results to output topics and writes state changes to the changelog. From time to time, it commits its progress by saving the offsets of the records it has finished. The gap between the last write and the commit is the dangerous moment.
sequenceDiagram autonumber participant IN as Input Topic participant ST as Streams Task participant OUT as Output Topic participant OF as Committed Offsets IN->>ST: Records at offsets 100 to 109 ST->>ST: Update state, balance changes ST->>OUT: Write 10 result records Note over ST: Crash before the commit ST--xOF: Offsets never saved Note over ST: Restart from the old bookmark, offset 100 IN->>ST: Same records at offsets 100 to 109 ST->>OUT: Write the same 10 results again Note over OUT: Duplicates now exist in the output
The problem is not only the output
Duplicate output is the visible symptom, but state is hurt as well. Suppose the task adds 500 to an account balance in its local store, and the store is backed up to the changelog topic. After the crash, the task restarts from the old bookmark and adds the 500 again on top of a store that already contains the first addition. Now the balance is wrong, and it will stay wrong, because the wrong value keeps flowing into every later calculation. This is why a simple “deduplicate the output” trick is not enough for stateful jobs.
The task reads a batch
Input records for its partitions arrive, and the task remembers the offsets of the last one.
The task changes its state
Counters and totals are updated in the local store, and the same changes are sent to the changelog topic.
The task publishes results
Output records are sent to one or more topics, which may live on different brokers.
The task commits progress
The input offsets are saved. Only now does the system consider the batch finished.
Exactly-once treats these four steps as one unit. If anything goes wrong before step 4 is complete, steps 2 and 3 are cancelled together. Their records still sit in the topics, but they are marked as aborted, so no properly configured reader will ever see them.
5Data Flow and Lifecycle: One Record Through a Multi-Stage Topology
Real topologies have several stages. Exactly-once holds across all of them, but it is built stage by stage.
Most Kafka Streams programs are more than one step. A typical program reads events, re-keys them so that related events meet in the same partition, aggregates them, and writes a result. When the key changes, Streams cuts the topology into sub-topologies and connects them with an internal repartition topic. Each sub-topology has its own tasks, its own threads’ transactions and its own commit cycle. To understand what exactly-once gives you end to end, follow one record all the way through.
flowchart LR SRC["Source App
writes payments"] --> IN["Input Topic"] IN --> S1["Sub-topology 1
validate and re-key by account"] S1 --> RP["Repartition Topic
internal"] RP --> S2["Sub-topology 2
aggregate balance in a state store"] S2 --> CL["Changelog Topic"] S2 --> OUT["Output Topic
balances"] OUT --> RD["Downstream Reader
read_committed"]
The source writes the record
An upstream application publishes a payment event to the input topic. This step is outside our stream application and has its own guarantee, for example an idempotent producer.
Stage one reads and re-keys
A task in the first sub-topology reads the event with read-committed, validates it and changes its key to the account number. It writes the result to the repartition topic inside a transaction, together with its own input offset.
Stage one commits
When its transaction commits, the re-keyed record becomes visible in the repartition topic, and the input offset moves forward. If it aborts, neither happens.
Stage two reads the re-keyed record
A task in the second sub-topology reads the repartition topic with read-committed, so it only ever sees committed records from stage one.
Stage two updates state and writes output
It updates the balance in the state store, and sends the change to the changelog and the new balance to the output topic, together with its own offsets, in one transaction.
Readers see one committed result
A downstream reader with read-committed receives the balance once, after stage two commits.
Exactly-once is a chain of small promises
Each stage promises that its inputs are consumed once and its outputs appear once. Because the next stage only reads committed output, the promises link together like the links of a chain, and the whole pipeline behaves as if every event moved through it once. The chain is only as strong as its weakest link, so a single stage with a different setting, or a single reader with the wrong isolation level, breaks the end-to-end story.
Picture a relay race in which each runner may only take the baton when the previous runner has stopped in the handover box and the referee has waved a flag. If a runner stumbles and has to start their leg again, the next runner never saw the attempt that failed, because the flag was never waved. The baton is handed over exactly once per leg.
Each stage waits for its own commit before the next stage can see the result. A topology with three stages and a 100 millisecond commit interval can add roughly 300 milliseconds to the path of a record, before counting processing time and network delays.
6Building Block One: The Idempotent Producer
The first tool removes duplicates that come from simple retries.
A producer sends records to a broker and waits for an acknowledgement. If the acknowledgement is lost on the network, the producer cannot tell whether the broker stored the record. A careful producer retries. Without protection, the retry might store the record twice. The idempotent producer solves this. The word idempotent means that doing an action many times has the same effect as doing it once.
How it works
When an idempotent producer starts, the broker gives it a unique number called a producer ID. The producer then numbers every batch it sends to each partition with a growing sequence number. The broker remembers the latest sequence numbers for each producer and partition. If a batch arrives with a number the broker has already stored, the broker simply acknowledges it again without writing a second copy. If a batch arrives with a number that skips ahead, the broker reports an error, because something was lost. This also keeps the order of records intact when retries happen.
Think of numbered cheque leaves. A bank that sees cheque number 57 twice knows it is the same cheque presented twice, and pays it only once. A bank that sees number 59 right after 57 knows that 58 is missing and asks about it. The producer ID is the account holder, and the sequence numbers are the cheque numbers.
What it does not do
What it gives you
- No duplicates from network retries inside one producer session.
- Correct ordering of records per partition, even with retries.
- Low overhead, and it is on by default in modern clients.
What it does not give you
- No protection when the application restarts, because a new producer gets a new ID.
- No protection for writes to several partitions or topics at once.
- No link between the output and the input bookmark.
Because of these limits, idempotence alone cannot give exactly-once for a whole read-process-write loop. It is the foundation, not the house. The next building block adds the missing walls.
7Building Block Two: Kafka Transactions
Transactions let a producer write to many partitions and topics and have all writes count together, or none at all.
A Kafka transaction is a group of writes that become visible at the same moment. A producer opens a transaction, sends records to as many topics and partitions as it needs, and then either commits or aborts. If it commits, every record in the group becomes readable. If it aborts, none of them do. The producer can even include the input offsets of a consumer in the same group, which is the trick that ties reading to writing.
The cast of characters
Transactional ID
A name chosen by the application that identifies one logical producer across restarts. Kafka uses it to find the right coordinator and to detect old instances.
Producer ID and epoch
A number handed out by the broker, plus a version counter called the epoch that grows each time the same transactional ID starts again.
Transaction coordinator
A broker that manages the transactions of a set of transactional IDs. It keeps track of which partitions each transaction touches and decides when it is finished.
Transaction log
An internal replicated topic called __transaction_state. The coordinator writes every step of every transaction here, so that a new coordinator can take over after a failure.
Control records
Special commit or abort markers written into each partition that took part. Readers use them to know how each transaction ended.
Group coordinator
The broker that stores consumer offsets. In a transaction, the offsets are held back until the transaction commits.
sequenceDiagram autonumber participant P as Producer participant TC as Transaction Coordinator participant TL as Transaction Log participant OP as Output Partition Leader participant GC as Group Coordinator P->>TC: Init, register transactional id TC-->>P: Producer id and epoch Note over P: Begin transaction on the client P->>TC: Add output partition to the transaction TC->>TL: Record the partition P->>OP: Send records marked transactional P->>TC: Add the offsets partition to the transaction P->>GC: Send consumer offsets, held as pending P->>TC: End transaction with commit TC->>TL: Write prepare commit TC-->>P: Commit is decided TC->>OP: Write commit marker TC->>GC: Write commit marker TC->>TL: Write complete commit
Two-phase commit in plain words
Committing to several places at once is a classic hard problem, and Kafka solves it with a two-phase approach. In the first phase, the coordinator writes a “prepare commit” entry into its replicated log. From this moment the outcome is settled: even if the coordinator crashes, a new coordinator will read the log and finish the job. In the second phase, the coordinator writes a commit marker into every partition that took part, including the internal partition that holds consumer offsets. When all markers are written, the coordinator records “complete” and forgets the transaction.
Think of a wedding registrar who has several witnesses sign several registers. First the registrar writes in a locked diary, “This marriage is decided.” Then the registrar walks around and stamps every register. If the registrar faints halfway, a colleague opens the diary, sees the decision, and finishes the stamping. Nobody has to ask the couple to start over.
What happens if a transaction never finishes
Every transaction has a time limit, set by transaction.timeout.ms. If the producer neither commits nor aborts within that time, the coordinator aborts the transaction on its own and writes abort markers. This limit is what stops a crashed producer from holding partitions in an undecided state forever. Later chapters show how this limit affects how quickly readers can see new data after a crash.
8Internal Working: The Transaction Coordinator and Its Log
A look behind the curtain at the referee that makes atomic commits possible.
Every transactional producer talks to one broker that acts as its transaction coordinator. Kafka chooses the coordinator with a simple rule. The transactional ID is turned into a number by a hash function, and that number selects one partition of the internal __transaction_state topic. The broker that currently leads that partition is the coordinator for every transactional ID that maps to it. By default the log has 50 partitions, so the coordinator duty is spread over many brokers, and it moves automatically whenever leadership of a partition moves.
What the coordinator remembers
For each transactional ID, the coordinator keeps a small record. It holds the current producer ID and epoch, the timeout, the current state of the transaction, and the list of partitions that the running transaction has written to. Every change to this record is first written to the transaction log, which is replicated like any other topic. That is why a new coordinator can rebuild everything after a failure: it simply reads the partition of the log that it now leads.
stateDiagram-v2 [*] --> Empty Empty --> Ongoing: First partition added Ongoing --> Ongoing: More partitions added Ongoing --> PrepareCommit: Producer asks to commit Ongoing --> PrepareAbort: Producer aborts or timeout PrepareCommit --> CompleteCommit: All commit markers written PrepareAbort --> CompleteAbort: All abort markers written CompleteCommit --> Empty: Ready for next transaction CompleteAbort --> Empty: Ready for next transaction Empty --> Dead: Transactional id expired
| State | Meaning | What can happen next |
|---|---|---|
| Empty | No transaction is running for this ID | A new transaction begins with the first write |
| Ongoing | The producer is writing to partitions | Commit, abort, or timeout |
| Prepare commit | The decision to commit is logged | Markers are written to all partitions |
| Prepare abort | The decision to abort is logged | Abort markers are written to all partitions |
| Complete commit or abort | All markers are in place | The record goes back to empty |
| Dead | The ID has expired after a long silence | It can be created again from scratch |
Writing the markers
After the decision is logged, the coordinator must deliver the commit or abort marker to every partition in the transaction. It does this by sending marker requests to the leaders of those partitions, often grouped by broker. Each leader appends a special control record to its partition. Only when all leaders have confirmed does the coordinator write the final complete entry. If a leader is temporarily unavailable, the coordinator keeps retrying, and the transaction stays in its prepare state until the marker lands.
What happens when the coordinator fails
If the broker that acts as coordinator stops, the transaction log partition it led is taken over by another broker. The new leader reads the log and rebuilds the records. Transactions that were ongoing simply continue, because the producer will discover the new coordinator and carry on. Transactions that were in the prepare state are finished by the new coordinator, which completes writing the markers. In both cases, no decided outcome is ever changed or forgotten.
The coordinator is like a court clerk who writes every step of a case in a book that is copied to several offices. If the clerk falls ill, a colleague reads the book and picks up exactly where the case stood. A verdict that was already written cannot be lost, and a case that was still being heard simply goes on.
Producer epochs and expiry
Each time a transactional ID initializes, the coordinator raises its epoch, which is how older producers with the same ID are shut out. The epoch is a small number, so Kafka also has rules for when it must move to a new producer ID. Transactional IDs that have not been used for a long time, seven days by default, are removed from the log’s records. That is safe, because a returning producer will simply be given a fresh identity, but it means that very long pauses in an application can lead to a reset of its transactional state.
9Building Block Three: read_committed Consumers
Atomic writes are useless if readers can still see half of them. The isolation level fixes that.
Records from a transaction are written into partitions immediately, mixed with records from other producers. They are in the log even before the transaction has ended. A reader therefore needs a way to decide whether to show them. That decision is controlled by the consumer setting isolation.level. The default, read_uncommitted, shows every record below the high watermark, including those from open and aborted transactions. The other choice, read_committed, shows only records from committed transactions, plus records that were never part of a transaction.
The last stable offset
Each partition has a number called the last stable offset, or LSO. It is the lower of two values: the high watermark, and the offset of the first record that belongs to a transaction that is still open. A consumer in read-committed mode is never sent anything above the LSO. This means that one open transaction can pause the visibility of everything written after it, even records from other producers that have already been committed. It is a small price for a simple and safe rule.
flowchart TD R["Consumer requests records
with read_committed"] --> Q1{"Is the record offset
below the last stable offset?"} Q1 -->|"No"| W["Not sent yet
an open transaction is in the way"] Q1 -->|"Yes"| Q2{"Is it a control record
commit or abort marker?"} Q2 -->|"Yes"| H["Hidden from the application"] Q2 -->|"No"| Q3{"Does it belong to an
aborted transaction?"} Q3 -->|"Yes"| D["Dropped by the client"] Q3 -->|"No"| OK["Delivered to the application"]
How the client learns which records to drop
Along with the records, the broker sends a small list of aborted transaction ranges. For each aborted transaction the list says which producer wrote it and at which offset it began. The client matches records against that list, skips the ones that belong to aborted ranges, and never shows the control markers to the application. All of this is invisible to the code that calls poll.
The promise is kept inside Kafka, but readers must play their part. A downstream consumer that uses the default read-uncommitted mode will see records that were later aborted, and will count them. Every consumer of an exactly-once output topic should use read-committed, or the guarantee is lost at the last step. Kafka Streams sets this for you inside your topology, but other applications are on their own.
Production example: the dashboard that counted twice
A team turns on exactly-once for a stream job that computes order totals, and the totals look perfect in the stream job’s own downstream topics. Yet a separate reporting service, which reads the same output topic with default settings, shows totals that are a little too high after each restart of the stream job. The reporting service was reading aborted records. Setting its isolation level to read-committed fixed the numbers. The lesson: check every reader of an exactly-once topic.
10Atomic Commit of Offsets, Output and State
This is the heart of exactly-once: the input bookmark is written inside the same transaction as the results.
In ordinary at-least-once mode, a Streams task commits its bookmark with the consumer’s own commit call, and it writes output with the producer, as two independent actions. With exactly-once, that changes. The bookmark is no longer committed by the consumer. Instead, the producer sends the offsets to the group coordinator as part of its transaction, using a call named sendOffsetsToTransaction. The group coordinator keeps them as pending until the transaction ends. If the transaction commits, they become real. If it aborts, they are discarded, and the bookmark stays at its old value.
| Write | Destination | Inside the transaction? |
|---|---|---|
| Result records | Output topics | Yes |
| State changes | Changelog topics | Yes |
| Re-keyed records | Repartition topics | Yes |
| Input progress | The consumer offsets topic | Yes, sent through the producer |
| Local state on disk | The instance’s own disk | No, but it is rebuilt from the changelog if needed |
Look at the table. Everything that lives in Kafka is inside the transaction. The one exception is the local copy of the state on disk, which is only a cache of what the changelog contains. Because the changelog is inside the transaction, the truth about the state is always consistent with the outputs and the bookmark. If a transaction aborts, the local copy may be a step ahead of the truth, and Streams knows how to repair it, as later chapters explain.
Why this removes duplicates and lost work
Consider the two dangerous moments from before. If the task crashes after writing outputs but before committing the bookmark, the transaction is never committed, so the outputs are aborted and invisible, and the bookmark still points to the start of the batch. The new run processes the batch again, and only its results become visible. If the task crashes after the commit, everything is already safe and the next run starts from the new bookmark. There is no moment in which outputs are committed and the bookmark is not, or the other way round.
A bank transfer works only because the money leaving one account and the money arriving in another are recorded in one action. A bank that removed money from the first account, then crashed before crediting the second, would have lost money. EOS gives stream processing the same safety: the results and the bookmark are moved together.
11Joins, Aggregations and Windows Under Exactly-Once
Some operations are harmed badly by duplicates, and others hardly notice. Knowing which is which shows where exactly-once matters most.
Kafka Streams offers many operations, and they react very differently to a repeated record. An operation is idempotent when applying it twice gives the same result as applying it once. Filtering a record twice, or writing the latest value for a key twice, changes nothing. An operation that adds, counts or appends is not idempotent. Every repeated record shifts the answer.
| Operation | Idempotent? | Effect of a duplicate under at-least-once |
|---|---|---|
| Map, filter, flat map | Yes, if the logic is deterministic | A duplicate output record, harmless if readers deduplicate |
| Count and sum aggregations | No | The count or sum is too high, permanently |
| Reduce with “keep latest” | Yes | The same latest value is stored again |
| Windowed counts | No | The count inside the window is too high |
| Stream-stream joins | No | A duplicated input can produce duplicated join results |
| Table-table and foreign-key joins | Mostly yes | The same updated row is emitted again |
| Suppress, emitting a final window result | Depends on the sink | A final result may be emitted twice |
Aggregations: the classic victim
A count per customer is the simplest example. Every event should add one. If a crash makes the task process a batch twice, the store adds the batch a second time and the count is wrong. Nothing will ever correct it later, because later events only add to the wrong number. Exactly-once keeps the state and the input bookmark in step, so that the second attempt starts from the state as it was before the first attempt, not after it.
Joins and windows
A stream-stream join keeps recent events from both sides in windowed stores and matches them by key and time. A duplicated event can match again and create a second copy of a joined result. Windowed aggregations behave like counts, but with a time limit on how long the window accepts updates, which is called the grace period. Exactly-once does not change these time rules. It only ensures that inside those rules, each event is applied once.
Emitting final results
Many applications want only the final answer for a window, not every intermediate update. The suppress operator holds results in a buffer until the window closes. Its buffer is backed by a changelog topic just like other state, so under exactly-once it survives a crash without emitting early or twice. The final result then appears once in the output topic, visible to read-committed readers after the commit.
Counting votes shows the difference. If a counting clerk reads the same ballot twice, the tally is wrong forever, because tallying is adding. If a clerk writes a voter’s latest choice on a board twice, the board shows the same thing as before. Exactly-once acts like a supervisor who makes sure no ballot is added to the tally more than once, even when clerks are replaced in the middle of the night.
If your topology contains only idempotent operations, and your readers can accept an occasional repeat, the default mode may be enough. The moment you count, sum, join streams or emit final window results, and the numbers matter, exactly-once earns its cost.
12Zombies and Fencing
The hardest part of exactly-once is not a crash. It is an instance that should have stopped but has not.
A zombie is an application instance that the rest of the system believes is gone, but which is still alive and still writing. Zombies appear in ordinary life. A long garbage-collection pause can freeze a thread for many seconds. A network cut can isolate a machine while its process keeps running. During that time, the consumer group notices missing heartbeats, declares the instance dead and gives its partitions to another instance. Then the frozen instance wakes up, unaware of what happened, and carries on writing results for partitions it no longer owns. Without protection, both the old and the new owner would write outputs for the same input, and the result would be duplicates or even wrong state.
Fencing in the first design
In the original exactly-once design, each task had its own producer with a transactional ID built from the application ID and the task ID. When a task moved to a new machine, the new producer registered the same transactional ID. The coordinator then raised the epoch for that ID, and from that moment any request from the old producer, carrying the older epoch, was rejected with a fenced error. The zombie was cut off by the epoch, like an old key that no longer opens a changed lock.
Fencing in EOS v2
The newer design uses one producer per stream thread, and a thread runs many tasks that can change over time. A fixed link between one transactional ID and one task no longer exists, so a different guard is needed. The guard is the consumer group itself. When the producer sends the consumer offsets in a transaction, it also sends the group metadata: the group’s current generation number and the member’s identity. The group coordinator checks them. If the generation is old, because a rebalance has happened since, the offsets are refused. A transaction cannot commit without its offsets, so a zombie’s transaction can never commit.
sequenceDiagram autonumber participant Z as Old Thread, the zombie participant GC as Group Coordinator participant N as New Thread participant TC as Transaction Coordinator Note over Z: Long pause, heartbeats stop GC->>GC: Member seems dead, start a new generation GC-->>N: Partitions now belong to the new thread N->>TC: Begin work and send results in a transaction Note over Z: Wakes up, still thinks it owns the partitions Z->>TC: Send its own results in a transaction Z->>GC: Send offsets with the old generation GC-->>Z: Refused, generation is out of date Z->>TC: Abort the transaction N->>GC: Send offsets with the current generation GC-->>N: Accepted, the transaction commits
The zombie may be allowed to write, but it is never allowed to commit. Written-but-uncommitted records are aborted and hidden from read-committed readers.
What the application sees
When Streams detects that it has been fenced, it raises an internal signal that the task has moved to another owner. The affected tasks are closed without committing, their open transactions are aborted, and the thread joins the group again with fresh information. This is not a bug in the application, but a normal and expected recovery step. You will see such events in the logs after long pauses or rebalances.
13EOS v1 Versus EOS v2
Why the community redesigned the mechanism and what changes for you.
| Aspect | First design | EOS v2 |
|---|---|---|
| Config name | exactly_once (deprecated, removed in recent major versions) | exactly_once_v2 |
| Producers | One for every task | One for every stream thread |
| Transactional ID | Application ID plus task ID | Application ID plus a process identifier plus the thread number |
| How zombies are stopped | Epoch bump for the task’s transactional ID | Group generation check when offsets are committed |
| Transactions per commit interval | One for every task | One for every thread |
| Connections and memory | Grow with the number of tasks | Grow with the number of threads |
| Broker version needed | 0.11 or newer | 2.5 or newer |
| Best for | Old deployments only | All new applications |
The scaling problem v2 solved
Imagine an application that reads a topic with 200 partitions. That means 200 tasks. In the first design, this required 200 producers, each with its own network connection, its own buffers, its own transactional ID and its own transaction every commit interval. The transaction coordinators had to handle thousands of tiny transactions each second, and each of them wrote markers to several partitions. With v2, if the same application runs on ten threads, it needs only ten producers and ten transactions per interval, each covering many tasks. The work per record stays the same, but the overhead of managing transactions drops sharply.
The guarantee is the same. Both designs commit output, state and progress atomically and hide aborted work from readers. The change is about scalability and about the way zombies are stopped, not about what you can promise to your users.
14Inside a Streams Commit Cycle, Step by Step
Everything you have learned so far, in the order it happens inside one stream thread.
The thread begins a transaction
The transaction is opened lazily on the client. No message needs to reach the broker until the first write.
The thread polls with read-committed
It fetches records for all its tasks, seeing only committed data from upstream topics.
Each task processes its records
Operators run, state stores are updated, and the producer sends outputs and changelog records. The first write to each new partition registers that partition with the transaction coordinator.
The commit interval expires
By default this is every 100 milliseconds under exactly-once. The thread stops taking new records for a moment and prepares to commit.
Stores and producer are flushed
State stores flush their caches, which sends the final changelog records, and the producer flushes so that every write has been acknowledged before the commit.
Offsets join the transaction
The thread sends the input offsets for all its tasks, together with the group metadata, to the group coordinator as part of the transaction.
The transaction commits
The coordinator writes prepare-commit, then commit markers into every partition touched. Output and changelog records become visible, and the offsets become real.
The next transaction starts
The thread goes straight back to polling. If there was no new data, it may skip the commit, so idle threads do not create empty transactions.
flowchart LR A["Poll
read_committed"] --> B["Process records
update state stores"] B --> C["Send outputs and
changelog records"] C --> D{"Commit interval
reached?"} D -->|"No"| A D -->|"Yes"| E["Flush stores and producer"] E --> F["Send offsets with
group metadata"] F --> G["Commit the transaction
markers written"] G --> A B -.->|"error or fenced"| X["Abort transaction
restart from last committed offsets"] X --> A
The commit interval is the size of your batch
Each transaction covers everything that happened during one commit interval. A short interval means small transactions, quick visibility of results and little work to redo after a failure, but more transactions per second. A long interval means fewer, larger transactions and better throughput, but results become visible later and a failure throws away more work. The default of 100 milliseconds under exactly-once favours low delay, and many teams raise it when they measure heavy coordinator load.
Under exactly-once, results are invisible to read-committed readers until the commit. So the end-to-end delay of a pipeline can never be lower than the commit interval of every exactly-once stage along the way. A pipeline of four such stages adds the delays of all four. Plan for this when you promise a latency to your users.
15State Stores, Changelogs and Recovery Under EOS
The local database is fast but fragile. The changelog topic is the source of truth that makes it trustworthy.
Stateful operations such as counts, sums, joins and windows keep their data in a local state store, usually a RocksDB database on the disk of the machine that runs the task. Local access is very fast, but a disk can fail and a task can move. So every change to a store is also sent to a changelog topic in Kafka. If the local copy is lost, Streams rebuilds it by reading the changelog from the beginning.
Under exactly-once there is one extra problem. A task updates its local store while it processes records, which is before the transaction commits. If the transaction later aborts, the local store now holds changes that Kafka considers cancelled. The store is a step ahead of the truth. Streams handles this by treating the local store as a cache that must never be trusted after an unclean stop.
flowchart TD
S["Task starts on an instance"] --> Q1{"Was the task closed cleanly
and a checkpoint written?"}
Q1 -->|"Yes"| A["Keep the local store
read the changelog from the checkpoint onward"]
Q1 -->|"No"| B["Delete the local store
the store may hold uncommitted changes"]
B --> C["Read the whole changelog
with read_committed"]
A --> D["Store is now equal to committed state"]
C --> D
D --> E["Resume processing from the committed input offsets"]
The checkpoint file
When a task shuts down cleanly, it writes a small checkpoint file that records how far the local store matches the changelog. On the next start, Streams reads the file and only needs to replay the newer part of the changelog. If the task stopped abruptly, there is no checkpoint, so Streams deletes the local store and rebuilds it from the committed part of the changelog. Because the rebuild reads with the read-committed isolation level, it never picks up changes from aborted transactions.
A cook keeps a small notepad of today’s orders at the stove, and the restaurant’s official order book is kept at the front desk. If the kitchen catches fire in the middle of a busy hour, nobody trusts the half-burnt notepad. The cook goes back to the front desk, copies the official book onto a new notepad, and carries on from the last order that the desk had confirmed.
Making recovery fast
Rebuilding a large store can take minutes or hours. Streams offers two tools to shorten the wait. Standby replicas, controlled by num.standby.replicas, keep a warm copy of the store on another instance by continuously reading the changelog. If the active task fails, the standby takes over almost at once. Warm-up replicas help when an instance is added or replaced: the new instance first catches up in the background, and the task moves only when the copy is nearly current. Both use extra disk and network, so choose them for the state that is worth protecting.
| Store size | Restore speed (illustrative) | Time without standby | Time with standby |
|---|---|---|---|
| 1 GB | 100 MB per second | About 10 seconds | Near instant |
| 20 GB | 100 MB per second | About 200 seconds | Near instant |
| 200 GB | 100 MB per second | About 33 minutes | Near instant |
Changelog topics are compacted, which means Kafka keeps only the latest value for each key over time. So the changelog does not grow forever, and a rebuild reads roughly the size of the live state, not the whole history of every update.
16What Exactly-Once Does Not Cover
The guarantee has a border. Knowing where it ends is the most important lesson in this guide.
Exactly-once in Kafka Streams is a promise about what happens inside one Kafka cluster. Everything that is read from Kafka topics and written to Kafka topics, including the internal ones, can be committed atomically. Anything outside that circle is not part of the transaction, and Kafka cannot cancel it.
flowchart LR
subgraph IN["Inside the guarantee"]
A["Input Topics"] --> B["Kafka Streams Application"]
B --> C["Output Topics"]
B --> D["Changelog and Repartition Topics"]
B --> E["Committed Offsets"]
end
subgraph OUT["Outside the guarantee"]
F["External Database"]
G["Email or SMS Service"]
H["HTTP API of a Partner"]
I["A Second Kafka Cluster"]
end
B -.->|"side effect, can repeat"| F
B -.->|"side effect, can repeat"| G
B -.->|"side effect, can repeat"| H
C -.->|"copy, separate guarantee"| I
Side effects can happen twice
Suppose a processor sends an email or calls a payment provider inside its logic. If the transaction aborts and the record is processed again, the email is sent again, because nobody can take back an email. The output topic will still contain each result exactly once, but the outside world has been told twice. The same holds for writes to a database, calls to a REST service, and even log lines or metrics. Anything you do that is not a Kafka write is repeated whenever the work is repeated.
How to handle the outside world
Write to Kafka, act later
Let the stream job write a result topic. A separate consumer reads it with read-committed and performs the side effect, using a unique event ID so that repeats can be recognized.
Idempotent upserts
When writing to a database, use the event’s key or ID to update a row, so that applying the same event twice gives the same result as once.
Idempotency keys for APIs
Many payment and messaging services accept a unique request key and will ignore a repeated request. Send the event ID as that key.
Connectors for sinks
Use Kafka Connect sink connectors that support idempotent or transactional writes into the target system, instead of writing the sink code yourself.
Other limits to remember
- Interactive queries. Reading a state store directly from your application reads the local copy, which can be a step ahead of the committed truth. Treat such reads as fresh but not final.
- Bad data and bugs. Exactly-once processes a poison record exactly once. It does not fix a program that computes the wrong answer or a record that cannot be parsed.
- Lost brokers data. If brokers lose committed data, for instance through unclean leader election or too few in-sync replicas, the promise built on top of that data breaks. Use a replication factor of 3, a minimum in-sync replica count of 2, and unclean election off for every topic that the application uses.
- Readers who do not cooperate. Consumers that read the output with read-uncommitted can see aborted records.
- Time-based logic. Windows, grace periods and wall-clock triggers behave as they always do. Exactly-once does not change how late records are treated.
Teams sometimes say, “We turned on exactly-once, so our database will never have duplicates.” That is only true if the database write is itself protected by an idempotent design. Exactly-once semantics in Kafka Streams protects the Kafka side of the pipeline, and you must design the other side yourself.
17Turning It On: Configuration
A single line enables the feature, but the brokers must be ready for it.
On the application side, the switch is the setting processing.guarantee. Setting it to exactly_once_v2 is enough for Streams to configure the rest by itself. It turns on the idempotent producer, uses transactions, sets acknowledgements to all, and makes the internal consumers read only committed data. It also changes the default commit interval to 100 milliseconds.
| Setting | Where | What it does | Guidance |
|---|---|---|---|
processing.guarantee | Streams application | Chooses at-least-once or exactly-once | Use exactly_once_v2 for new work |
application.id | Streams application | Names the app, its consumer group and its transactional IDs | Make it unique for every logical application |
commit.interval.ms | Streams application | How often a transaction is committed | Default 100 ms. Raise it if coordinators are busy |
num.stream.threads | Streams application | Number of threads, and so producers, per instance | Size to your cores and partitions |
replication.factor | Streams application | Copies of internal topics | Set to 3 explicitly |
num.standby.replicas | Streams application | Warm copies of state stores | 1 for state that is slow to rebuild |
transaction.timeout.ms | Producer, set with a prefix | Time before an unfinished transaction is aborted | Streams lowers it from the plain default. Keep it short but above your worst pause |
transaction.state.log.replication.factor | Broker | Copies of the transaction log | 3 in production |
transaction.state.log.min.isr | Broker | Minimum in-sync copies to write the log | 2 in production |
transaction.state.log.num.partitions | Broker | Partitions of the transaction log | 50 by default. Hard to change later |
transaction.max.timeout.ms | Broker | Largest timeout a producer may request | Must not be smaller than what your app asks for |
transactional.id.expiration.ms | Broker | How long an unused transactional ID is remembered | Default is seven days |
The single-broker development trap
A very common surprise happens on a laptop. The transaction log is created with a replication factor of 3 and a minimum in-sync count of 2 by default. On a development cluster with only one broker, the log can never be created, and the application appears to hang while starting its first transaction. Set both transaction log values to 1 on the development broker, and leave the production values strict.
The transaction log is the diary that lets a new coordinator finish a decided commit. If it can lose data, then the guarantee can silently break. Keep it replicated three times with a minimum of two, exactly like your most important data topics.
Thread count and the group
Every thread has its own transactional producer, so increasing num.stream.threads increases the number of concurrent transactions. This is fine, but it is one reason to watch the coordinator load as you scale. Also keep the consumer settings for polling healthy. If your processing is slow and you exceed the maximum poll interval, the group kicks the thread out, and all its open transactions are aborted.
18Performance and Scalability
Exactly-once costs something. The cost is small if you understand where it comes from.
Transactions add extra messages between the application and the brokers. Each commit involves the transaction coordinator, the group coordinator and the leader of every partition that was written. The costs fall into a few groups, and each one has a knob.
| Source of cost | Why it appears | What helps |
|---|---|---|
| Commit frequency | Every interval ends in a full transaction commit | Raise commit.interval.ms when latency allows |
| Marker fan-out | A commit marker is written to every partition touched, including changelogs and the offsets partition | Fewer output partitions and larger intervals |
| Fewer coalesced updates | Short intervals flush state caches early, so the changelog gets more small updates | Longer intervals let caches merge repeated updates |
| Delayed visibility | Read-committed readers wait for the commit | Budget for the interval in every stage |
| Stronger producer settings | All in-sync replicas must acknowledge each write | Healthy brokers, fast disks, zone-aware placement |
| Coordinator load | Many threads open many transactions per second | Right-size the threads, spread the transaction log |
| Recovery time | Unclean stops force a state rebuild | Standby and warm-up replicas, clean shutdowns |
A worked feel for the numbers
Imagine an application with 20 threads and a commit interval of 100 milliseconds. Each thread commits ten times a second, so the cluster sees about 200 transactions per second from this one application. If the interval is raised to 500 milliseconds, that falls to about 40 per second. If each transaction touches 30 partitions, the difference is between about 6,000 and about 1,200 markers per second. These are illustrative figures, but they show why a modest change of interval is the most powerful tuning step for many workloads.
Scaling out
EOS v2 scales with threads, not with tasks, which was the main reason for the redesign. You can add instances and threads up to the number of input partitions without creating a producer per task. The practical limits are the number of partitions, the capacity of the transaction coordinators, and the time needed to restore state on a new instance. A cluster with dozens of exactly-once applications should keep an eye on the transaction log partitions, because each application’s transactional IDs are spread across them by hashing.
The overhead of exactly-once depends on record size, number of partitions touched and how heavy your processing is. A test that runs your real topology at your real rate, first with at-least-once and then with exactly-once, gives you the honest number for your case.
19Security and Access Control
Exactly-once adds new resources to protect, and new ways for one application to disturb another.
When a Streams application runs with transactions, it does more than read and write topics. It also uses transactional IDs, internal topics and its consumer group. A secure cluster needs permissions for each of these, and the permissions should be as narrow as possible.
Transactional ID access
The application’s identity needs write and describe rights on its transactional IDs. Because they start with the application ID, a prefixed rule on that name is the cleanest way to grant them.
Topics and internal topics
Read on input topics, write on output topics, and create, read and write on internal topics that carry the application ID as a prefix.
Consumer group
Read access to the group named by the application ID, since offsets are committed to it through the transaction.
Unique application IDs
Two different applications with the same ID would share a group and could fence each other’s transactions. Give every application its own name.
Protect changelog topics
Changelogs contain the full contents of state stores, which may include personal or financial data. Restrict who can read them as strictly as the source data.
Encrypt and authenticate
Use encrypted connections and proven identities between the application and the brokers, and between the brokers themselves.
Fencing as a safety feature and a risk
The fencing mechanisms are designed to stop stale writers, and they can also stop the wrong party. If an outside program obtained the right to use a transactional ID that belongs to your application, it could fence your producer and disturb your processing. This is one more reason to keep transactional ID permissions tight and to avoid sharing identities between applications.
Retention and privacy
Aborted records are not deleted at once. They stay in the log until normal retention removes them, hidden from read-committed readers but still present on disk. If a topic contains sensitive data, remember that aborted attempts also contain that data. Use encryption at rest and sensible retention on every topic the application touches, and avoid putting personal data into keys and headers when it is not needed.
Administrative tools can list, describe and even abort transactions. These actions can release stuck readers, but they can also cancel real work. Grant such rights only to the platform team and record their use in audit logs.
20Monitoring, Logging and Metrics
What to watch so that you notice trouble before your users do.
An exactly-once application produces signals in three places: the Streams library, the embedded producer and consumer, and the brokers. Names can vary slightly between versions, so confirm them in the documentation for the version you run.
Application metrics
| Metric | What it tells you | Suggested action |
|---|---|---|
| Commit rate and commit latency | How often and how slowly transactions commit | Alert if latency climbs steadily |
| Process rate | Records handled per second | Compare with the input rate |
| Poll latency and time between polls | Whether the thread is close to the poll deadline | Alert before rebalances start |
| Task closed and created rates | How often tasks move or restart | Investigate a steady non-zero rate |
| Failed and alive stream threads | Whether threads are dying | Page on any failed thread |
| End-to-end record latency | Total delay from source to result | Track against your promised latency |
| Restore progress and lag | How far a rebuilding store has come | Alert on rebuilds that run too long |
Producer and broker metrics
| Metric | What it tells you |
|---|---|
| Producer transaction times: begin, send offsets, commit, abort | Where a transaction spends its time, and how often aborts happen |
| Broker request time for end-transaction, add-partitions, write-markers and offset-commit requests | Load and delay on the transaction path |
| Under-minimum-ISR and under-replicated counts for the transaction log | Whether the diary of transactions is healthy |
| Read-committed consumer lag compared with the high watermark | Whether an open transaction is holding the last stable offset back |
flowchart LR APP["Streams application
commit, poll, failed threads"] --> COL["Metrics Collector"] PR["Embedded producer
transaction timings"] --> COL BR["Brokers and transaction log
request times, ISR health"] --> COL COL --> DASH["Dashboards"] COL --> AL{"Aborts rising, or reader
lag stuck at the stable offset?"} AL -->|"Yes"| PG["Page the on-call engineer"] AL -->|"No"| REV["Weekly review"] PG --> RB["Runbook: find stuck or failing transactions"]
Logs to read
The application logs show fencing events, task migrations and aborted transactions. Messages that mention a fenced producer, a migrated task or an invalid producer epoch are normal in small numbers after rebalances, and worrying in large numbers. Broker logs show the coordinator loading and unloading transaction log partitions, which happens during failovers. Learn what a calm system looks like so that unusual bursts stand out.
The hanging transaction
The most confusing symptom of all is a read-committed consumer that stops receiving new records although producers keep writing and the log keeps growing. The cause is usually one open transaction that never finished, which holds the last stable offset back for everyone behind it. Kafka eventually aborts it when the timeout expires, but if the timeout is long, readers wait. Recent Kafka versions include a command-line tool for transactions that can list, describe, find hanging ones and abort them, and it is worth knowing before you need it.
21Failure Scenarios and a Recovery Playbook
Exactly-once is built for failure. Here is what actually happens in each common case.
| Scenario | What happens | What users and readers see |
|---|---|---|
| Application instance crashes mid-transaction | The open transaction is aborted by timeout. Another instance takes the tasks, rebuilds state if needed, and reprocesses from the last committed offsets. | A pause of up to the timeout plus rebalance time. No duplicates. |
| Long pause makes a zombie | The group moves the tasks away. The zombie’s commit is refused, and its transaction aborts. | Nothing wrong in the output. Fencing messages in the logs. |
| Partition leader fails during a transaction | The producer retries against the new leader. If it cannot, the transaction aborts and is redone. | A short delay. |
| Transaction coordinator broker fails | Another broker loads that part of the transaction log and continues. Decided commits are finished. | Brief stall, then normal flow. |
| Transaction log below minimum in-sync replicas | Transactions cannot start or commit until the log recovers. | The application stalls. Output stops. |
| Processing takes longer than the transaction timeout | The coordinator aborts the transaction, and the work repeats and aborts again. | Progress slows or stops, with many aborts. |
| A poison record crashes the code | The thread fails again and again on the same record. Exactly-once does not skip it. | The application is down or looping. |
| Downstream reader uses read-uncommitted | It sees aborted records. | Inflated or duplicated numbers in that reader only. |
| Brokers lose committed data | The base assumption of the guarantee fails. | Missing or inconsistent results. |
flowchart TD
S["Readers are stuck or output has stopped"] --> A{"Are all stream threads alive
and processing?"}
A -->|"No"| B["Read the thread errors
fix the crash or poison record"]
A -->|"Yes"| C{"Is the transaction log healthy
and fully in sync?"}
C -->|"No"| D["Restore the brokers
the log must reach minimum ISR"]
C -->|"Yes"| E{"Is the last stable offset behind
the high watermark on output topics?"}
E -->|"Yes"| F["Find the open or hanging transaction
wait for timeout or abort it"]
E -->|"No"| G{"Are aborts frequent?"}
G -->|"Yes"| H["Check processing time against the timeout
and look for rebalances"]
G -->|"No"| I["Look at capacity, lag and network"]
A playbook for an incident
Confirm what is broken
Decide whether the problem is that processing has stopped, or that output is not visible. They look similar to a reader but have different causes.
Check the threads
Look for failed or restarting stream threads and read the last error. A repeating error on one record points to a poison record.
Check the transaction path
Look at the health of the transaction log and the request times for commit and marker writing.
Compare stable offset and high watermark
If the stable offset is far behind, an open transaction holds readers back.
Resolve the stuck transaction
Wait for the timeout, or use the transactions tool to inspect and abort a hanging one, with approval from the topic owner.
Look for the cause of aborts
Compare processing time to the timeout and the poll deadline, and check for rebalance storms.
Verify the result
Count the output with a read-committed consumer and compare with expectations. Duplicates or gaps in a verified output mean that some assumption was broken, and you must find which one.
22Error Handling in an Exactly-Once Application
Exactly-once repairs half-finished work, but it does not decide what to do with a record your code cannot handle.
Errors in a stream application come in two families. The first family is about infrastructure: a broker is slow, a network drops, a task is moved to another thread. Streams handles these automatically, retrying, aborting and restarting as needed. The second family is about data and logic: a record that cannot be parsed, a value that breaks a business rule, a bug that throws an exception. These need decisions from you, and under exactly-once the choices behave in a particular way.
The three handlers you can configure
| Handler | When it is called | Typical choices |
|---|---|---|
| Deserialization exception handler | A record’s bytes cannot be turned into an object | Fail, or skip and log, or send to a dead-letter topic |
| Production exception handler | Writing a result to Kafka fails, for example a record that is too large | Fail, or skip the record and continue |
| Uncaught exception handler | Any other exception kills a stream thread | Replace the thread, shut down the client, or shut down the whole application |
Replace, shut down, or stop everything
When a thread dies because of an uncaught exception, Streams lets you choose one of three reactions. Replace the thread starts a new thread that takes over the work, which suits rare and temporary errors. Shut down the client stops the instance, and leaves the others running. Shut down the application tells every instance to stop, which suits an error that would only repeat everywhere, such as a bad record that every instance will meet.
Under exactly-once, a dying thread aborts its open transaction, so nothing half-finished escapes. But remember that a poison record does not go away. If you replace the thread and it reads the same record again, it will fail again, and the application will loop. The transaction protects your output from that loop, but it cannot end it.
Dead-letter topics fit naturally
A dead-letter topic is a place where you send records that could not be processed, together with a note about why. Writing to it from inside the topology is an ordinary Kafka write, so under exactly-once it is part of the same transaction. If the transaction aborts, the dead-letter entry is cancelled with everything else, and when the record is retried it is either processed properly or diverted again. You will never get a dead-letter entry for a record whose progress was not committed.
Production example: the malformed payment
A payment event arrives with an amount field that says “N/A”. The parsing code throws, the thread dies, and its replacement reads the same record and dies too. The application loops until an engineer notices. After adding a validation step that sends malformed events to a dead-letter topic with the reason, the same bad record is set aside in one transaction, the balance is left alone, and the stream carries on. The lesson: plan for bad data before it appears.
When a handler chooses to skip a record, that record is gone from the results. For money or counts, skipping silently can be worse than stopping. Log every skipped record, count them in a metric, and agree with the business which kinds of records may be dropped.
23Migrating to EOS v2
Moving an existing application safely from at-least-once or from the first exactly-once design.
The right path depends on where you start. An application that is already exactly-once with the old setting needs a careful two-step change, because old and new instances must be able to coexist during a rolling restart. An application that is currently at-least-once needs a different kind of care, because its output topics may already contain duplicates that downstream systems have learned to live with.
Upgrade the brokers first
EOS v2 needs brokers at version 2.5 or newer. Upgrade the cluster and confirm that the transaction log is healthy and replicated three times.
Upgrade the Streams library
Move the application to a version that supports v2. If you are still on the first design, keep that setting for this rolling restart.
Switch the setting in a second rolling restart
Change to exactly_once_v2 and restart the instances one at a time. Follow the upgrade guide for your exact versions, which describes the required order.
Update the readers
Make sure every consumer of the application’s output topics uses read-committed, or they will see aborted records.
Watch the new metrics
Track commit latency, abort counts and coordinator load for a few days before declaring success.
Plan for the past
Duplicates that were already written under at-least-once stay in the topics. Decide whether downstream systems should clean them up.
Running some instances of an application with the first design and others with v2 at the same time, outside the documented upgrade steps, can leave zombies unfenced. Always follow the rolling-restart order in the upgrade guide, and do not skip steps to save time.
Going the other way
Turning exactly-once off is simpler, but it is not free. Switching back to at-least-once means that after a crash, duplicates can appear again, and any reader that relied on the guarantee must be reviewed. Remember also that transactional markers already in the topics remain, and read-committed readers continue to handle them correctly.
24Design Patterns and Anti-Patterns
Habits that let exactly-once do its job, and habits that quietly defeat it.
Patterns that work
Keep the pipeline inside Kafka
Do the heavy lifting in read-process-write stages between topics, and keep external calls at the edges of the system.
Idempotent edges
Where the pipeline touches the outside world, use unique event IDs, upserts and idempotency keys so that repeats are harmless.
Read-committed everywhere
Standardize the isolation level for every consumer of transactional topics, and check it in code review.
Strong infrastructure baseline
Replication factor 3, minimum in-sync count 2, unclean election off, for data topics, internal topics and the transaction log.
Deterministic logic
Write processors that give the same answer for the same input, so that a retry produces the same result as the aborted attempt.
Verification stream
Run a small check job that counts or sums outputs against inputs with read-committed, and alerts on any difference.
Anti-patterns to avoid
What people do
They call a payment or email service directly from inside a processor and assume that exactly-once will protect the call.
Why it hurts
The call is outside the transaction. After an abort the same record is processed again, and the customer is charged or emailed twice.
Do this instead
Write the intent to a topic, and let a separate step perform the call with an idempotency key.
What people do
They leave downstream consumers on the default isolation level.
Why it hurts
Those consumers see aborted records. The producing side is perfect, and the reading side quietly breaks the guarantee.
Do this instead
Set read-committed for every reader of an exactly-once topic and test it.
What people do
They create internal topics or the transaction log with a replication factor of 1 to save disk space.
Why it hurts
A single broker failure can lose changelog data or transaction decisions, which breaks the very state that exactly-once relies on.
Do this instead
Use a replication factor of 3 and a minimum in-sync count of 2 for everything the application uses.
What people do
They give two different applications the same application ID, or reuse one ID for a test copy running against production.
Why it hurts
The applications join the same consumer group, steal each other’s partitions and fence each other’s transactions.
Do this instead
Give each application, and each environment, its own unique ID.
25Real-World and Industry Examples
Where exactly-once earns its cost, and where it does not.
Advertising platforms: counting money spent
Companies that sell advertising must not count an impression or a click twice, because a duplicate becomes a bill that a customer disputes. Pinterest has publicly described using Kafka Streams in its advertising systems. The lesson: when a counter turns into an invoice, exactly-once processing is cheap insurance compared with refunds and lost trust.
Banks and payment platforms: balances and ledgers
Financial firms use stream processing to update balances, detect fraud and produce statements from streams of transactions. A duplicated or lost transaction changes a balance. These teams usually combine exactly-once processing with strict infrastructure settings and independent checks that compare totals. The lesson: the guarantee is one layer of a defence, and it works best beside reconciliation.
Online retail: stock and order counts
A shop that tracks “reserved” and “released” stock from events needs each event to be applied once. A duplicated release inflates stock and a lost reservation oversells it. Exactly-once keeps the counts right through restarts and rebalances. The lesson: stateful counting is the classic use for the feature.
Telemetry and click analytics: when not to pay
A team that builds a dashboard of approximate page views may see no benefit from exactly-once. A handful of duplicates does not change a trend line, and the extra delay from the commit interval may even be unwelcome. The lesson: use the default at-least-once mode where small errors are harmless, and save the transactional cost for data that must be exact.
These descriptions summarize general and public uses of Kafka Streams, and the lessons are common patterns in the industry. Internal designs at any company change over time, so treat them as teaching examples, and read current public engineering posts if you need exact details.
26Best Practices and Common Mistakes
A checklist for your next design review.
Do
- Use
exactly_once_v2for every new exactly-once application. - Give each application a unique application ID.
- Replicate internal topics and the transaction log three times, with a minimum of two in sync.
- Set every reader of an exactly-once topic to read-committed.
- Keep processors deterministic and free of hidden side effects.
- Make external writes idempotent with event IDs and idempotency keys.
- Measure the commit interval trade-off with your own workload.
- Use standby replicas for large state.
- Alert on aborts, failed threads and a stuck stable offset.
Avoid
- Assuming that exactly-once covers databases, emails or other clusters.
- Leaving downstream readers on read-uncommitted.
- Letting processing time exceed the transaction timeout.
- Copying single-broker development settings into production.
- Setting a very long timeout to hide a slow processor.
- Sharing application IDs between applications or environments.
- Ignoring poison records, since retries will not fix them.
- Turning on exactly-once everywhere without asking who needs it.
- Skipping the upgrade steps when moving from the first design.
Common mistakes in more detail
Mistake one: reading “exactly-once” as “the code runs once”. The code can run many times, and often does. What is guaranteed is the committed effect. Design processors so that a cancelled run leaves no lasting trace outside Kafka.
Mistake two: forgetting the delay. Results wait for the commit. Teams that promise a latency of a few milliseconds and then enable exactly-once with the default interval are surprised by a rise in delay. Decide the interval together with the latency target.
Mistake three: trusting a green dashboard. An application that shows no errors can still feed wrong numbers to a reader with the wrong isolation level. Verify the output from the reader’s side.
Mistake four: testing only the happy path. Exactly-once is about failures. If you have never killed an instance in the middle of a busy period and checked the output, you do not yet know that your setup works.
27Testing Exactly-Once Applications
A guarantee that has never been tested under failure is only a hope.
Unit tests for a Streams topology usually run without a real broker, using a test driver that feeds records directly to the topology. This is excellent for checking logic, but the driver does not run real transactions, so it cannot prove the exactly-once behaviour. You need at least two more layers of testing.
Test the logic with a test driver
Check that the topology computes the right answers for a fixed set of inputs, including out-of-order and repeated records.
Test with real brokers
Run the application against a real or containerized cluster with the same transaction settings as production, at least in miniature.
Send numbered input
Produce records with strictly increasing sequence numbers so that a missing or repeated number is easy to see.
Cause failures on purpose
While the load runs, stop instances abruptly, pause them, force rebalances and restart brokers, one failure at a time.
Verify with a read-committed reader
Read the whole output and check for gaps, repeats and wrong totals. Any difference is a finding to investigate.
Repeat regularly
Run the same failure tests after upgrades and before major changes, because settings and versions change behaviour.
Include at least one downstream consumer in your test, with the isolation level you use in production. Many failures are found there, not in the stream application.
28Worked Example: An Imaginary Wallet App
Let us put numbers on a made-up company to see the design choices in action.
Imagine a wallet company called PayNest. Customers add money, pay shops and send money to friends. Every action becomes an event in a topic named wallet-events with 48 partitions, keyed by account number. A Kafka Streams application checks each event, updates the account balance in a state store, and writes the new balance to a topic named balances and a copy of the action to an audit topic. The cluster has six brokers across three zones. The application runs on six instances with four threads each, so 24 threads share 48 tasks, two tasks per thread.
Choosing the guarantee and the settings
| Decision | Choice | Reason |
|---|---|---|
| Processing guarantee | exactly_once_v2 | A duplicated action changes a customer’s money |
| Commit interval | 200 ms | Small enough for a lively app, large enough to halve the transaction rate |
| Internal topics | Replication factor 3, minimum in-sync 2 | The state must survive a broker loss |
| Standby replicas | 1 | A lost instance must not mean minutes of rebuilding |
| Downstream readers | All set to read-committed | Notification and reporting services must never see aborted balances |
| Bank transfer calls | Done by a separate service with idempotency keys | External calls are outside the transaction |
What the design costs
| Measure (illustrative) | First design | EOS v2 |
|---|---|---|
| Producers in the application | 48, one per task | 24, one per thread |
| Transactions per second at 200 ms | About 240 | About 120 |
| Partitions touched per transaction | About 4 to 5 per task | About 9 per thread, covering two tasks |
| Marker writes per second | About 1,100 | About 1,100, but from half as many transactions |
The number of markers stays similar, because the same partitions are written, but the coordinators handle half as many transactions, half as many producers hold connections, and each commit does more useful work. The savings grow as the number of tasks grows relative to the number of threads.
A bad night
During a busy evening, one instance loses power in the middle of a commit. Its two threads had open transactions with records in the balances and audit topics. Here is how PayNest fares. The group notices the missing heartbeats and moves the eight tasks of that instance to other threads. The open transactions are not committed, so after the short transaction timeout the coordinator aborts them, and the records in them stay hidden. The new owners rebuild their stores. Thanks to the standby replicas, they take over within seconds, and they restart from the last committed offsets. Read-committed readers pause briefly, then continue, and they see each balance change exactly once.
What the on-call engineer checks in the morning
She looks at the abort count, which rose for a short period, and at the fencing messages in the logs, which are expected after a lost instance. She checks that the last stable offset caught up with the high watermark, and she runs the verification job that compares total credits and debits between input and output. The totals match to the last rupee, and the incident closes with no customer impact.
29Where Exactly-Once Is Heading
The idea is stable, and the edges keep getting smoother.
The Kafka community keeps improving the transaction path. Proposals such as KIP-890 aim to make transactions more robust, for example by letting brokers verify that a partition really belongs to a transaction before accepting writes, which reduces the chance of stuck transactions, and by cutting some round trips between producers and coordinators. Newer consumer group and Streams rebalance protocols move more coordination to the broker side, which can make rebalances calmer and reduce the number of fencing events. The exact status of each of these differs between releases, so read the release notes of the version you run before relying on them.
Meanwhile, the advice stays steady. Use the current mechanism, keep the infrastructure strong, make the edges idempotent, and test your failure handling. Applications built on those habits will benefit from every improvement without needing a redesign.
Exactly-once is like the seat belt in a car. The design has been refined over the years to be lighter, easier to fasten and more reliable, but the reason for wearing it has not changed. It does not stop you from making a wrong turn, and it does not fix the road, but when something goes wrong, it makes sure that you come through unharmed.
30Frequently Asked Questions
processing.guarantee to exactly_once_v2. Streams then configures the idempotent producer, transactions and read-committed consumers for you.31Summary and Key Takeaways
The whole guide in a few sentences.
Key Takeaways
- What it promises. Exactly-once means the committed effect of each input record appears once in Kafka, even though your code may run more than once after failures.
- Three building blocks. The idempotent producer removes retry duplicates, transactions make outputs, state changes and input offsets commit together, and fencing stops zombies from committing.
- EOS v2 in one line. One transactional producer per stream thread, and zombies stopped by the consumer group generation check, which scales far better than the first design.
- Readers must cooperate. Every consumer of an exactly-once topic needs read-committed, or it will see aborted records and break the guarantee at the last step.
- Know the border. The guarantee covers Kafka topics in one cluster. Databases, emails and APIs need idempotent design, such as unique event IDs and idempotency keys.
- Infrastructure matters. Keep internal topics and the transaction log replicated three times with two in sync, and keep unclean leader election off, or the base of the promise can break.
- Tune, watch and test. The commit interval trades delay for throughput, aborts and a stuck stable offset are the signals to watch, and failure tests are the only real proof that it works.