Before we can talk about doing something exactly once, we need to see why doing it reliably is so hard in a system made of many machines.

A quick Kafka refresher

Apache Kafka stores streams of events in topics. An event is a small record of something that happened, such as a payment, an order or a sensor reading. Producers write events to topics, consumers read them, and brokers, the servers of the Kafka cluster, store and serve them. Each topic is divided into partitions, which are ordered logs, and every event in a partition has a position number called an offset. A consumer remembers how far it has read by saving an offset.

Machines fail in the middle of work

In a system spread over many machines, anything can fail at any moment. A network cable may drop a message. A server may crash after doing half of a job. A reply may be lost even though the work succeeded. The sender cannot tell the difference between “the request never arrived” and “the request worked but the answer was lost”. The only safe reaction, retrying, risks doing the work twice. Not retrying risks not doing it at all.

Why this matters for business data

For some data, duplicates or gaps hardly matter. A duplicated click in a website analytics stream barely changes a chart. But for money, stock levels, bookings and billing, a duplicate or a missing event produces wrong numbers that real people notice. For these workloads, teams need a guarantee, not a hope.

Every messaging system offers one of three levels of promise about how many times an event is handled.

flowchart LR
  subgraph AMO["At-most-once"]
    A1["Save progress"] --> A2["Do the work"]
    A2 -. "crash here: work lost" .-> A3["Event skipped"]
  end
  subgraph ALO["At-least-once"]
    B1["Do the work"] --> B2["Save progress"]
    B1 -. "crash before saving: work repeated" .-> B3["Event handled twice"]
  end
  subgraph EO["Exactly-once"]
    C1["Work and progress"] --> C2["Committed together or not at all"]
  end
        

What Kafka gives by default

A plain Kafka producer with safe settings and a consumer that saves its offset after processing gives at-least-once behaviour. Nothing is lost, but retries and restarts can create duplicates. The features covered in this tutorial extend this default up to exactly-once for a defined scope.

To fix a problem we must find exactly where it appears in the journey of an event.

Failure point one: producer retries

A producer sends a batch of events to a broker and waits for an acknowledgement. If the acknowledgement is lost, or a timeout occurs, the producer does not know whether the broker stored the batch. It retries, and if the first attempt had actually succeeded, the broker now holds the batch twice.

sequenceDiagram
  participant P as Producer
  participant B as Broker
  P->>B: Send batch of events
  B->>B: Store batch in the log
  B--xP: Acknowledgement lost on the network
  Note over P: No reply, so producer retries
  P->>B: Send the same batch again
  B->>B: Store batch a second time
  Note over B: Duplicate events in the log
        

Failure point two: the consumer’s read-process-write cycle

Many applications read an event, process it and write a result to another topic, then save the input offset. If the application crashes after writing the result but before saving the offset, it restarts from the old offset, reads the same event again and writes the result a second time.

Failure point three: zombie instances

A zombie is an old copy of an application that everyone thought was dead, but which is actually just slow or disconnected. Meanwhile a replacement copy has taken over. If the zombie wakes up and keeps writing, both copies produce output for the same input, causing duplicates or corrupted results.

Failure point four: broker failures

If a broker holding a partition leader fails, another replica takes over. Events acknowledged only by the failed broker but not yet copied may be lost, unless the producer required acknowledgement from all in-sync replicas. Safe settings are the foundation for any stronger guarantee.

The phrase sounds absolute, so it is worth being precise about what is and is not promised.

The honest definition

Exactly-once semantics in Kafka means that when an application reads events from Kafka topics, processes them and writes results back to Kafka topics, the results appear in the output exactly once, as if there had been no failures. The guarantee covers the whole read-process-write cycle, including the saving of input progress. It is achieved by combining idempotent producers, transactions and consumers that read only committed data.

Effect, not execution

Exactly-once does not mean your code runs exactly once. After a crash, your code may well run again on the same input. What it means is that the effects inside Kafka, the output events and the saved offsets, are visible exactly once. Aborted attempts leave no visible trace for properly configured readers.

The boundary of the guarantee

The guarantee covers work that starts and ends inside Kafka. If your application also sends an email, calls a payment service or updates a database while processing an event, those external actions are outside Kafka’s transaction. Kafka cannot undo an email that has already been sent. This boundary is the most important limit to remember, and later chapters show how to handle it.

flowchart LR
  subgraph IN["Inside Kafka: guaranteed"]
    T1["Input topic"] --> APP["Processing application"]
    APP --> T2["Output topic"]
    APP --> OFF["Saved offsets"]
  end
  APP -. "not covered by Kafka transactions" .-> EXT["Email, database, HTTP call"]
        

The first mechanism removes duplicates caused by producer retries.

What idempotent means

An operation is idempotent when doing it many times has the same effect as doing it once. Pressing the lift call button five times does not summon five lifts. Setting a light switch to “on” repeatedly leaves the light on. An idempotent producer can safely resend the same batch, because the broker will store it only once.

How the broker recognises repeats

When idempotence is on, the producer receives a unique number from the broker called a producer ID, and every batch it sends to a partition carries a sequence number that increases by one each time. The broker remembers the latest sequence numbers per producer and partition. If a batch arrives with a sequence number it has already stored, the broker recognises the repeat, does not write it again, and simply acknowledges it.

sequenceDiagram
  participant P as Producer, ID 42
  participant B as Broker
  P->>B: Batch, sequence 7
  B->>B: Store batch, remember last sequence 7
  B--xP: Acknowledgement lost
  P->>B: Retry batch, sequence 7
  B->>B: Sequence 7 already stored, ignore
  B-->>P: Acknowledge without storing again
        

Settings that go with it

In recent Kafka versions, idempotence is turned on by default for producers. It requires that the producer waits for acknowledgement from all in-sync replicas, that retries are enabled, and that the number of unacknowledged requests per connection stays within a small limit, so ordering is preserved even when retries happen. If you configure conflicting values, the producer refuses to start instead of silently weakening the guarantee.

A closer look at the machinery shows both why it works and where its edges are.

Producer ID and epoch

Each producer session is identified by a producer ID and an epoch. The epoch is a counter attached to the ID that can be increased to signal a newer session. The pair is what the broker uses to decide whether a batch is a repeat or new work.

Sequence numbers per partition

Sequence numbers are tracked separately for each partition that a producer writes to. The broker expects the next batch to have exactly the next number. If a number is skipped, meaning something was lost, the broker raises an out-of-order error. If a number is repeated, it is treated as a duplicate and ignored.

The limited memory of the broker

The broker does not remember every batch forever. It keeps information only about a limited number of the most recent batches for each producer and partition. This is enough to catch ordinary retries, because retries happen quickly, but it also explains the requirement to limit the number of unacknowledged requests in flight.

What happens on restart

A plain idempotent producer that is restarted receives a new producer ID, because the broker has no way to know the new process is the continuation of the old one. Its sequence numbers start again from zero, so it cannot be recognised as a replacement. This is why idempotence alone protects only within one running session, and why transactions introduce a stable identity that survives restarts.

The idempotent producer solves one problem well, but real pipelines have bigger ones.

A transaction makes several writes and the saving of offsets succeed or fail as a single unit.

What a Kafka transaction is

A Kafka transaction lets a producer write events to several partitions, possibly in several topics, and include the saving of consumer offsets, all as one atomic operation. Atomic means all or nothing. Either every write in the transaction becomes visible to readers, or none of them does.

The transactional identifier

To use transactions, the application sets a transactional identifier, a name that you choose and keep the same across restarts. Kafka maps this name to a producer ID. When a new instance starts with the same name, Kafka recognises it as the continuation of the old one, hands it the same producer ID with a higher epoch, and finishes or cancels any half-done transaction left by the previous instance. This is what fixes the identity problem of the idempotent producer.

Someone has to keep the books for every transaction. In Kafka, that job belongs to a broker role and an internal topic.

The transaction coordinator

Every transactional identifier is assigned to one broker acting as its transaction coordinator. The coordinator hands out producer IDs and epochs, records which partitions a transaction has written to, and drives the commit or abort process. The assignment is decided by hashing the transactional identifier onto a partition of the internal transaction log, and the leader of that partition acts as the coordinator.

The transaction log

The coordinator stores its state in a special internal topic. Every change in a transaction’s state, such as ongoing, preparing to commit, or completed, is written there before it takes effect. Because it is a replicated Kafka topic, the state survives broker failures, and a new coordinator can read it to continue where the old one stopped.

flowchart TB
  APP["Producer with a transactional ID"] --> CO["Transaction coordinator broker"]
  CO --> TL[("Internal transaction log topic")]
  APP --> P1["Data partition A"]
  APP --> P2["Data partition B"]
  CO --> P1
  CO --> P2
        

Safety settings of the log

Because the transaction log is the source of truth, its replication and minimum in-sync replica settings matter greatly. Kafka defaults these to safe values for production clusters. Small development clusters with a single broker need these lowered, but they should never be lowered in production.

Following one transaction from start to finish shows how every piece fits.

flowchart TD
  R["Read batch of input events"] --> B["Begin transaction"]
  B --> W["Process and write output events"]
  W --> O["Add input offsets to the transaction"]
  O --> C{"All succeeded?"}
  C -- "Yes" --> CM["Commit: output and offsets visible together"]
  C -- "No" --> AB["Abort: nothing becomes visible"]
  CM --> R
  AB --> R
        

The heart of atomicity is a careful two-step ending that survives crashes at any moment.

Phase one: decide

When the producer asks to commit, the coordinator first writes a “preparing to commit” record in the transaction log. Once this is safely stored, the decision is final. Even if the coordinator crashes now, its replacement will read the log, see the decision, and carry it out. The transaction can no longer be aborted at this point.

Phase two: tell everyone

The coordinator then sends a commit marker to every partition that took part in the transaction. A commit marker is a special control record appended to the partition’s log, not visible to normal applications. It states that all events from this producer’s transaction are now committed. After all partitions have the marker, the coordinator writes a “completed” record in its own log, and the transaction is finished.

sequenceDiagram
  participant P as Producer
  participant C as Coordinator
  participant L as Transaction log
  participant D as Data partitions
  P->>C: Commit transaction
  C->>L: Write "preparing to commit"
  Note over C,L: Decision is now final
  C->>D: Write commit markers
  D-->>C: Markers written
  C->>L: Write "committed, complete"
  C-->>P: Commit successful
        

Abort works the same way

An aborted transaction follows the same two steps, but the markers say “abort”. Readers that respect transactions then skip the events belonging to the aborted transaction as if they were never written. The events remain in the log until normal retention removes them, but they are ignored.

A revived old instance must never be able to spoil the work of its replacement. Epochs make that possible.

How fencing works

Every time an application starts up and registers its transactional identifier, the coordinator increases the epoch for that identifier. From then on, the coordinator and the partition leaders only accept writes carrying the newest epoch. If an old instance wakes up and tries to write with its lower epoch, the request is rejected with an error stating that the producer has been fenced. The zombie learns it has been replaced and must stop.

sequenceDiagram
  participant Z as Old instance, epoch 3
  participant N as New instance
  participant C as Coordinator
  participant D as Partition
  Note over Z: Old instance stalls
  N->>C: Register same transactional ID
  C-->>N: Epoch 4 granted, epoch 3 fenced
  N->>D: Write with epoch 4
  D-->>N: Accepted
  Z->>D: Wakes up, writes with epoch 3
  D-->>Z: Rejected: producer fenced
        

Why a stable transactional identifier matters

Fencing works only if the replacement uses the same transactional identifier as the instance it replaces. If identifiers were random per start, the coordinator would treat each as a new, unrelated producer and could not fence anyone. This is why identifiers should be derived from the logical role, for example the input partition being processed, rather than generated freshly at each launch.

Reacting to being fenced

A fenced producer is not a temporary error to retry. It means a newer instance owns the work. The correct reaction is to stop using that producer and let the application shut down or rebuild its state, depending on design.

Atomic writes are useless if readers can still peek at half-finished transactions. Consumers must be told to respect them.

Two isolation levels

A consumer can be set to one of two isolation levels. In the read-uncommitted level, which is the default, it sees every event as soon as it is written, including events from transactions that are still open or later aborted. In the read-committed level, it sees only events that belong to committed transactions, plus events that were not written in a transaction at all.

The last stable offset

To support read-committed, each partition keeps track of a position called the last stable offset. It is the offset up to which every transaction has been decided, either committed or aborted. A read-committed consumer never reads beyond this point. Events belonging to a still-open transaction sit beyond the last stable offset until the transaction finishes.

flowchart LR
  A["Committed events"] --> B["Decided transaction"] --> LSO["Last stable offset"] --> C["Open transaction events"] --> HW["Newest written offset"]
        

A stuck transaction blocks readers

Because read-committed consumers cannot pass the last stable offset, one transaction that stays open for a long time holds up all readers of that partition. The transaction timeout exists to prevent this: if a producer disappears without finishing, the coordinator eventually aborts its transaction, letting the last stable offset move on. Choose the timeout with this in mind.

Now we can put the pieces together into the classic exactly-once loop.

The problem it solves

Recall the crash between writing output and saving the input offset. In the classic loop, the application reads a batch, transforms it and writes output, and then saves the offset. To repair the gap, the offset must be saved not by the consumer on its own, but through the producer’s transaction, so it commits or aborts together with the output.

The steps in words

Rebalances and the consumer group

When a consumer group rebalances, partitions move between instances. Including the consumer group’s metadata in the offset submission lets the broker check that the instance committing offsets still owns those partitions, and refuse commits from a stale owner. This closes another zombie loophole on the consumer side.

Most teams do not write the transaction loop by hand. Kafka Streams does it for them behind a single setting.

One switch

Kafka Streams is a library for building stream processing applications. To enable exactly-once, you set its processing guarantee to the exactly-once option. The current recommended option is often called exactly-once version two. Older options are deprecated. With this on, the library configures idempotent, transactional producers and read-committed consumers, and wraps its processing in transactions automatically.

What goes into each transaction

A stream application does more than read and write topics. It also updates local state stores, and it sometimes writes to internal repartition topics. Under exactly-once, the transaction contains all of these together: the output events, the writes to the state store’s backup topic, the repartition topic writes and the input offsets. They all commit or all abort.

flowchart TB
  IN["Input topic"] --> T["Stream task"]
  T --> ST[("Local state store")]
  subgraph TX["One transaction"]
    O["Output topic writes"]
    CL["State store backup topic writes"]
    OF["Input offsets"]
  end
  T --> O
  T --> CL
  T --> OF
        

Version two and efficiency

Earlier exactly-once versions used one transactional producer per task, which meant many producers and heavy overhead for applications with many partitions. Version two uses one producer per stream thread, which greatly reduces resource use while keeping the same guarantee. It requires reasonably recent brokers.

Commit interval changes

Under exactly-once, transactions are committed frequently, with a much shorter default interval than under at-least-once. Shorter intervals lower the delay before downstream readers see results, at the cost of more transactions. You can tune this to trade latency against overhead.

State stores and unclean shutdowns

Local state may hold changes from a transaction that never committed. After an unclean crash, Kafka Streams may discard the local files and rebuild them from the committed backup topic, so that state matches the committed truth. This costs restoration time but protects correctness.

Real pipelines start and end outside Kafka, and that is where the boundary of the guarantee matters most.

Getting data in: source connectors

Kafka Connect moves data between Kafka and other systems. Source connectors read from a database or a service and write to Kafka. Recent versions support exactly-once for source connectors that are able to participate: the connector’s progress and its written records commit together in a transaction. Not every connector supports it, so check the documentation of each one.

Getting data out: sink connectors

For sink connectors, which write from Kafka into another system, the outside system usually decides the guarantee. If the target can accept a write with a unique key and ignore repeats, or can store the Kafka offset atomically together with the data, then repeated delivery causes no harm. Otherwise, duplicates may appear on the far side.

Outside systems cannot join Kafka transactions

A Kafka transaction cannot include a database update, an email or a call to another service. If your application updates a database and then crashes before committing the Kafka transaction, the database keeps the change while Kafka rolls back. On restart, the update may run again. For those cases you need extra patterns, described in the next chapter.

flowchart LR
  SRC["Source system"] -->|"Source connector: can be exactly-once"| K["Kafka topics"]
  K -->|"Kafka Streams: exactly-once inside Kafka"| K2["Kafka output topics"]
  K2 -->|"Sink connector: depends on the target system"| DST["Database or service"]
        

When you cannot make an outside action part of the transaction, you make it safe to repeat. The result behaves as if it happened once.

Exactly-once depends on safe underlying settings. Weak settings quietly undermine it.

Version awareness

Exactly-once features have evolved across Kafka releases, and the newest Streams option needs reasonably recent brokers. Before adopting it, confirm the versions of your brokers, clients and libraries, and read the release notes for your version.

Stronger guarantees are not free. Knowing the costs helps you decide where they are worth paying.

Where the extra cost comes from

What influences the overhead

The fixed cost per transaction is spread across the events inside it. Larger batches per transaction dilute the overhead and raise throughput, while a very short commit interval adds cost and reduces efficiency. The number of partitions written per transaction also matters, because each needs a marker. Modest transactions over a limited set of partitions are cheaper than sprawling ones.

The latency trade-off

The commit interval directly sets the delay before downstream read-committed consumers can see new results. A short interval reduces the delay but increases overhead. A longer interval improves throughput but means results arrive in larger, later chunks. Choose based on how fresh the downstream data must be.

The best way to trust the design is to watch it handle each failure in turn.

Transactions add new resources and new powers, and they need the same care as any sensitive feature.

A guarantee you cannot observe is a guarantee you cannot trust. These signals show whether things are healthy.

End-to-end checks

Add business-level reconciliation on top of technical metrics. For example, compare the total of input amounts with the total of output amounts over a period. Such checks catch problems that no low-level metric reveals, including duplicates introduced at the outside edges of the pipeline.

Practices that make exactly-once work smoothly, and the traps that quietly defeat it.

Patterns worth adopting

Anti-patterns to avoid

Exactly-once is powerful but not always necessary. A simple decision path helps you choose.

flowchart TD
  S["What happens if an event is duplicated or lost?"] --> Q1{"Is loss acceptable?"}
  Q1 -- "Yes, gaps are fine" --> AMO["At-most-once"]
  Q1 -- "No" --> Q2{"Can consumers safely handle duplicates?"}
  Q2 -- "Yes, idempotent logic" --> ALO["At-least-once with idempotent producer"]
  Q2 -- "No, results would be wrong" --> Q3{"Does the whole cycle stay inside Kafka?"}
  Q3 -- "Yes" --> EOS["Exactly-once with transactions or Streams"]
  Q3 -- "No, outside systems involved" --> MIX["Exactly-once inside plus idempotent outside actions"]
        

A quick reference for teams adopting or reviewing exactly-once designs.

Five common situations show how the pieces combine in practice.

No. After a failure your code may run again on the same input. What is guaranteed is that the effects inside Kafka, the output events and the saved offsets, become visible exactly once.

Idempotence removes duplicates from retries within one producer session on a single partition. Transactions add atomic writes across partitions, atomic saving of consumer offsets, stable identity across restarts and fencing of zombies.

Only if you need output and input progress to commit together, atomic writes across partitions, or protection across restarts. For simple producers where retry duplicates are the only concern, idempotence is enough.

Otherwise they see events from open and aborted transactions. Read-committed makes them read only decided, committed data, which is what makes the atomicity visible to them.

A zombie is an old instance that has been replaced but is still running. Kafka gives the replacement a higher epoch for the same transactional identifier, and rejects writes with older epochs from the zombie.

No. Kafka transactions cover only Kafka topics and offsets. For databases, use the outbox pattern, store offsets with the data, or make the writes idempotent.

Commit and abort markers are control records that occupy offsets in the partition. Applications do not receive them, so offsets appear to skip numbers. Nothing is missing.

Through a single processing guarantee setting. The library then uses idempotent transactional producers, read-committed consumers and wraps output, state backup writes and offsets in transactions automatically.

There is some overhead from coordinator traffic and commit markers, and downstream readers see results after commits, adding latency. The impact depends on batch sizes, partition counts and commit interval, so measure with your own workload.

Read-committed consumers of the affected partitions are held back at the last stable offset until the transaction is finished. After the timeout, the coordinator aborts it and readers move on.

Recent versions support it for source connectors that are able to take part. For sink connectors, exactly-once behaviour depends on how the target system handles repeated or offset-tagged writes.

No. If your logic is naturally repeat-safe or small errors are acceptable, at-least-once with an idempotent design is simpler and cheaper. Choose exactly-once where duplicates or losses would produce wrong business results.

Failures in distributed systems force a choice between losing events and repeating them. Kafka’s exactly-once semantics removes that choice inside Kafka by combining three ingredients: the idempotent producer, which lets brokers discard retried batches; transactions, which commit output and consumer offsets together and fence out zombie instances using epochs; and read-committed consumers, which see only decided data. Kafka Streams wraps all of it behind a single setting.

The guarantee has a clear boundary. It covers the read-process-write cycle inside Kafka, and it needs safe replication settings and cooperating consumers. For outside systems, pair it with idempotent design. If you remember one sentence, let it be this: exactly-once is about visible effects inside Kafka, and your design must make everything outside Kafka safe to repeat.