Before talking about exactly-once, we need to know what the other options are and why they exist.

When a system moves a message from one place to another, three outcomes are possible if something can fail in the middle. The message may arrive zero times, once, or more than once. Engineers name the guarantees after what they promise, and each one is a trade between simplicity, speed, and correctness.

What Exactly-Once Really Means

The phrase is slightly misleading. In a network where messages can be delayed or resent, a message may physically be transmitted more than once. What exactly-once promises is about effects, not transmissions. The final result must look as if each message was processed one single time. The system achieves this by making repeated attempts harmless, through identifiers, transactions, or atomic updates.

You need a clear mental picture of the moving parts before you can see where things go wrong.

Kafka Connect is a framework, shipped with Apache Kafka, for moving data between Kafka and other systems without writing custom applications. Instead of coding a producer or consumer for every database, file store, or search index, you configure a ready-made connector and let the framework run it.

A connector is the plug-in that knows how to talk to one specific system. A source connector reads from an outside system and writes into Kafka topics, for example reading changes from a database. A sink connector reads from Kafka topics and writes into an outside system, for example loading events into a data warehouse or a search index.

flowchart LR
  E1["External system A"] --> S["Source task"]
  S --> K["Kafka topics"]
  K --> D["Sink task"]
  D --> E2["External system B"]
  W["Connect worker hosts the tasks"] -.-> S
  W -.-> D
        

Every duplicate and every loss traces back to how progress is recorded. This is the key to the whole topic.

A task that moves data forever needs a bookmark, so that after a restart it can continue instead of starting over or skipping ahead. Source and sink connectors keep their bookmarks in completely different places, and this difference explains why their exactly-once stories are so different.

Source Connector Bookmarks

A source connector reads from a system that is not Kafka, so Kafka has no idea what position it has reached. Therefore each source record carries two small maps. The source partition describes which slice of the outside system the record came from, such as a table or a file name. The source offset describes the position inside that slice, such as a row number or a log position. The Connect framework periodically saves the latest source offsets into an internal Kafka topic. After a restart, the connector asks for its saved offsets and resumes from there.

Sink Connector Bookmarks

A sink connector reads from Kafka topics, so its bookmark is an ordinary consumer group offset. The framework commits these offsets to Kafka after the sink task has handed the records to the outside system. After a restart, the task continues from the last committed offset.

The failure windows are small and precise. Knowing them turns a mysterious problem into a solvable one.

The Classic Source Connector Window

Without special support, a source task produces records to Kafka, and later, on a timer, the framework saves the source offsets. Suppose the task has produced records for positions one to five hundred, but has saved offsets only up to four hundred. If the worker crashes now, the restart resumes at four hundred. The records from four hundred one to five hundred are produced a second time. Nothing was lost, but duplicates now sit in the topic.

The Classic Sink Connector Window

A sink task receives records from Kafka, writes them to the outside system, and afterwards commits the consumer offsets. If it crashes after the write but before the offset commit, the restart resumes at the older offset and writes the same records again. If the outside system is not protected against repeats, duplicates appear there.

The Opposite Failure: Loss

Loss happens when the bookmark moves ahead of the actual work. For example, a sink that commits offsets first and writes afterwards can crash in between, and the restart skips records that were never written. That is why well-designed frameworks always move the bookmark after the work, accepting duplicates rather than loss. This is at-least-once behavior, and it is the default posture of Connect.

Connect does not invent exactly-once. It borrows tools that Kafka already provides for producers and consumers.

Understanding these tools first makes the Connect features easy to follow. There are four building blocks, each solving a different part of the puzzle.

Idempotence: Solving Retry Duplicates

When a producer sends a message and does not get a reply, it retries. The first message may have actually arrived, so the retry would create a duplicate. With idempotence turned on, the producer includes a sequence number, and the broker accepts each number only once per partition. A repeated message is quietly recognized and ignored. This covers duplicates from retries within a single producer session, but not duplicates created when a whole application restarts and reproduces old data.

Transactions: Solving Multi-Write Atomicity

A transaction lets a producer say begin, write many messages, possibly to many partitions, and then commit or abort. If it commits, all writes become visible together. If it aborts, none of them do. The transaction is tracked by a transactional identifier, which the producer chooses and reuses across restarts. Kafka writes a small control record called a marker into every involved partition to record the outcome.

Read-Committed: Seeing Only Final Data

Consumers have an isolation setting. In the default mode, they read everything, including messages from transactions that are still open or later aborted. In the read-committed mode, they receive only messages from committed transactions, and they wait at a boundary called the last stable offset until open transactions finish. Without read-committed consumers, the benefit of transactions is invisible to readers.

Fencing: Solving the Zombie Problem

Suppose a worker becomes unresponsive, so the cluster starts a replacement task. Later the old task wakes up, unaware it has been replaced, and starts writing again. This is a zombie. Because the replacement has registered with the same transactional identifier and a higher epoch, the broker rejects every request from the old producer. The zombie is fenced out and cannot corrupt the data.

The big insight is to write records and bookmarks in the same Kafka transaction.

Exactly-once support for source connectors arrived in Kafka version 3.3. The insight behind it is elegant. Both the produced records and the source offsets ultimately live in Kafka topics. If they are written in one transaction, then they are committed together or aborted together. A crash can no longer leave records written but the bookmark stale, because both are in the same sealed envelope.

sequenceDiagram
  participant T as Source task
  participant P as Transactional producer
  participant D as Data topic
  participant O as Offsets topic
  T->>P: Begin transaction
  T->>P: Send batch of records
  P->>D: Write records
  P->>O: Write source offsets
  T->>P: Commit transaction
  P-->>D: Records become visible
  P-->>O: Bookmark becomes durable
        

A Worked Example With Simple Numbers

Let us follow an invented source connector that reads rows from a table using an ever-increasing row identifier as its source offset. The task starts with a committed bookmark of one thousand. It polls and receives rows one thousand one to one thousand fifty. It begins a transaction, writes those fifty records to the data topic, and writes the new bookmark, one thousand fifty, to the offsets topic. Then it commits.

Now suppose the worker crashes at three different moments in the next batch. In the first case, the crash comes after the task has written twenty records of the next batch but before it commits. The transaction never completes, so the twenty records are aborted, and the bookmark still says one thousand fifty. The restarted task reads rows one thousand fifty-one onward again and produces them in a new transaction. A read-committed consumer sees each row once.

In the second case, the crash comes right after the commit is requested. Kafka’s transaction coordinator either had recorded the commit decision, in which case the batch and its bookmark are both final, or it had not, in which case both are aborted. There is no third possibility where only one of them is final. In the third case, the crash comes just after a successful commit, so the restart begins from the new bookmark, and nothing is repeated or skipped.

Exactly-once is opt-in, and it is controlled at two levels: the whole worker cluster and each individual connector.

Worker-Level Setting

Every worker in the Connect cluster has a setting that controls whether exactly-once support for sources is active. It has three values. Disabled means the feature is off, which is the default. Preparing means the worker is getting ready for the feature, and is used during a rolling upgrade. Enabled means the worker fully supports exactly-once source tasks. The feature requires distributed mode, because it depends on the cluster’s coordination and fencing abilities.

Connector-Level Settings

Before creating an exactly-once connector, you can also ask the framework to validate its configuration. The validation step reports whether the connector supports exactly-once, whether it can define its own boundaries, and whether the settings are consistent. It is a good habit to validate first and create second.

A transaction boundary is the moment of commit, and choosing it shapes latency, throughput, and cost.

A boundary decides how many records share one transaction. Small transactions commit often, so consumers see data quickly, but each commit has a fixed cost. Large transactions amortize that cost, but readers must wait longer before data becomes visible, and a failure throws away more in-flight work.

Distributed systems replace workers often, so protection against old, forgotten tasks is essential.

Connect clusters are dynamic. Workers join and leave, tasks are reassigned, and connectors may be reconfigured to run more or fewer tasks. Every one of these events risks leaving a stale task alive somewhere, writing to Kafka based on an outdated assignment.

To prevent this, exactly-once source tasks use transactional identifiers that are derived from the cluster, the connector, and the task number. Before a new generation of tasks starts, the framework initializes producers with those same identifiers, which raises their epochs and immediately blocks any older producer using the same identifier. A zombie’s next write is rejected, and it learns it was fenced.

sequenceDiagram
  participant L as Leader worker
  participant Old as Old task
  participant B as Kafka broker
  participant New as New task
  Old->>B: Writes inside transaction
  Note over Old: Worker stalls
  L->>B: Fence old producers with same identifiers
  L->>New: Start new task generation
  New->>B: Begin transaction with higher epoch
  Old->>B: Wakes up and tries to write
  B-->>Old: Rejected as fenced
        

When the Number of Tasks Changes

If a connector is reconfigured to use fewer tasks, the identifiers of the removed tasks would never be reused, and old instances of them might still be alive. The framework therefore fences those identifiers too, before allowing the new configuration to run. This is one reason why exactly-once source connectors have a somewhat more careful startup than ordinary ones.

Permissions Needed for Fencing

Fencing and transactions require the right permissions in secured clusters. The identity used by the workers must be allowed to write and describe the transactional identifiers it uses, and to write to the offsets topic. Missing permissions are a very common cause of failed exactly-once startups, and the error messages usually point to authorization.

You should not flip a cluster-wide switch on a live system without a staged plan.

All workers in a Connect cluster must agree on how exactly-once tasks are fenced and started. If some workers are enabled while others are still disabled, the cluster could misbehave. The middle preparing state exists for exactly this reason, allowing a two-phase rollout with no downtime.

Rolling Back

Going back is also a staged process, moving from enabled to preparing and then to disabled, again worker by worker. Before rolling back, connectors that still require exactly-once should be reconfigured, or they will fail. Because downgrading affects duplicates, plan it with the same care as the upgrade.

The framework supplies the transactions, but the connector must be worthy of them.

A source connector cannot be exactly-once just because the framework is. The connector has to make certain promises, and the framework asks the connector to state them explicitly. This is why a connector may or may not support the feature, and why checking its documentation matters.

Transactional writing is only half of the guarantee. Readers must cooperate.

When a source task writes in transactions, records from open or aborted transactions physically exist in the topic. They are hidden only from consumers that ask for committed data. A consumer using the default setting will happily read records from aborted transactions, and will see the very duplicates that the transaction was meant to hide.

Side Effects of Read-Committed Reading

Sinks write to systems that Kafka cannot control, so the framework alone cannot promise exactly-once.

The transaction trick works for sources because both records and bookmarks are written into Kafka. In a sink, the records go to an outside system, such as a database, a search index, or object storage. Kafka’s transactions cannot include a write to an arbitrary outside system, so the sink’s two steps, writing to the destination and committing consumer offsets, cannot be made one atomic step by the framework.

Instead, exactly-once in a sink is achieved by making the destination side safe. There are three well-known strategies. Which one you can use depends on what the destination system offers.

flowchart TD
  K["Kafka topic records"] --> S["Sink task"]
  S --> A["Strategy A: idempotent writes by key"]
  S --> B["Strategy B: store offsets with data atomically"]
  S --> C["Strategy C: deterministic output names"]
  A --> X["Destination system"]
  B --> X
  C --> X
        

Choosing the Right Strategy

The choice depends on the destination. Start by asking whether the destination can recognize a repeated record on its own. Relational databases, key-value stores, and search indexes usually can, through primary keys or document identifiers, so idempotent writes are the natural first choice. It needs no special connector logic, and it survives every kind of replay, including ones caused by operators who reset offsets by hand.

If the destination supports local transactions but has no natural key in the data, storing offsets alongside the data becomes attractive. The price is custom logic, because the connector must read stored positions at startup and override the framework’s own starting point. If the destination is a file or object store, deterministic naming is usually the cleanest answer, provided the connector always cuts files at repeatable boundaries. A connector that cuts files by wall-clock time, for example, may produce differently sized files on a replay and leave overlapping content behind.

Small details in how sink tasks commit progress decide how many duplicates you will see.

The Sink Task Cycle

A sink task repeatedly receives a batch of records from the framework, writes them to the destination, and periodically reports how far it can safely commit. The framework then commits consumer offsets up to that point. The frequency of commits is controlled by a timer, so between commits, a crash replays everything since the last commit. Longer intervals mean better performance and more replay after a crash.

Sink tasks may also hold records in a buffer, for example to build larger files. A well-behaved task must report only offsets that have really been written, not offsets it has merely received. If it reported buffered records as done, a crash would lose them. This is a frequent source of subtle loss in custom connectors.

Rebalances

When the set of tasks changes, partitions are reassigned, and each sink task is told which partitions it lost and gained. A careful connector flushes and commits work for lost partitions before releasing them, and starts from the right position for gained partitions. Improper handling here creates duplicates or gaps around every rebalance, which is why cooperative rebalancing and careful hand-off logic matter.

Errors, Retries, and Dead-Letter Queues

Connect can be told to tolerate errors, retry failed operations, and send bad records to a dead-letter topic. These settings interact with exactness. If the framework skips or diverts a record, the sink no longer contains every input record, so completeness is deliberately given up in exchange for progress. A retry may repeat a write, which is harmless only with idempotent destinations.

A pipeline is only as exact as its least careful stage.

Real pipelines chain several stages. Data may flow from a database through a source connector into Kafka, through a stream processing application, into another topic, and finally through a sink connector into a warehouse. Each stage has its own guarantee, and the guarantee of the whole is the weakest of them.

flowchart LR
  DB["Source database"] --> SC["Source connector: transactional"]
  SC --> T1["Kafka topic"]
  T1 --> SP["Stream processor: transactional"]
  SP --> T2["Kafka topic"]
  T2 --> SK["Sink connector: needs idempotent destination"]
  SK --> DW["Data warehouse"]
        

Stronger guarantees are not free, and you should know the price before paying it.

Transactions add work. Every commit involves extra messages between the producer and the broker, and control records are written into every partition touched. Records are invisible to careful readers until commit, and tasks must fence and coordinate on startup. For most workloads the overhead is modest, but it is real, and it grows when transactions are tiny.

Estimating the Overhead for Your Workload

The cost of a transaction is roughly fixed per commit, so the number of commits per second matters more than the number of records. If a connector polls ten times a second and uses the per-poll boundary, that is ten transactions a second per task. With fifty tasks, the cluster handles five hundred commits a second, each touching the data topics and the offsets topic. That is usually fine, but it is very different from a single task using a one-second interval.

A practical way to estimate is to run a short load test with the boundary you plan to use, then compare throughput and latency with a non-transactional run. Check three numbers: the delay between a record being produced and being visible to committed readers, the broker’s request rates, and the size of the internal transaction state. If the delay is acceptable and the broker load is comfortable, keep the setting. If not, lengthen the boundary a little and test again.

When Exactly-Once Is Not Worth It

Not every pipeline needs the strongest promise. Metrics, logs, and clickstreams that are aggregated or sampled tolerate an occasional duplicate, and idempotent downstream logic can cheaply remove them. Paying the transaction cost for data that no one would miss is wasted effort. A helpful question is: what is the business cost of one duplicate, and one missing record?

The best way to trust a guarantee is to watch it survive the failures you fear.

Exactly-once is a property you must keep verifying, not a setting you set once.

Metrics to Watch

Reconciliation Checks

Independent counts are the ultimate test. Periodically compare the number of source records for a time window with the number of visible records in Kafka, and again with the number in the destination. Small differences during in-flight windows are normal, but persistent gaps or surpluses reveal a broken guarantee, often long before customers notice.

Security

Exactly-once adds security requirements. The workers need permission to use transactional identifiers and to write to the offsets topics, and those topics contain positions that reveal information about the source. Protect them like any sensitive topic, limit who can change them, and audit changes. An attacker who could rewrite offsets could force replays or skips, so treat offsets topics as control data.

Teams that run reliable pipelines share a handful of habits.

A short checklist prevents most painful surprises.

Duplicate data hurts in different ways in different industries.

For years, the standard advice was to accept at-least-once behavior in Connect and to deduplicate downstream. That advice still holds for many pipelines. Native support for exactly-once source connectors, added in Kafka 3.3, gave teams a stronger option where duplicates are expensive or hard to remove later. Connector projects have gradually adopted the feature, and the level of support varies by connector and version, so always confirm with the connector’s own documentation.

A guarantee that has never been attacked is only a hope.

Testing exactly-once means creating the very failures that would break it, and then checking the result with independent counts. It does not require complex tools. It requires a controlled source with a known number of records, a running pipeline, and a plan for causing trouble.

No. The default behavior is at-least-once. Exactly-once for source connectors must be enabled at the worker and connector level, and the connector itself must support it.

Not through the framework alone. Sinks write to outside systems, so exactness comes from idempotent writes, from storing offsets together with data, or from deterministic output naming.

Because records from aborted transactions still exist in the topic. Only committed reading hides them. Without it, readers see the duplicates that transactions were designed to remove.

The feature is designed for distributed mode, because it relies on cluster coordination and fencing between workers. Standalone mode is meant for testing and simple cases.

Creating the connector fails with a validation error. That is the intended behavior, since it prevents a silent downgrade to weaker guarantees.

An old task instance that was replaced but is still running and does not know it. Fencing with transactional identifiers and epochs makes the broker reject its writes.

Use per-poll transactions for simplicity and low delay, interval-based ones for high volume sources that tolerate delay, and connector-defined ones when the source has natural atomic units and the connector supports them.

With committed reading, consumers cannot pass an open transaction. A stuck or long transaction can hold back readers until it commits or times out. Check transaction age and task health.

Usually modestly. The cost is higher when transactions are tiny, and visibility is delayed until commit. Measure with your own workload and tune the boundary accordingly.

Not exactly. It combines no loss with no visible duplicates, but only within its defined scope. Skipped records from error tolerance, or data that never reached Kafka, are outside that promise.

For critical data, keeping idempotent downstream logic is wise as a second line of defense. It protects you from connector bugs, configuration mistakes, and stages that cannot offer transactions.

Then exactly-once cannot be guaranteed for that source, because the replay is not the same data. The source must provide a stable, repeatable position for the guarantee to hold.

Verify versions and permissions, restart workers one by one into the preparing mode, verify, restart them again into the enabled mode, and then move connectors to the required level one at a time.

Load a source with numbered records, cause random failures during the run, read the results with committed reading, and confirm every number appears exactly once. Repeat with the destination for sinks.

Yes, once the workers are enabled, by updating the connector’s configuration. Plan for a short restart of its tasks, validate first, and expect that records already written before the change may include earlier duplicates.

No. Transactions work across any number of partitions and topics. They do add markers to each partition touched, so extremely wide transactions across many partitions cost more per commit.

Kafka Connect moves data between Kafka and the outside world, and its default posture is at-least-once, because progress is saved after work is done. Duplicates appear when a crash falls between the work and the bookmark. Exactly-once removes that gap by making the two inseparable, using idempotent producers, transactions, committed reading, and fencing.

For source connectors, recent Kafka versions write records and source offsets in a single transaction, provided the cluster runs in distributed mode, the workers are enabled in a staged rollout, and the connector supports the feature and reads from a replayable source. For sink connectors, exactness has to be built on the destination side, through idempotent writes, offsets stored with the data, or deterministic outputs. Across a whole pipeline, the weakest stage decides the outcome, so every stage must be examined and tested.