Before learning the tools, it helps to see the exact problems they were built to solve.

Retries create duplicates

When a producer sends a batch of messages to a broker, it waits for an acknowledgment. If the acknowledgment does not arrive, the producer cannot know what happened. Perhaps the broker never received the batch. Perhaps the broker stored it, but the acknowledgment was lost on the way back. The safe choice for the producer is to send the batch again, because losing data is worse than repeating it. But in the second case, the broker now stores the same batch twice.

This is the classic trade-off of distributed systems. A sender that never retries can lose messages. A sender that always retries can create duplicates. Kafka delivery guarantees are usually described by three levels: at most once, where messages may be lost but never repeated; at least once, where messages are never lost but may be repeated; and exactly once, where each message takes effect one time only.

Crashes and restarts add a second problem

Duplicates from retries happen inside a single running producer. A second family of problems appears when a producer program crashes or restarts. A new copy of the program does not remember what the old copy sent, so from the broker’s view it is a stranger. If the old copy had sent half of a group of related messages before dying, the new copy must decide whether to resend them, and any decision can lead to loss or repetition.

Zombies make everything worse

A zombie is a producer instance that the system believes is dead but that is actually still alive, perhaps frozen by a long pause or cut off by a network problem. When it wakes up, it continues sending messages, unaware that a replacement has taken over. Two producers writing the same logical stream of results can corrupt the outcome, for example by writing results twice or by writing results based on outdated input.

Why not simply remove duplicates downstream?

A natural question is whether every reader could remove duplicates on its own, for example by remembering the identifiers of messages it has already handled. This works for small systems, but it scales badly. Every consumer must store and search a growing list of identifiers, agree on what an identifier is, and keep that list safe across its own restarts. If one consumer forgets to do it, or does it slightly differently, results diverge between systems that should agree.

Handling the problem at the source is cleaner. When the broker discards a repeated batch at the moment it arrives, every reader benefits automatically, and no reader needs special logic. Downstream deduplication still has a place, especially for actions outside Kafka, but it works best as a second line of defense rather than the only one.

A dozen terms appear throughout this topic, and knowing them well makes everything after it easy.

Common misunderstandings about these numbers

Several wrong ideas circulate about these values. Some people think the producer ID identifies a machine, but it identifies a producer session, and one machine can run many. Some think the epoch counts messages, but it counts changes of ownership. Some think sequence numbers are global, but they are kept separately for every partition and for every producer identity. And some believe the transactional ID is only a label for logs, when in fact the cluster uses it as the key of its coordination table.

Getting these straight early saves confusion later. When a message in a log mentions a producer ID and an epoch, read it as “this session of this identity, in its current generation.” When you see a sequence number, read it as “the position of this batch in the stream this identity has sent to this one partition.”

Where these values live

Every batch of messages written by a producer has a small header. Inside that header are three values: the producer ID, the producer epoch, and the sequence number of the first message in the batch. These travel with the data all the way into the log on disk, which means every broker that stores the partition can check for duplicates and outdated senders, and can continue to do so after a leadership change.

The first defense handles duplicates from retries, and it is switched on by default in modern Kafka.

How idempotence works

When idempotence is enabled, the producer asks the cluster for a producer ID at startup. From then on, for every partition it writes to, it keeps a counter and gives each batch the next number. The broker that leads the partition remembers the latest sequence numbers it has accepted for each producer ID. When a batch arrives, the broker compares its sequence number with what it expects.

If the number is exactly the next expected one, the broker stores the batch. If the number is one the broker has already stored, the batch is a duplicate from a retry, so the broker quietly discards it and returns a successful answer, as if it were stored. If the number is beyond the next expected one, something was lost or reordered, and the broker returns an out-of-order error so the producer can react.

sequenceDiagram
    participant P as Producer
    participant B as Partition Leader
    P->>B: "Batch with producer ID 7 and sequence 41"
    B->>B: "Store batch, expect sequence 42 next"
    B--xP: "Acknowledgment lost on the way back"
    P->>B: "Retry the same batch with sequence 41"
    B->>B: "Sequence 41 already stored - discard duplicate"
    B-->>P: "Success acknowledgment"
    P->>B: "Next batch with sequence 42"
    B->>B: "Store batch, expect sequence 43 next"
    B-->>P: "Success acknowledgment"
        

The settings that come with it

Idempotence works only with certain settings. The producer must wait for all in-sync replicas to acknowledge, retries must be allowed, and the number of unacknowledged requests per connection must be small, up to five. The broker keeps a short memory of the most recent batches for each producer and partition, which is why this limit exists: the memory must be large enough to recognize any batch that could still be in flight. In modern versions, idempotence is on by default, and the client refuses conflicting settings so that the guarantee cannot be silently broken.

The broker’s memory of recent batches

The partition leader cannot remember every batch ever sent, so it keeps a small window for each producer and partition: the sequence range of the last few batches it accepted. That is enough because a producer with a limited number of unacknowledged requests can only ever retry something within that window. Anything older has certainly been acknowledged already. This memory is rebuilt from the log itself after a restart or a leader change, since the identity and sequence values are written into every batch.

The window also explains a subtle rule. If a producer with idempotence enabled were allowed a large number of requests in flight, a retried batch could fall outside the window and no longer be recognized. Limiting the in-flight requests keeps the guarantee sound. It also preserves the order of messages, because a retried batch is slotted into its exact place by its sequence number, not appended at the end.

What idempotence does not cover

Idempotence protects only one producer session against duplicates caused by its own retries. If the program restarts, the cluster hands the new session a completely new producer ID, and the sequence counters start over. The broker has no way to connect the old session with the new one, so messages that the old session sent and the new session resends are stored twice. Idempotence also cannot make several writes to different partitions succeed or fail together.

To survive restarts, a producer needs an identity that outlives any single program run.

The missing piece: an identity that persists

The producer ID is temporary. It belongs to one session. What is missing is a durable name that says “this is the same logical producer as before, even though the program restarted.” That name is the transactional ID. The application picks it, keeps it the same across restarts, and gives it to the producer when it is created.

When a producer with a transactional ID starts and calls the initialization method, the cluster looks the name up. If the name is new, a new producer ID is assigned along with a starting epoch. If the name already exists, the cluster reuses the same producer ID but increases the epoch. The increase is the key trick: it tells everyone that a new instance now owns this identity, and that any request carrying the older epoch must be refused.

flowchart TD
    A["Producer starts with transactional ID payments-writer-1"] --> B["Calls initialize transactions"]
    B --> C{"Is the ID already known to the coordinator?"}
    C -->|"No"| D["Assign new producer ID with epoch 0"]
    C -->|"Yes"| E["Reuse producer ID and increase epoch"]
    E --> F["Finish or abort any open transaction of the older instance"]
    D --> G["Ready to begin transactions"]
    F --> G
        

What initialization also cleans up

Initialization does more than hand out numbers. If the previous instance of this identity left a transaction half-finished, the coordinator resolves it before allowing the new instance to proceed. A transaction that was already on its way to commit is completed. A transaction that was still open is aborted. This gives the new instance a clean starting point, with no leftover half-written work visible to careful readers.

Why a central coordinator decides ownership

One might ask why the partition leaders do not decide ownership among themselves. The answer is that a producer may write to many partitions, each with a different leader, and ownership must be one consistent decision, not many separate ones. A single coordinator per identity gives one place where the epoch is raised, one place where the transaction state is recorded, and one place that decides commit or abort. The partition leaders simply learn the outcome and enforce it on their own data.

This division of labor keeps the data path fast. Data batches go straight from the producer to partition leaders without passing through the coordinator, and the coordinator handles only small state messages. It also keeps the design understandable: identity and decisions live in one place, and data lives in another.

Why atomic groups of writes matter

Consider a program that reads an order event, calculates a result, writes the result to one topic, writes an audit record to another, and saves its reading position. If it crashes between these steps, the outside world sees a half-finished story: a result without an audit record, or work done but not remembered. A transaction groups all these writes into a single unit. Either every piece becomes visible together, or none of it does.

The transactional ID looks like a simple text setting, but choosing it well is one of the most important design decisions in a transactional application.

What it is and what it is not

The transactional ID is a text name supplied by the application through the setting transactional.id. It identifies a logical producer, meaning a role in your system that should have only one active instance at a time. It is not a session identifier, not a unique random value for each run, and not a user or account name. It is more like the name of a desk in an office, which different people may occupy over time but never at the same moment.

Rules for a good transactional ID

Naming strategies compared

Naming in containers and orchestrators

Container platforms make naming harder, because instance names are often random and short-lived. A default pod name that changes on every restart would behave like a random transactional ID and lose all the benefits. Prefer platforms and settings that give each replica a stable ordinal identity, and build the transactional ID from that ordinal together with the application name.

Autoscaling adds another wrinkle. When the number of replicas shrinks and later grows, the same ordinals return, which is fine, because each returning name simply raises the epoch. But if your naming scheme includes the total replica count, resizing changes every name and breaks continuity. Keep the naming scheme independent of how many replicas are currently running, and make sure the mapping between work and names does not shift when the fleet changes size.

What the coordinator remembers about a name

For each transactional ID, the coordinator stores the producer ID, the current epoch, the state of the current transaction, the list of partitions involved in it, and the timeout it was given. When a name has not been used for a long time, controlled by the setting transactional.id.expiration.ms, which defaults to seven days, the coordinator forgets it. Using the name again afterwards is treated as a brand new identity. This cleanup keeps the coordinator’s memory from growing forever, but it also means a producer that pauses for longer than this window loses its history.

The epoch is a tiny number with a big job: it decides who is allowed to speak.

What an epoch is

The producer epoch is a small whole number that travels with the producer ID in every batch and every request to the transaction coordinator. It starts at a low value when an identity is first created and increases whenever the identity changes hands. The rule for everyone who receives a request is simple: if the epoch in the request is lower than the epoch I know about, refuse the request.

When the epoch increases

The most important moment is initialization. Each time a producer with a given transactional ID initializes, the epoch for that name goes up. This immediately outdates every earlier instance. The epoch may also increase in some recovery situations, for example when the coordinator times out a transaction and must ensure the producer that owned it cannot continue, or when a producer recovers from certain errors and asks for a fresh start. Newer Kafka versions also refine when epochs change, including at transaction boundaries, to close subtle gaps, so it is worth reading the release notes for the version you run.

Epoch versus sequence number

Beginners often mix up the epoch and the sequence number. They work at different levels. The sequence number counts batches inside one producer ID for one partition, and it exists to detect repeats and gaps. The epoch counts generations of ownership of a producer ID, and it exists to detect outdated owners. A batch can have a correct sequence number and still be rejected because its epoch is old.

Fencing is the practical payoff of transactional IDs and epochs, and it is easiest to understand as a short story.

The story of a frozen instance

Instance A of a stream application is processing messages under the transactional ID of a certain worker. It begins a transaction, writes some results, and then its machine freezes because of a long pause or a network cut. The orchestration system decides A is gone and starts instance B with the same transactional ID. B initializes, and the coordinator raises the epoch, aborts A’s unfinished transaction, and lets B continue from the last committed reading position.

Later, A wakes up. It has no idea anything happened. It tries to write another message or to commit its transaction. But every request it sends carries the old epoch. The coordinator and the partition leaders compare that epoch with the newer one and refuse. A receives an error telling it that it has been fenced. A must stop, because it is no longer the rightful owner of this identity.

sequenceDiagram
    participant A as Instance A - epoch 4
    participant Co as Transaction Coordinator
    participant B as Instance B - new
    A->>Co: "Begin work, write results in transaction"
    Note over A: "Instance A freezes or is cut off"
    B->>Co: "Initialize with the same transactional ID"
    Co->>Co: "Raise epoch to 5 and abort A's open transaction"
    Co-->>B: "Producer ID with epoch 5"
    B->>Co: "Normal transactions with epoch 5"
    Note over A: "Instance A wakes up"
    A->>Co: "Commit request with epoch 4"
    Co-->>A: "Rejected - producer is fenced"
        

What the fenced instance sees

The client library reports a fencing error to the application. This error is fatal for that producer object. The correct reaction is to close the producer and stop doing work with that identity, letting the newer instance carry on. In a cluster of workers, a fencing error is also a signal that something is wrong in the orchestration, because a replacement was started while the original was still alive. Frequent fencing errors deserve investigation, since they reveal pauses, network problems, or a misconfigured deployment.

How zombies come to exist

Zombies are more common than most teams expect. Long garbage collection pauses freeze a program for seconds or minutes. A virtual machine may be paused during host maintenance. A network partition can cut an instance off from the cluster while it keeps running. A slow storage device or a starved processor can leave a program alive but barely moving. Orchestrators that replace instances after a health check failure may start the new instance before the old one has really stopped.

In every one of these cases the old instance does not know it has been replaced. That is why fencing has to be enforced by the cluster and not by trusting the instance to notice. The instance’s own view of time and health cannot be relied upon, but a number that only goes up, held by the coordinator, can.

Fencing needs both sides to check

Fencing works because more than one component enforces it. If only the coordinator checked epochs, a zombie could still send data batches directly to partition leaders during the small window before it is discovered. So the partition leaders also record the latest epoch they have seen for each producer ID and refuse batches with older epochs. This double check means that once the new epoch has been established and seen by a partition leader, no outdated write can enter that partition.

Just as consumer groups have a coordinator, transactional producers have one too, found by a similar trick.

The hash and the transaction log

Kafka keeps an internal topic called __transaction_state. By default it has fifty partitions, controlled by transaction.state.log.num.partitions. To find the coordinator for a transactional ID, Kafka hashes the name, divides by the number of partitions of this internal topic, and takes the remainder. The broker that currently leads that partition is the coordinator for that transactional ID.

A producer that calls the initialization method first asks any broker for the coordinator of its transactional ID, using a lookup request of the transaction type. It then talks to that broker for everything about transaction state. Data messages still go directly to the leaders of the data partitions, so the coordinator never handles your payload.

flowchart LR
    P["Transactional producer"] -->|"Find coordinator for my transactional ID"| B1["Any broker"]
    B1 -->|"Hash of ID modulo partition count"| T["Partition of the transaction log"]
    T -->|"Leader of that partition"| C["Transaction coordinator"]
    C -->|"Address returned"| P
    P -->|"Init, add partitions, commit, abort"| C
    P -->|"Data batches"| L["Leaders of data partitions"]
        

Durability of transaction state

Every change of state, such as a transaction starting, adding a partition, or preparing to commit, is written as a record into the transaction log before the coordinator acts on it. The log is replicated like any topic. Its replication factor is set by transaction.state.log.replication.factor, which defaults to three, and it demands a minimum in-sync count set by transaction.state.log.min.isr, which defaults to two. If the log cannot meet its minimum, the coordinator cannot safely record changes, and transactional operations fail even if ordinary topics still work.

Failover of the coordinator

If the broker acting as coordinator fails, a new leader is elected for the log partition. The new coordinator reads the partition and rebuilds the table of transactional IDs, epochs, and transaction states. Transactions that were in the middle of committing are completed from the recorded state. Producers receive an error saying the coordinator has moved or is loading, look it up again, and continue. Because the state is in the log, no transaction is lost or half-decided by a coordinator crash.

Walking through one transaction from start to finish shows how every earlier idea fits together.

sequenceDiagram
    participant P as Producer
    participant Co as Transaction Coordinator
    participant L1 as Partition Leader 1
    participant L2 as Partition Leader 2
    P->>Co: "Initialize with transactional ID"
    Co-->>P: "Producer ID and epoch"
    P->>Co: "Register partition 1"
    P->>L1: "Data batches"
    P->>Co: "Register partition 2"
    P->>L2: "Data batches"
    P->>Co: "End transaction - commit"
    Co->>Co: "Record decision in transaction log"
    Co->>L1: "Write commit marker"
    Co->>L2: "Write commit marker"
    Co->>Co: "Record completion"
    Co-->>P: "Commit confirmed"
        

Batching inside a transaction

Inside a transaction, the producer still groups messages into batches exactly as usual, using its buffer, batch size, and linger time settings. The commit call flushes anything still waiting in the buffer before asking the coordinator to finish the transaction, so no message is left behind. This means a transaction can hold many batches per partition, and the commit waits for all of them to be acknowledged. If any batch fails permanently, the commit fails, and the application must abort.

Because the commit waits for every batch, slow partitions dominate commit time. A transaction that touches many partitions is only as fast as its slowest partition. Keeping the set of partitions per transaction reasonable, and spreading heavy partitions across healthy brokers, keeps commit times steady.

Making writes to many partitions succeed together requires a careful two-step decision process.

Phase one: record the decision

When the producer asks to commit, the coordinator changes the state of the transaction to prepare-commit and writes this decision into the transaction log. This write is the point of no return. Once it is durable, the transaction is guaranteed to be committed, even if the coordinator fails immediately afterward, because the next coordinator will read the recorded decision and finish the job. If the producer asked to abort, the same steps apply with prepare-abort.

Phase two: write markers to every partition

Next, the coordinator sends a request to the leader of every partition that took part in the transaction, asking it to write a control marker. A control marker is a special record, invisible to normal application code, that says this transaction is committed or aborted. It sits in the log among the data batches, so the position of the marker relative to the data is exact.

When every partition leader has written its marker, the coordinator changes the state to complete-commit and records this in the log. The transaction is now fully finished, and the coordinator can allow the same transactional ID to start a new one. The producer usually receives its answer for the commit call once the decision is safely recorded, and the markers are completed in the background before the next transaction can begin.

flowchart TD
    E["Empty"] -->|"First partition registered"| O["Ongoing"]
    O -->|"Commit requested"| PC["Prepare commit"]
    O -->|"Abort requested or timeout"| PA["Prepare abort"]
    PC -->|"Markers written everywhere"| CC["Complete commit"]
    PA -->|"Markers written everywhere"| CA["Complete abort"]
    CC -->|"Next transaction begins"| O
    CA -->|"Next transaction begins"| O
        

Why markers are used instead of rewriting data

Kafka logs are append-only, and this property is central to their speed and safety. Rewriting or deleting records inside a log to cancel an aborted transaction would break that property, slow everything down, and disturb readers who are partway through the file. Instead, the log records the outcome as one more small entry placed after the data. Readers combine the data and the marker to learn what to keep, so nothing already written is ever changed.

This also makes the outcome permanent and visible to every replica. Because the marker is replicated like any other record, all copies of the partition agree on what was committed and what was aborted, and a new leader can serve readers correctly from the first moment it takes over.

Transactions are only useful if readers can choose to see committed data alone.

The isolation setting

Consumers choose between two modes with the setting isolation.level. In the default read-uncommitted mode, a consumer sees every data record as soon as it is replicated, including records from transactions that are still open or that will later abort. In read-committed mode, the consumer sees only records from committed transactions and from non-transactional writes, and it never sees records of aborted transactions.

The last stable offset

To support read-committed mode, each partition tracks the last stable offset. This is the position below which no transaction is still open. A read-committed consumer never reads beyond it. If a transaction that started early is still open, the last stable offset stays just before its first record, even if many newer messages have been written by other producers. Those newer messages are held back from read-committed consumers until the open transaction ends.

flowchart LR
    A["Offset 10 committed data"] --> B["Offset 11 committed data"]
    B --> C["Offset 12 open transaction begins"]
    C --> D["Offset 13 other producer data"]
    D --> E["Offset 14 open transaction data"]
    E --> F["High watermark - replicated up to here"]
    B -.->|"Last stable offset stops before offset 12"| C
        

Aborted data and control records

Aborted transactions leave their data in the log. Read-committed consumers skip it. The broker tells the consumer which ranges belong to aborted transactions, and the consumer’s library filters them out before handing records to the application. Control markers are also filtered out, so application code never sees them, although the offsets they occupy still count. This is why offsets in a transactional topic can appear to skip a number: the missing number is a marker.

Tools and connectors that read transactional topics

Not every reader is a hand-written consumer. Command line consumers, connectors that copy data to other systems, and monitoring tools all read topics too, and each has its own isolation setting. A connector left in the default read-uncommitted mode will happily copy data from aborted transactions into a database or a warehouse, creating exactly the inconsistencies transactions were meant to prevent.

Audit every reader of a transactional topic and confirm that it uses read-committed mode wherever correctness matters. Remember also that lag numbers shown by tools can differ between modes, since one measures against the high watermark and the other against the last stable offset. When comparing lag values, make sure both are computed the same way.

Consequences for lag and latency

Because read-committed readers wait for transactions to end, the end-to-end delay of messages depends on how long transactions stay open. Short transactions mean small delays. Long or stuck transactions cause lag to grow even though brokers and consumers are healthy. When you see read-committed consumers falling behind while everything else looks normal, look for a transaction that has been open for a long time.

The most common use of transactions is a pipeline that reads from Kafka, transforms data, and writes back to Kafka.

The pattern

A program reads a batch of input messages, processes them, and writes the outputs. To make this exactly-once, it wraps two actions in one transaction: writing the outputs and committing the input reading positions. The positions are sent through the producer to the coordinator rather than committed by the consumer directly, so that they become visible only if the output is committed. If anything fails, the whole transaction aborts, the outputs are hidden, the reading position does not move, and the work is repeated safely on the next attempt.

flowchart TD
    R["Read input batch"] --> P["Process records"]
    P --> B["Begin transaction"]
    B --> W["Write outputs to result topics"]
    W --> O["Add input offsets to the transaction"]
    O --> C{"Commit succeeds?"}
    C -->|"Yes"| D["Outputs visible and input progress saved together"]
    C -->|"No"| A["Abort - outputs hidden, input will be read again"]
    A --> R
    D --> R
        

Fencing by group generation

In the early design, each input partition needed its own transactional ID and producer, so that a zombie processing the same partition would be fenced by epoch. This worked but created many producers and heavy overhead. A later improvement lets the transaction carry the consumer group’s membership information when sending input positions. The coordinator then also checks the group generation, rejecting positions from a member that no longer owns the partition. This allows one producer to serve many partitions while still fencing zombies, greatly reducing the number of producers needed.

How stream processing libraries use this

Kafka Streams offers an exactly-once processing mode that hides all these details. It derives transactional IDs itself, runs transactions on a timer controlled by the commit interval, and handles fencing and recovery internally. In its newer mode, one producer per processing thread is enough instead of one per task. Applications simply choose the processing guarantee and let the library do the rest, which is far safer than writing the same logic by hand.

What exactly-once does and does not cover

The guarantee covers reading from Kafka topics and writing to Kafka topics, including saved positions. It does not cover side effects outside Kafka. If your program also writes to a database or calls a payment service, that action is not part of the transaction. To stay correct, external actions must be repeatable safely, for example through unique request keys, or the results must be written to Kafka and moved to the outside system by a separate, carefully designed step.

A transaction that never ends is a problem for everyone waiting behind it, so Kafka uses several clocks to clean up.

The transaction timeout

Each producer states how long its transactions may stay open, using transaction.timeout.ms, which defaults to one minute on the client. The broker enforces an upper limit with transaction.max.timeout.ms, which defaults to fifteen minutes. A producer that asks for more than the limit is refused at initialization. The coordinator regularly scans for transactions that have been ongoing longer than their timeout. When it finds one, it aborts it and raises the epoch of that transactional ID, so the producer that owned it, if still alive, is fenced.

This is a safety net for producers that crashed and never returned. Without it, the first record of an abandoned transaction would hold the last stable offset back forever, and every read-committed consumer of that partition would be stuck.

Choosing a good timeout

The timeout should be long enough for the longest normal transaction, including pauses, but not much longer. A timeout that is too short aborts healthy transactions during a busy moment and wastes work. A timeout that is too long leaves the last stable offset stuck for a long time after a crash. A good starting point is a small multiple of the longest normal transaction time, and it should always be less than the broker maximum.

Expiration of idle identities

Two more clocks control how long the cluster remembers identities. The coordinator forgets a transactional ID that has been idle longer than transactional.id.expiration.ms. Partition leaders forget the state of idle producer IDs after a period set by producer.id.expiration.ms, which defaults to one day. If a producer comes back after its state was forgotten, the cluster may report that the producer is unknown, and newer clients handle this by requesting a clean start and continuing. Understanding these clocks helps explain surprising errors after long idle periods.

Hanging transactions

A hanging transaction is one that stays open on a partition without the coordinator knowing or ending it. It can happen when a delayed data batch from a fenced or aborted producer arrives at a partition leader after the transaction was already finished. The batch looks like the start of a new open transaction, but no coordinator will ever close it. Over time, the last stable offset stops moving, and read-committed consumers stall.

Newer versions address this by making the partition leader verify with the coordinator’s records, through registration checks, that a partition truly belongs to the transaction before accepting the data, and by tightening epoch changes at the end of each transaction. Older clusters may still see hanging transactions, and administrators have a tool to find and abort them.

The value of a design shows most clearly when parts of it fail, so here are the common cases.

Why partition leader changes are safe

The producer ID, epoch, and sequence number are stored inside the batches themselves, and batches are replicated to followers. When a follower becomes the new leader, it rebuilds its memory of each producer’s latest sequence numbers by reading the log. This is why duplicate detection continues to work after leadership moves, and why the design does not depend on any single broker’s private memory.

Why coordinator failure is safe

The coordinator writes each decision to its replicated log before acting. A failure between the write and the action leaves a recorded intention. The next coordinator sees the intention and continues. A failure before the write leaves no record, so the producer, which never received confirmation, simply retries the request. In both cases, the state is consistent and the outcome is either complete or cleanly repeated.

Testing failures on purpose

The best way to trust these mechanisms is to watch them work. In a test environment, run a transactional pipeline under steady load and then break it in controlled ways. Kill a producer process in the middle of a transaction and confirm that the replacement resolves it and no partial output becomes visible to read-committed readers. Freeze a process for longer than the session and transaction timeouts, let a replacement take over, and check that the frozen one is fenced when it wakes. Restart the coordinator broker during a commit and verify that the transaction still finishes exactly once.

Record what you observe in each case: how long the pause lasted, what errors the application saw, whether any duplicates or gaps appeared, and how long the readers were held back. These rehearsals reveal weak spots in error handling, in timeouts, and in deployment procedures long before a real incident does.

A transactional producer must react differently to different kinds of errors, and confusing them causes subtle bugs.

Three families of errors

Errors fall into three groups. Fatal errors mean the producer can no longer be used, and the only safe action is to close it. Abortable errors mean the current transaction failed, but the producer is fine, so the application should abort the transaction and start a new one. Retriable errors are handled inside the client library, which retries automatically until the delivery timeout runs out.

The abort-and-rewind rule

After aborting a transaction in a read-process-write loop, the program must also rewind its consumer to the last committed position. Otherwise it would continue from where it stopped reading, skipping the records of the aborted batch, and lose them. The safe pattern is to seek each assigned partition back to its last committed offset, or to restart the consumer, so the aborted batch is read again.

Do not ignore fatal errors

Continuing to use a producer after a fatal error can produce confusing follow-up failures or, worse, hide the fencing that the design intended. Treat these errors as a signal to stop and investigate why the identity changed hands. Log the transactional ID, the epoch if available, and the time, so that you can line the event up with restarts or pauses.

Logging that makes incidents easier

When a transactional application misbehaves, the first questions are always the same: which identity, which epoch, which state, and when. Make the application log its transactional ID at startup, log every fencing or abort event with a timestamp, and include the transaction outcome and size in periodic summaries. With this information, an engineer can line the events up with deployments, pauses, and broker changes in minutes instead of hours.

Keep such logs free of message contents, since they may contain private data. The identity and timing information is enough for diagnosis, and it is safe to share widely across teams during a review.

Transactions add coordination, so it is worth understanding what they cost and how to keep the cost small.

Where the cost comes from

A transaction adds a few extra round trips: registering partitions with the coordinator, sending the end request, and writing markers to every involved partition. The markers themselves are small, but each one is a replicated write. The producer also waits for the commit to be recorded before starting the next transaction. Finally, read-committed consumers wait for transactions to end, which adds delay to visibility.

Transaction size is the main dial

The fixed cost of a transaction is roughly the same whether it holds ten messages or ten thousand. So very small transactions waste effort, while very large transactions hold the last stable offset back longer and risk timeouts and heavy rework if they abort. The best size balances throughput against visibility delay. In stream processing, the commit interval controls this directly, and a shorter interval means lower delay but more transactions per second.

Other factors

The replication settings of data topics and of the transaction log affect commit latency, since markers and state changes must be replicated. Slow or unhealthy brokers show up as slow commits. Large messages increase the time between registering and committing. And when the coordinator broker is busy with many active identities, commit times rise. Measure commit latency as a first-class metric, not only send latency.

A short tuning checklist

Start by measuring the current commit time and transaction rate before changing anything. If commits are frequent and tiny, lengthen the interval between transactions until visibility delay is just acceptable. If commits are slow, check partition leaders and the transaction log for unhealthy brokers before tuning the client. If read-committed consumers lag, compare transaction lengths against the delay the business can accept. And if fencing errors occur, look at pauses and deployments rather than at transaction settings.

Change one thing at a time and watch for at least a full traffic cycle. Transactional systems have several interacting clocks, and a change that helps at midday can hurt at peak load. Recording the settings and results of each experiment builds a small body of knowledge that is far more useful than folklore.

A transactional system needs its own kinds of visibility, both in the applications and on the brokers.

Client-side signals

Transactional producers publish timing metrics for each phase: initialization, beginning a transaction, sending consumed offsets, committing, and aborting. These are cumulative time values, so dividing changes by the number of calls shows average duration. A rise in commit time points at slow markers or a busy coordinator. A rise in initialization time points at coordinator trouble, such as a loading state or an unhealthy transaction log. Also count fencing errors and aborts over time, since a burst of either is an early warning.

Broker-side signals

On the brokers, watch the health of the internal transaction log like any critical topic: under-replicated partitions, offline partitions, and leadership balance. Watch request timing for the transaction related requests and the metrics of the marker channel that the coordinator uses to reach partition leaders. Also watch the last stable offset of busy partitions compared with their high watermark. A large and growing gap means an open transaction is holding readers back.

The command line for transactions

Kafka ships an administrative tool for transactions. It can list the transactional IDs the cluster knows, describe one to show its producer ID, epoch, state, timeout, and the partitions involved, find hanging transactions on given partitions, and abort a hanging transaction after careful review. Using it turns an invisible problem into a concrete answer, such as which identity owns the oldest open record on a stuck partition.

A compact reference of the settings that shape identity, epochs, and transactions.

Transactional identities are powerful, so who may use them and how the cluster is run both matter.

Access control for transactional identities

Kafka treats each transactional ID as a resource that can be protected by access rules. A producer needs permission to write using its transactional ID, and a permission to describe it, in addition to the usual permission to write to the data topics. Grant these narrowly, by application, so that one application cannot use another’s identity. This matters because using someone else’s transactional ID would fence their producer, which is an easy and damaging way to disrupt a running pipeline, either by accident or on purpose.

Also protect the consumer side. A group that reads transactional data needs the normal read permissions on topics and its group. Reading in read-committed mode needs no special permission, but the transaction administration tool that can abort transactions should be restricted to a small group of trusted operators, since aborting a transaction discards data from the view of committed readers.

Sizing and capacity

The coordinator keeps a record in memory for every active transactional ID and writes state changes into the transaction log for every transaction. A very large number of identities, or a very high transaction rate, increases the load on the coordinators, and because coordinators are chosen by hash, some brokers may carry more than their share. Keep the number of identities bounded, avoid patterns that create new names constantly, and balance the leadership of the transaction log partitions across brokers.

Upgrades and version changes

Transaction behavior has been refined across releases, including how epochs change at transaction boundaries and how partitions are registered and verified. Upgrade brokers before enabling newer client behavior, follow the documented order for rolling upgrades, and test on a non-production cluster first. During an upgrade, expect brief coordinator movements, and confirm that transactional applications handle temporary coordinator errors properly.

Operational habits

These mistakes appear again and again, and each has a straightforward alternative.

These illustrative stories show how the ideas combine in practice. They describe common patterns and are not tied to any specific company.

flowchart LR
    A["Duplicates after restart"] --> B["Check for a stable transactional ID and read committed"]
    C["Constant fencing errors"] --> D["Check for shared names and overlapping deployments"]
    E["Readers stuck on a partition"] --> F["Check open transactions and the stable offset gap"]
    G["Many pipelines fail together"] --> H["Check the coordinator broker and the transaction log"]
        

Short, direct answers to the questions that come up most often.

The producer ID is a number the cluster assigns to a session. The transactional ID is a name the application chooses and keeps across restarts. The cluster maps the name to a producer ID and keeps that mapping between sessions.

It marks the generation of ownership of a producer identity. Whenever a new instance takes over a transactional ID, the epoch goes up, and requests carrying an older epoch are refused. This is how zombies are fenced.

No. It removes duplicates caused by retries within one producer session. Restarts, atomic writes across partitions, and coordinated offset commits need transactions.

By hashing the transactional ID onto a partition of the internal transaction log. The broker that leads that partition acts as the coordinator for that identity.

Commit and abort markers occupy offsets but are hidden from applications. The missing numbers are these markers.

It stays in the log but is hidden from read-committed consumers, which skip it. Read-uncommitted consumers still see it.

An open transaction may be holding back the last stable offset. Look for a long or hanging transaction, and check the transaction timeout.

Not usefully. The second to initialize raises the epoch and fences the first, and if both keep restarting they will fence each other in a loop.

Close the producer and stop working with that identity. A newer instance owns it now, and you should investigate why a replacement started while this instance was alive.

No. They cover reading from and writing to Kafka, including saved reading positions. External systems need their own protection, such as unique request keys.

Set it a little above the longest normal transaction, and below the broker maximum. Too short causes needless aborts, and too long leaves readers stuck after a crash.

They add a fixed cost per transaction and add delay for read-committed readers. Choosing a sensible transaction size, not too small and not too large, keeps the cost modest for most workloads.

Kafka’s exactly-once features rest on a small set of ideas that work together. A producer ID and sequence numbers let brokers discard duplicates caused by retries. A transactional ID gives a logical producer a name that survives restarts. The producer epoch marks each new owner of that name and refuses every older one. The transaction coordinator records every decision durably, and control markers turn a group of writes across many partitions into a single all-or-nothing outcome that read-committed consumers can trust.

Once you understand who owns an identity, how ownership changes, and what each component checks, failures stop being mysterious. Duplicates, zombies, stuck readers, and fencing errors each map to a clear cause and a clear response.