Before we meet any protocol, we must feel the problem it solves.

The world of many partitions

A Kafka topic is divided into partitions, and each partition is stored on a different broker as its leader. A single application may write to many partitions during a single piece of work. For example, an order-processing service might read one order, then write a payment event to one topic, an inventory event to another and a notification event to a third. Each of these writes goes to a different partition, often on a different machine.

What can go wrong without transactions

The wish: exactly-once processing

What applications want is the effect of exactly-once processing: every input is reflected in the outputs precisely one time, even when machines crash and networks fail. Kafka’s transactions, combined with a few supporting features, are the foundation for delivering this within Kafka itself.

The two-phase commit is one of the oldest ideas in distributed computing. Understanding the classic version makes Kafka’s version easy to see.

The cast of characters

In the classic protocol there is one coordinator and several participants. The coordinator is the one that wants a bundle of work done atomically. The participants are the independent parties that each perform a portion of that work, such as different databases.

Phase one: prepare and vote

The coordinator asks every participant: “Can you promise to commit your part, if I tell you to?” Each participant does everything it can short of the final commit, writes its promise to durable storage, and answers yes or no. A “yes” is a binding promise: from that moment the participant may not change its mind unless the coordinator says so.

Phase two: commit or abort

If every participant said yes, the coordinator decides to commit and tells all participants to complete. If anyone said no, or failed to answer, the coordinator decides to abort and tells all participants to undo their work. The decision is final and is recorded durably before any participant is told.

sequenceDiagram
    participant C as Coordinator
    participant P1 as Participant 1
    participant P2 as Participant 2
    C->>P1: Prepare - can you commit
    C->>P2: Prepare - can you commit
    P1-->>C: Yes, promise recorded
    P2-->>C: Yes, promise recorded
    C->>C: Record decision - commit
    C->>P1: Commit
    C->>P2: Commit
    P1-->>C: Done
    P2-->>C: Done
        

The classic protocol has a famous problem, and Kafka’s design sidesteps most of it.

The blocking problem

Suppose a participant has said yes and is now waiting for the coordinator’s decision. If the coordinator crashes at that moment, the participant is stuck. It cannot commit, because the decision might have been abort. It cannot abort, because the decision might have been commit. It must wait, often holding locks that block other work, until the coordinator returns. This state is called being in doubt, and it is why classic two-phase commit is described as a blocking protocol.

How Kafka changes the picture

A different flavor of the same idea

It is fair to say that Kafka implements a two-phase commit variant rather than the textbook version. The important features remain: a durable decision point, a second phase that spreads the decision to all involved partitions, and completion that is guaranteed to happen eventually once the decision is recorded. The absence of a voting phase and the presence of a replicated coordinator make it far more practical for a streaming platform.

Five kinds of participants make Kafka transactions work. Meeting them first makes the later story easy to follow.

flowchart LR
    P["Transactional producer"] -->|"1 register and coordinate"| TC["Transaction coordinator"]
    TC -->|"2 record state"| TL["Transaction log topic"]
    P -->|"3 write records"| L1["Partition A leader"]
    P -->|"3 write records"| L2["Partition B leader"]
    TC -->|"4 write markers"| L1
    TC -->|"4 write markers"| L2
    L1 --> C["Read-committed consumer"]
    L2 --> C
        

The coordinator is a role, not a separate server

There is no separate transaction server to install. Every broker can act as a transaction coordinator for some share of the transactional identifiers. Which broker is responsible for a particular transactional identifier is decided by a simple rule, described in a later chapter, so both the producer and the cluster always agree on who the coordinator is.

Transactions are built on top of a simpler feature that removes duplicates caused by retries.

The retry problem

A producer sends a batch of records to a broker and waits for an acknowledgement. If the acknowledgement never arrives, the producer does not know whether the broker stored the batch. To be safe it retries. But if the first attempt had actually succeeded and only the reply was lost, the retry stores the batch a second time, creating duplicates.

The idempotent solution

An idempotent producer attaches two numbers to every batch: a producer identifier assigned by the cluster, and a sequence number that increases by one for each batch sent to a given partition. The broker remembers the latest sequence numbers it has accepted from each producer for each partition. If a batch arrives with a sequence number it has already stored, the broker recognizes the duplicate and quietly discards it while still acknowledging success.

What idempotence does and does not cover

A stable name for the producer, plus an ever-growing epoch, protects the system from stale instances.

The transactional identifier

A transactional producer is configured with a name called the transactional identifier. This is chosen by the application, not by Kafka, and it must stay the same across restarts of the same logical worker. The cluster uses it to remember which transaction belongs to which worker, and to link the worker’s identity across different sessions.

What happens when a producer starts

What is a zombie?

A zombie is an old application instance that the rest of the system believes is dead, but which is actually still alive. It may have been paused by a long garbage collection, frozen by a virtual machine migration, or disconnected by a network partition. A replacement instance has already started. If the zombie wakes up and resumes writing, two instances do the same job at once, and results are duplicated or interleaved.

sequenceDiagram
    participant Z as Old instance - epoch 4
    participant N as New instance
    participant TC as Coordinator
    participant PL as Partition leader
    Z->>Z: Long pause
    N->>TC: Initialize with same transactional id
    TC->>TC: Raise epoch to 5, abort old transaction
    TC-->>N: Producer id, epoch 5
    Z->>Z: Wakes up
    Z->>PL: Write with epoch 4
    PL-->>Z: Rejected - fenced by newer epoch
    N->>PL: Write with epoch 5
    PL-->>N: Accepted
        

The fence in action

Every write by a transactional producer carries its epoch. Brokers and the coordinator compare it with the latest epoch known for that identifier. A request with an older epoch is refused with an error that tells the producer it has been fenced. The producer is expected to treat this as a fatal error: it must stop, discard its work, and let the new instance take over. This guarantees that at any moment only one live instance can make progress for a given transactional identifier.

The coordinator’s memory lives in an internal topic. Its layout explains both the durability and the failover behavior.

An internal topic with many partitions

The transaction log is an internal Kafka topic with a fixed number of partitions, fifty by default. It is replicated across brokers just like any topic, and is typically configured with a higher replication factor and a strict minimum number of in-sync replicas to protect it against data loss. Because it is a normal replicated topic, its content survives the failure of individual brokers.

Choosing the coordinator by hashing

To decide which broker coordinates a given transactional identifier, Kafka hashes the identifier and maps it to one of the partitions of the transaction log. The broker that currently leads that partition is the coordinator for the identifier. There is nothing to configure and no separate election. If the partition’s leadership moves to another broker, the coordinator role moves with it.

flowchart TB
    T1["Transactional id: orders-worker-1"] -->|"hash modulo partitions"| P7["Transaction log partition 7"]
    T2["Transactional id: billing-worker-3"] -->|"hash modulo partitions"| P31["Transaction log partition 31"]
    P7 --> B2["Leader on broker 2 - coordinator"]
    P31 --> B5["Leader on broker 5 - coordinator"]
        

What is stored in the log

Compaction and recovery

The transaction log is compacted, meaning only the latest entry for each transactional identifier is eventually kept. This keeps it small. When a broker becomes the leader of a transaction log partition, its coordinator reads the partition from start to end to rebuild an in-memory table of every transaction it is responsible for. Any transaction found in the middle of committing or aborting is then driven forward to completion. This loading step is how Kafka avoids the classic blocking problem.

Each transaction moves through a small set of states. Knowing them turns error messages and tool outputs into plain sentences.

stateDiagram-v2
    [*] --> Empty
    Empty --> Ongoing: first partition added
    Ongoing --> PrepareCommit: producer requests commit
    Ongoing --> PrepareAbort: producer requests abort or timeout
    PrepareCommit --> CompleteCommit: all commit markers written
    PrepareAbort --> CompleteAbort: all abort markers written
    CompleteCommit --> Empty: ready for next transaction
    CompleteAbort --> Empty: ready for next transaction
    Empty --> Dead: identifier expires
        

Reading the states as a story

The two prepare states are the heart of the two-phase idea. Entering a prepare state means the decision has been made and written down. Everything that follows is mechanical: send markers, wait for confirmations, write the completion state. If anything crashes, whoever takes over reads the prepare state from the log and simply carries on.

Now we combine every idea and walk through one successful transaction, step by step.

Suppose a transactional producer must write one record to topic orders partition zero and another record to topic payments partition three, atomically, as one bundle.

sequenceDiagram
    participant P as Producer
    participant TC as Coordinator
    participant TL as Transaction log
    participant A as Orders partition 0 leader
    participant B as Payments partition 3 leader
    P->>TC: Add partitions orders-0 and payments-3
    TC->>TL: Record state ongoing with partitions
    P->>A: Write order record
    P->>B: Write payment record
    P->>TC: Commit
    TC->>TL: Write prepare commit
    TL-->>TC: Replicated
    TC-->>P: Commit accepted
    TC->>A: Write commit marker
    TC->>B: Write commit marker
    A-->>TC: Marker written
    B-->>TC: Marker written
    TC->>TL: Write complete commit
        

Everything important happens in one small durable write.

What the coordinator does

When the producer asks to commit, the coordinator first checks that the request comes from the current epoch, that the transaction is in the ongoing state, and that its recorded list of partitions is complete. It then writes a single entry to the transaction log saying the transaction is now in the prepare-commit state, together with the full list of partitions involved. It waits until this entry has been copied to the required in-sync replicas of the transaction log.

Why this write is the decision

After this entry is safely stored, no failure can change the outcome. If the coordinator broker crashes, another broker will become leader of that transaction log partition, read the entry, and know for certain that the decision was commit. There is no scenario in which the transaction can later become an abort. Conversely, if the entry was not stored, then the transaction was never decided, and a recovered coordinator will treat it as still ongoing, and it will eventually time out and be aborted.

The difference from classic voting

In the classic protocol, this is where participants would each vote. In Kafka, no such vote is needed. The partitions have already stored the transaction’s records, and there is nothing further they must check. The prepare step is therefore quick and involves only the coordinator and its replicated log.

With the decision safely recorded, the coordinator spreads the news to every partition involved.

The marker request

The coordinator sends a special request to the leader of each partition listed in the transaction. The request says, in effect, “write a marker for this producer identifier and epoch saying that its transaction committed” or “aborted”. Several partitions may share one broker, so the coordinator batches these requests per broker for efficiency.

What the partition leader does

On receiving the request, the leader appends a special control record to the partition log. This marker is a small batch flagged as a control batch. It is replicated to the followers of the partition like any other record. When the marker has been replicated, the leader acknowledges to the coordinator. The marker is a permanent part of the partition log and acts as the official “commit” or “abort” stamp for all earlier records from that producer’s transaction.

Retrying until done

If a partition leader is temporarily unavailable, or the request times out, the coordinator simply retries, sending the request again to whichever broker now leads the partition. Because the decision is already recorded, retries continue until every partition acknowledges. Marker writes are safe to repeat: a repeated marker is recognized and does no additional harm.

The partition log itself carries the evidence of every transaction outcome.

Data records and control records

A partition that takes part in transactions contains two kinds of batches. Data batches hold the actual records written by producers. Control batches hold small markers written by the coordinator. Control batches carry a flag in their header, so readers can tell them apart from data instantly. Applications never see control records: the client library reads them, uses them to decide what to show, and then hides them.

Offsets and gaps

Because each marker occupies a position in the log, offsets seen by consumers sometimes appear to skip a number. This is normal: the skipped offset belongs to a marker the consumer does not receive. Applications must never assume offsets are perfectly consecutive. Aborted records also keep their offsets in the log, although they are filtered out for read-committed consumers.

flowchart LR
    R1["Offset 10 - record from txn A"] --> R2["Offset 11 - record from txn B"]
    R2 --> R3["Offset 12 - record from txn A"]
    R3 --> M1["Offset 13 - commit marker for A"]
    M1 --> R4["Offset 14 - ordinary record"]
    R4 --> M2["Offset 15 - abort marker for B"]
        

Consumers choose what they want to see, and brokers help them see the right thing efficiently.

Two isolation levels

The last stable offset

For each partition, the broker tracks a position called the last stable offset. It is the offset of the earliest record that still belongs to a transaction that has not finished, or the end of the log if there is no such transaction. A read-committed consumer is never given anything at or beyond this position. Everything before it is settled: either committed or aborted or not transactional.

flowchart LR
    A["Offsets 0 to 20 - all settled"] --> B["Offset 21 - first record of open transaction"]
    B --> C["Offsets 22 to 30 - later records"]
    A -. "read committed consumers can read up to here" .-> LSO["Last stable offset = 21"]
    C --> HW["High watermark = 31"]
        

Aborted transaction index

To let consumers skip aborted data quickly, brokers keep an index of aborted transactions for each partition segment. When a consumer fetches a range of records, the broker also returns a list of the aborted transactions that overlap that range. The client library uses this list, together with markers, to drop aborted records before handing data to the application. The consumer does not need to read the whole log to know what to skip.

The most common use of transactions is to read input, compute, and write output, all as one atomic unit.

The problem of the input position

A stream processing application reads messages from an input topic, transforms them and writes results to an output topic. To avoid reprocessing on restart, it records how far it has read by committing the consumer offset. If the output write and the offset commit are separate steps, a crash between them leaves either a duplicate output or a lost input.

Making the offset commit part of the transaction

Kafka lets a transactional producer include the consumer’s offsets inside the transaction. The offsets are written to the internal offsets topic as part of the same transaction as the output records. When the transaction commits, both the outputs and the new input position become visible together. When it aborts, neither does, and the application restarts from the previous position and redoes the work cleanly.

flowchart LR
    IN["Input topic"] --> CO["Consumer"]
    CO --> APP["Application logic"]
    APP --> PR["Transactional producer"]
    PR -->|"outputs"| OUT["Output topics"]
    PR -->|"consumer offsets"| OFF["Offsets topic"]
    PR -->|"commit or abort"| TC["Coordinator"]
        

Half of a good protocol is what it does when things do not go to plan.

Ways an abort begins

The mirror image of commit

Aborting follows exactly the same two-phase pattern as committing. The coordinator writes prepare-abort to the transaction log, which is the decision point. It then sends abort markers to all involved partitions, and finally writes complete-abort. The records that were already written to the partitions stay in the log physically, but the abort markers and the aborted transaction index make read-committed consumers ignore them.

sequenceDiagram
    participant P as Producer
    participant TC as Coordinator
    participant TL as Transaction log
    participant L as Partition leaders
    P->>TC: Abort
    TC->>TL: Write prepare abort
    TL-->>TC: Replicated
    TC-->>P: Abort accepted
    TC->>L: Write abort markers
    L-->>TC: Markers written
    TC->>TL: Write complete abort
        

A protocol proves itself on the worst day. Here is what happens when each part fails.

The recovery routine is the reason this protocol is safer than its textbook ancestor.

When a broker becomes a coordinator

When a broker becomes leader for a partition of the transaction log, it runs a loading routine. It reads the partition from the beginning, applying each entry in order to build an in-memory table with the latest state of every transactional identifier that maps to that partition. During this time it refuses to serve requests for those identifiers and tells clients to retry shortly.

flowchart TB
    A["Coordinator broker fails"] --> B["Another broker becomes leader of the log partition"]
    B --> C["Load and replay the partition"]
    C --> D{"Any transaction in a prepare state?"}
    D -->|"yes"| E["Send markers and write completion"]
    D -->|"no"| F["Keep ongoing ones under timeout watch"]
    E --> G["Serve requests"]
    F --> G
        

Time limits are the safety valve that keeps a broken transaction from holding up the rest of the system.

The transaction timeout

Every transactional producer declares how long it is willing to keep a transaction open. The coordinator watches the clock. If a transaction remains open beyond its timeout without a commit or abort request, the coordinator aborts it. The broker also enforces an upper limit on what a producer may ask for, so no client can request an unreasonably long timeout.

Why timeouts are essential

Recall that the last stable offset stops at the earliest open transaction, and read-committed consumers cannot read past it. A crashed producer that left a transaction open would otherwise freeze every read-committed consumer of that partition forever. The timeout guarantees that the freeze lasts at most a bounded time.

Hanging transactions

Occasionally, because of bugs or unusual failure sequences, a transaction can remain open on a partition even though the coordinator no longer considers it active. Such a transaction is called hanging. It holds the last stable offset back and causes consumer lag to grow while data keeps arriving. Kafka provides an administrative tool to find such transactions and to abort them safely.

Exactly-once is a powerful phrase and a frequently misunderstood one. Precision here saves real money and reputation.

The boundary of the guarantee

Kafka’s exactly-once guarantee means that, within a pipeline that reads from Kafka topics and writes to Kafka topics, each input record affects the output exactly once, even when processes and brokers fail and recover. It is achieved by combining idempotent writes, atomic multi-partition transactions with offsets, zombie fencing and read-committed consumers.

The trap of external side effects

Suppose an application inside a transaction calls an external payment service. If the transaction later aborts and the input is reprocessed, the payment service is called a second time. Kafka cannot undo the first call. The application must therefore make external calls safe to repeat, for example by sending a unique request identifier that the external service uses to ignore duplicates.

Most people meet Kafka transactions through higher-level tools that do the plumbing for them.

Stream processing libraries

Kafka’s own stream processing library offers an exactly-once processing mode that wraps every processing step in the consume, transform and produce transaction pattern described earlier. It commits regularly on a short interval, keeps its internal state changelogs inside the same transactions, and manages transactional identifiers and fencing automatically. Users select the mode through a single setting and do not touch producers or coordinators directly.

Connector frameworks

Connector frameworks can provide exactly-once delivery for certain source and sink connectors, where the connector and the external system support the necessary coordination. For sources, the framework can commit source records and their source offsets together in one transaction. For sinks, exactly-once outcomes usually rely on the external system accepting idempotent or transactional writes, since Kafka’s transaction cannot extend into it.

The transaction protocol has been refined over the years to close subtle gaps found in real deployments.

The gap that motivated the changes

In the original design, a delayed write from a producer could arrive at a partition leader after the transaction it belonged to had already been completed. The leader, unaware, might store it as though it were part of a new, still-open transaction or leave it dangling. This class of problem is sometimes called a hanging transaction, and it is very hard to detect, since every individual step looked correct at the time it happened.

Two kinds of improvement

Real applications must often update a database and Kafka together. This is where transactions meet their limits, and where new proposals and older patterns help.

The dual-write problem

An application that saves an order to a database and then publishes an order event to Kafka is doing two writes to two systems. If it crashes between them, either the database has the order but Kafka does not, or the reverse. Kafka’s own transactions cannot cover the database, and the database’s transactions cannot cover Kafka.

Kafka as a participant in an outside two-phase commit

The classic way to combine systems is to let an external transaction manager act as the coordinator and treat each system as a participant. For that to work, Kafka would need to support a prepared state that survives producer failure and that can be resolved later by the external coordinator. A Kafka improvement proposal was written to add exactly this capability, allowing a producer to keep a prepared transaction alive across restarts and to be told later whether to commit or abort. The work has been progressing across recent releases, so anyone considering it should read the current release notes to see which parts are available in their version.

Transactions are not free, but their cost is often smaller than people fear when used well.

Where the extra work comes from

The fixed cost is per transaction, not per record

The extra requests and writes depend mostly on the number of transactions and the number of partitions involved, not on how many records are inside. This is the key to performance planning. A transaction containing thousands of records pays roughly the same overhead as one containing ten. Small transactions therefore waste a lot of effort, while very large ones increase latency and memory use and lengthen the time the last stable offset is held back.

You do not need every setting memorized, but you must understand what each family controls.

On the producer

On the consumer

On the broker

The transaction machinery introduces its own resources that must be protected.

Transactional identifier as a protected resource

When access control is enabled, the transactional identifier is treated as a resource in its own right. A producer must be allowed to write using that specific identifier, in addition to being allowed to write to the topics involved. This prevents one application from hijacking another’s identifier and thereby fencing it out or aborting its transactions.

Protecting the wider picture

Healthy transactions are quiet. Good monitoring lets you notice trouble before consumers do.

Logs worth reading

Broker logs record coordinator loading after a failover, timed-out transactions being aborted, and marker delivery problems. Application logs record fenced errors, abort decisions and retries. When investigating, line up the timestamps of the producer’s log, the coordinator’s log and the affected partition leader’s log to reconstruct the sequence of a transaction.

Kafka includes tools that let you see transactions rather than guess about them.

Practical uses

Every guarantee has a price. Seeing both sides clearly helps you decide when transactions are worth it.

When to use transactions

These habits keep transactional pipelines calm and predictable.

A checklist before going live

Each of these looks harmless until a failure makes it expensive.

A few persistent misunderstandings cause a surprising number of production problems.

Transactions appear wherever duplicates or partial results are costly.

Transactions are one point on a spectrum. Seeing the whole spectrum shows when each option fits.

Choosing the right level

Let us apply everything to a small, realistic pipeline and watch the guarantees at work.

A shop publishes an order event to a topic named orders. A processing service reads each order, then writes three events: a payment request to one topic, a stock reservation to another and an audit entry to a third. The service runs as three instances, each responsible for a share of the input partitions. The team wants no order to be processed twice and no order to be half processed.

What happens when an instance crashes halfway

Suppose an instance dies after writing the payment request and reservation for one order, but before committing. The downstream services see nothing, because the transaction is still open. After the timeout, the coordinator aborts it and the three records stay hidden forever. A replacement instance starts with the same transactional identifier, is given a higher epoch, and re-reads the batch from the last committed input position. It processes the same orders again in a fresh transaction, and this time the results are committed. Downstream services see exactly one payment request per order.

sequenceDiagram
    participant I1 as Instance one
    participant TC as Coordinator
    participant D as Downstream services
    participant I2 as Replacement instance
    I1->>TC: Begin and write partial outputs
    I1->>I1: Crash
    D->>D: Sees nothing, transaction still open
    TC->>TC: Timeout reached, abort
    I2->>TC: Initialize with same identifier, epoch raised
    I2->>I2: Re-read from last committed offsets
    I2->>TC: Write outputs and commit
    D->>D: Sees each result exactly once
        

Guarantees that are never tested under failure are only hopes. A modest testing routine turns hope into evidence.

Not exactly. It shares the durable decision point and the second phase that spreads the decision, but it has no voting step, the coordinator is replicated, and data is written to partitions before the decision.

The moment the coordinator durably records the prepare-commit or prepare-abort state in the transaction log. After that, the outcome cannot change.

No. Its commit call returns once the decision is durably recorded. The coordinator writes markers to partitions afterwards.

A new coordinator loads the transaction log, finds the transaction in a prepare state and finishes the second phase by sending the markers.

It is an older instance that was thought dead but is still running. Each initialization for a transactional identifier raises the epoch, and requests with an older epoch are rejected, which stops the zombie.

They can only read up to the last stable offset, which stops at the earliest open transaction. A long or hanging transaction therefore holds them back until it finishes or times out.

Commit and abort markers take up offsets in the log but are not delivered to applications, so consumers see gaps where the markers sit.

No. They remain in the log until normal retention removes them, and are simply hidden from read-committed consumers.

Only if you need atomicity across several partitions or need to commit consumer offsets together with outputs. Idempotence alone handles retry duplicates within one partition.

Not by themselves. Patterns like the outbox, change data capture, or an external transaction manager are used to coordinate Kafka and a database, and newer proposals aim to let Kafka join an outside two-phase commit.

The identifier is hashed to a partition of the transaction log, and the broker leading that partition acts as coordinator. If leadership moves, the coordinator role moves with it.

Longer than your longest normal transaction, but short enough that a failure does not freeze read-committed consumers for long. Measure real transaction durations and choose accordingly.

Because offsets must be committed through the transaction, together with outputs. Automatic commits would record progress independently and break the atomic link.

A transaction that remains open on a partition even though the coordinator no longer treats it as active. It holds back the last stable offset and requires administrative action or protocol improvements to clear.

The overhead is mostly fixed per transaction, so moderately sized transactions cost little relative to their content. The main visible effect is added latency for read-committed consumers, roughly the duration of the transaction.

It is unsafe on a live cluster because it changes the mapping from identifiers to coordinators. Choose the number carefully when creating the cluster.

Producers begin and end transactions, but consumers must choose the read-committed level to honor them, and operators must monitor and protect the coordinator layer and the transaction log.

Kafka’s transaction protocol takes the classic two-phase commit idea and adapts it to a streaming platform. A stable transactional identifier gives each worker a durable identity, epochs fence out stale instances, and a replicated coordinator records every decision in a transaction log. Records are written to partitions during the transaction, the decision to commit or abort is made durable in phase one, and markers are written to every partition in phase two so that read-committed consumers see all of the results or none of them. Combined with transactional offset commits, this yields exactly-once processing inside Kafka, and understanding its boundaries lets you design safe pipelines around the outside world.