A Kafka transaction is a group of writes, and optionally offset commits, that become visible together or not at all.

Start with the building blocks. A message is a small record of something that happened, such as “order 91 was paid”. A topic is a named log that stores messages in order. Each topic is split into partitions, which are separate ordered logs that let many machines share the work. A producer is a program that writes messages, and a consumer is a program that reads them. An offset is the position number of a message inside a partition, and consumers remember offsets to know where they stopped reading.

A Kafka transaction lets a producer say: “Begin. Write these messages to topic A, partition 2; these to topic B, partition 5; and record that I have finished reading up to offset 1200 of my input. Now commit.” Kafka then guarantees that all of those writes become visible to transaction-aware readers together. If the producer changes its mind or crashes, the whole bundle is thrown away as if it never happened.

Where transactions came from

Transactions were added in Kafka version 0.11, released in 2017, together with the idempotent producer. The goal was to allow stream processing applications to produce correct results even when machines fail and messages are retried. Since then, transactions have become the base for exactly-once processing in Kafka Streams and in newer Kafka Connect features.

Networks fail, machines crash and messages get retried. Delivery guarantees describe what the system promises when that happens.

Whenever two programs exchange messages, something can go wrong. A message may be lost on the wire. A confirmation may be lost, so the sender does not know if the message arrived. A program may crash between two steps. Systems describe their behavior in these situations using three standard levels of promise.

How duplicates and losses appear in Kafka

Before transactions, Kafka added a smaller feature that removes duplicates caused by producer retries.

An operation is idempotent if doing it many times gives the same result as doing it once. Pressing the “call elevator” button ten times does not summon ten elevators. An idempotent producer is a producer whose retries never create duplicate messages in a partition.

How it works

What it covers and what it does not

If you have used a database, Kafka transactions will feel familiar in spirit and different in the details.

Database transactions are often described by four properties, known as ACID: atomicity, consistency, isolation and durability. Kafka transactions provide some of these in a form that suits an append-only log.

Key differences to remember

A transaction involves a producer, a special broker role called the coordinator, an internal log, and the partitions being written to.

flowchart LR
  CL["Client Requests via API Gateway and Load Balancer"] --> APP["Application with Transactional Producer"]
  APP --> TC["Transaction Coordinator on a Broker"]
  TC --> TL["Transaction Log Internal Topic"]
  APP --> PA["Output Topic A Partition 2"]
  APP --> PB["Output Topic B Partition 5"]
  APP --> OFF["Consumer Offsets Internal Topic"]
  TC --> PA
  TC --> PB
  TC --> OFF
  PA --> RC["Consumers with Read Committed"]
  PB --> RC

The components in plain words

The transactional ID

The transactional ID is the name you give a producer so that Kafka can recognize it across restarts. It must be unique per logical producer instance and stable over time. If a program restarts with the same transactional ID, Kafka knows it is the same actor coming back, finishes or cancels whatever transaction the previous life left open, and blocks any older copy that may still be running. Choosing this ID well is the single most important design decision when using transactions.

Follow one transaction from the first call to the final commit and see what each broker and client does.

The states of a transaction

Inside the coordinator, each transaction moves through a small set of states. It starts as empty, becomes ongoing while messages are being written, and moves to a prepare state when the producer asks to end it. A prepare state means the decision has been made and safely recorded, even though the markers are not yet in place. Once the markers are written everywhere, the state becomes complete. A transaction that runs too long is aborted by the coordinator.

Kafka uses a classic idea, the two-phase commit, adapted for a distributed log.

The challenge is that a transaction may touch partitions on many different brokers. How do you make all of them agree, when any machine can fail at any moment? The answer is to separate the decision from the announcement.

flowchart TD
  A["Producer asks coordinator to commit"] --> B["Phase 1: Coordinator writes prepare commit to transaction log"]
  B --> C["Decision is now durable and cannot be reversed"]
  C --> D["Phase 2: Coordinator writes commit markers to every involved partition"]
  D --> E["Coordinator writes complete commit to transaction log"]
  E --> F["Read committed consumers can now see the messages"]

What a marker is

A control record, also called a marker, is a tiny special message written into a partition to say “the transaction with this producer ID has committed” or “has aborted”. Applications never see markers. Consumers use them, behind the scenes, to know which earlier messages to reveal or skip. That is why a topic in which a lot of transactions run holds slightly more entries than the number of messages your code wrote.

A transaction is only useful if readers respect it. The consumer’s isolation level decides whether they do.

Every Kafka consumer has a setting called the isolation level. It has two values.

The last stable offset

A read committed consumer never reads beyond a position called the last stable offset. This is the offset just before the earliest transaction that is still open. Messages after that point may belong to unfinished transactions, so the consumer waits. When the open transaction commits or aborts, the position moves forward and the waiting messages are either delivered or skipped.

flowchart LR
  M1["Offset 10 committed"] --> M2["Offset 11 committed"]
  M2 --> M3["Offset 12 in open transaction"]
  M3 --> M4["Offset 13 committed by another producer"]
  M4 --> M5["Offset 14 non transactional"]
  M2 -.-> LSO["Last stable offset stops before offset 12"]

Notice that message 13 was committed, and yet the consumer cannot read it yet. Ordering by offset is preserved, so one slow open transaction can hold back the visibility of later messages in the same partition. This is called head-of-line blocking, and it is the main reason to keep transactions short.

The most important use of transactions is a program that reads from one topic, transforms the data, and writes to another.

Suppose a service reads payment events, calculates a fee, and writes fee events to another topic. Two things must be saved after processing a batch: the output messages, and the fact that the input has been consumed. In an ordinary program these are two separate steps. If the program crashes between them, either the output is written twice or it is never written.

flowchart TD
  IN["Input Topic"] --> R["Consumer reads a batch"]
  R --> P["Application processes the batch"]
  P --> B["Begin transaction"]
  B --> W["Write results to output topics"]
  W --> O["Add consumer offsets to the transaction"]
  O --> C{"Commit succeeds"}
  C -->|"Yes"| OK["Results visible and input marked as consumed"]
  C -->|"No"| AB["Abort: results hidden and input will be read again"]

By including the input positions in the same transaction as the output messages, Kafka makes the two effects inseparable. Either both happen, and the program moves forward, or neither happens, and the program will reread the same input and produce the same output again. Because the first attempt was hidden by the abort, the retry does not create a duplicate. This is the heart of exactly-once processing in Kafka.

A slow or disconnected program can wake up after it has been replaced. Kafka uses epochs to shut it out.

A zombie is an old instance of a program that the system believes is dead, but that is actually alive. It might have been paused by a long garbage collection, cut off by a network problem, or simply slow. Meanwhile, the system started a replacement. If both write to the same output, the results are duplicated or mixed. This is one of the hardest problems in distributed systems.

Kafka solves it with an epoch, a number attached to the producer ID. Each time a producer registers with a given transactional ID, the coordinator increases the epoch. Every write carries the epoch, and brokers refuse writes from an older epoch than the latest one they know. The zombie, still holding the old epoch, receives a “fenced” error and must stop.

flowchart TD
  A["Instance 1 starts with transactional ID X and epoch 1"] --> B["Instance 1 becomes slow or disconnected"]
  B --> C["Instance 2 starts with same transactional ID X"]
  C --> D["Coordinator raises epoch to 2 and aborts unfinished work"]
  D --> E["Instance 1 wakes up and tries to write with epoch 1"]
  E --> F["Broker rejects write: producer fenced"]
  F --> G["Instance 1 must shut down"]

Why the transactional ID must be stable

Fencing works only when the old and the new instance use the same transactional ID. If each restart picked a random name, the coordinator would treat them as unrelated, and the zombie would never be blocked. For stream processing, each unit of work, such as an input partition, is usually mapped to a fixed ID, so that when work moves from one machine to another, the new owner always takes over the same identity.

A transaction that never finishes blocks readers. Timeouts are the safety valve.

Imagine a program begins a transaction, writes some messages, and then freezes forever. The messages are stored but never committed. Read committed consumers cannot pass the earliest open transaction, so the last stable offset stops moving. Every consumer of that partition is stuck.

Choosing the timeout

Too short, and slow but healthy transactions are aborted and retried, wasting work. Too long, and a crashed producer blocks readers for a long time before the coordinator steps in. A good rule is to set the timeout to a small multiple of the longest normal transaction, and to keep normal transactions short in the first place, from a fraction of a second to a few seconds for streaming workloads.

Most people meet Kafka transactions through Kafka Streams, where a single setting turns the whole machinery on.

Kafka Streams is a library for building stream processing applications. It reads from topics, transforms data, keeps state such as counts and totals, and writes results. Internally it combines every one of the pieces described so far. When you switch on exactly-once processing, Streams wraps each processing cycle in a transaction that contains three things: the output messages, the updates to the internal state changelog topics, and the input offsets.

What changes in behavior

Kafka can make its own writes atomic. It cannot reach into other systems and make them part of the same bundle.

Suppose a consumer reads an order, sends a confirmation email, and writes a result to Kafka inside a transaction. If the transaction aborts and is retried, the email is sent again. The email service knows nothing about Kafka transactions. The same is true for writes to a relational database, calls to a payment gateway, or updates to a search index.

Patterns for the outside world

flowchart LR
  SVC["Order Service"] --> DB["Database with Orders and Outbox Table"]
  DB --> CDC["Outbox Reader or Change Data Capture"]
  CDC --> KT["Kafka Topic"]
  KT --> PROC["Transactional Stream Processor"]
  PROC --> OUT["Output Topic"]
  OUT --> SINK["Sink with Upserts to Warehouse"]

Misunderstandings cause more incidents than bugs. Learn the limits before they teach you.

A handful of settings control nearly all transactional behavior. Names are shown so that you can find them in the documentation.

Transactions sit on top of the ordinary producer safety settings. If those are weak, the whole structure is weak.

A producer sends messages to a partition leader, which copies them to follower replicas. The producer chooses how much confirmation it wants before it considers a message safely stored. This choice is controlled by a setting called acks. With acks set to zero, the producer does not wait for any reply and may lose messages silently. With acks set to one, the leader confirms once it has written the message, but a crash of the leader right after can lose it before any follower copies it. With acks set to all, the leader confirms only after every in-sync replica has the message, which is the safe choice and the one transactions require.

Retries and ordering

When a send fails for a temporary reason, the producer retries. Without idempotence, a retry can reorder messages: batch one fails, batch two succeeds, and then batch one is retried and lands after batch two. With the idempotent producer, sequence numbers let the broker put batches in the right order and drop duplicates, even with several requests in flight at once. The recommended limit of in-flight requests with idempotence is five or fewer.

The minimum in-sync replicas rule

Acks all means “all replicas currently in sync”, which could shrink to just the leader if followers fall behind. The topic setting min.insync.replicas sets a floor: if fewer than the required number of replicas are in sync, the leader refuses the write instead of accepting it with weak protection. The common production pairing is three replicas with a minimum of two, so that one broker can fail without losing data or stopping writes.

Aborted messages stay in the log. The consumer library quietly removes them before your code sees anything.

When a read committed consumer fetches data, the broker sends the raw messages along with a short list of aborted transactions in the range that was fetched. Each entry names the producer ID that aborted and the offset where its transaction began. Brokers keep an index file for each log segment to answer this quickly, so they do not need to scan every message.

Because this filtering happens inside the client library, your code stays simple. The cost is a little extra memory and bookkeeping in the consumer, which grows when many transactions are open at once. A workload with a very high abort rate makes consumers do more filtering, so abort rates should stay low in healthy systems.

Markers use up offsets and aborted data lingers. Both surprise people who look closely at their topics.

Gaps in offsets

Every commit or abort marker takes one offset in each partition it is written to. If a partition holds ten messages and one transaction marker, the offsets run from zero to ten, with the marker at the end. A consumer never receives the marker, so it sees a gap in offsets where the marker sits. Tools that compute lag by subtracting offsets, and any code that assumes offsets increase by exactly one, must allow for these gaps. A partition full of very small transactions has many gaps.

Retention and compaction

Aborted messages are removed by normal cleanup. Under time or size based retention, whole old segments are deleted, including their aborted messages and markers. Under log compaction, where Kafka keeps the latest value for each key, the cleaner also removes aborted data during its work and eventually cleans up the markers themselves. Until that cleanup happens, aborted data takes disk space, which is another reason to keep abort rates low.

Most teams do not start with transactions. They add them to a pipeline that already runs. Do it in a careful order.

The transaction log is small compared with your data topics, but it sits on the critical path of every commit.

The log is an internal, compacted, replicated topic with a fixed number of partitions, fifty by default. Each transactional ID is mapped to one of these partitions by hashing, and the broker that leads that partition is the coordinator for the ID. The number of partitions is chosen when the cluster first creates the log, and it should not be changed later, because changing it would remap IDs to different coordinators.

The best way to trust a transaction system is to know exactly what it does in every bad situation.

Reading the table

Notice how each row has the same shape: something goes wrong, Kafka relies on a durable record such as the transaction log or the epoch, and the outcome is either a hidden abort followed by a clean retry, or a completed commit. No row leaves half of a transaction visible. That consistent shape is the entire purpose of the design.

Transactions are valuable, not free. Use this guide to decide pipeline by pipeline.

If most answers point toward transactions, adopt them for that pipeline and keep the rest simple. There is no reward for turning them on everywhere. A focused use on the few paths where exactness matters gives the benefits while keeping overhead and operational effort small.

Transactions buy correctness and pay for it with some speed and some complexity. Most teams should use them only where the trade is justified.

The main trade-offs

Transactions add work, but a well-tuned system can handle very high volumes.

The extra cost of a transaction comes from a few fixed steps: registering partitions with the coordinator, writing state changes to the transaction log, and writing markers to each partition. These steps are the same whether a transaction holds ten messages or ten thousand. That is why the size of the transaction matters so much for efficiency.

A transaction system must survive broker failures, network splits and application crashes without changing its answer.

flowchart TD
  F["Coordinator broker fails during commit"] --> L["New coordinator elected from replicas of the transaction log"]
  L --> R["New coordinator reads log and finds prepare commit"]
  R --> M["It finishes writing commit markers"]
  M --> D["Transaction completes with the original decision"]

Handling errors correctly

Errors fall into three groups. Retriable errors, such as a temporary network problem, are handled by the client library. Abortable errors mean the current transaction cannot complete, so the application aborts it and starts over with the same input. Fatal errors, such as being fenced by a newer instance, mean the producer must close and stop writing. Confusing these groups, for example retrying after a fatal error, can create the very duplicates transactions try to prevent.

Transactions add new resources that need protection. Treat the transactional ID like a credential.

A few signals reveal almost every transactional problem before customers notice.

Log the transactional ID, the epoch and the outcome of each transaction at a low level of detail, so that when something looks wrong, you can trace a specific run. Add an alert when a transaction stays open for longer than a few times your normal duration, since this is the earliest warning of trouble.

Walk through a realistic service, first without transactions and then with them, and see exactly what changes.

A bank publishes transfer requests to a topic. A service reads each request and writes two events: a debit event to a topic named “debits” and a credit event to a topic named “credits”. Each request should produce exactly one debit and one credit.

Without transactions

The service reads request 55, writes the debit event, and then its machine loses power before it writes the credit. On restart, the service has not saved its position, so it reads request 55 again. It writes the debit again, and then the credit. The account holder is debited twice and credited once. Alternatively, if the service saved its position before writing, a crash would lose the transfer entirely.

With transactions

Now consider the same power failure. If it happens before the commit, the coordinator aborts the transaction after a timeout or as soon as the service restarts with the same ID. The debit and credit are hidden, the input position was never advanced, and the service reads request 55 again and repeats the work. Readers of the debit and credit topics never saw the aborted events. If the failure happens after the commit decision is recorded, the coordinator finishes writing the markers. Either way, the account holder sees exactly one debit and one credit.

Failures are rare in production, so you must create them on purpose in testing.

Kafka Streams also ships with a test driver that runs a topology in memory, which is excellent for checking logic. However, the driver does not reproduce real broker failures, so it complements, but cannot replace, tests against a real cluster with deliberate crashes.

A few proven shapes solve most needs. A few tempting shortcuts cause most of the outages.

Patterns that work

Anti-patterns to avoid

A short checklist gathered from production systems. Follow it and you avoid most transaction pain.

Mistakes seen again and again

Kafka transactions are one tool among several. Knowing the alternatives helps you pick the simplest one that works.

Transactions appear wherever a wrong or duplicated result costs money or trust.

Kafka began at LinkedIn to move activity data, and transactions were added to support stream processing that businesses could trust with critical numbers. Public talks and articles from many companies describe patterns like the ones below. The internal designs of specific companies change over time, so treat these as illustrations of the technique rather than exact blueprints.

A quick reference for the vocabulary used throughout this guide.

Quick answers to the questions that come up most when teams first use Kafka transactions.

Kafka transactions make a bundle of writes succeed or fail as one, so that pipelines can stay correct through failures.