Before any Kafka words, let us understand the simple problem that transactions solve.

Suppose a small program reads one message that says “customer Asha bought a phone.” The program must do three things: write a message to an “orders-confirmed” topic, write a message to a “stock-reduced” topic, and remember that it has finished reading that input message. If the program crashes after the first write but before the other two, the world is left in a strange state. Asha’s order is confirmed but the stock was never reduced, and after a restart, the program will read the same input message again and confirm it a second time.

What a transaction promises

A Kafka transaction promises atomicity. That word means “all or nothing.” Either all the messages written inside the transaction become visible to readers, or none of them do. This holds even when the writes go to many different topics and partitions living on many different brokers, and even when the program or a broker crashes in the middle.

Who keeps the promise?

A single producer program cannot keep this promise alone, because it might crash at the worst moment. Something else must remember what the producer started, decide the final outcome, and finish the job even if the producer is gone. That “something else” is the transaction coordinator. It is not a separate machine. It is a role played by a normal Kafka broker.

A short tour of the Kafka words we will use again and again.

Producers are programs that write messages. Consumers are programs that read them. A message is only added to the end of a partition, and it is never changed afterwards. Consumers remember how far they have read using a number called an offset, which Kafka stores in a special internal topic. This is important later, because committing that number is part of what a transaction can do.

Internal topics

Kafka keeps some of its own working notes in topics too. They usually have names that start with two underscores. One holds consumer bookmarks. Another holds the transaction coordinator’s notes, which we will study closely in this guide. Using ordinary Kafka topics for its own memory is a favorite Kafka habit: it gets copying, durability, and recovery for free.

Three promises that a messaging system can make, and why the third is the hardest.

Why duplicates happen

A producer sends a message and waits for the broker’s reply. If the reply is lost in the network, the producer cannot tell whether the message arrived. The safe thing is to send it again. If the first message had in fact arrived, the partition now contains two copies. This is the classic origin of duplicates, and it happens even in a perfectly healthy system when the network hiccups.

Where “exactly once” is real

Kafka’s exactly-once feature is real inside Kafka: reading from Kafka topics, processing, and writing to Kafka topics. It cannot magically make an outside system, such as an email service or a database, act exactly once. For outside systems you need extra techniques, such as making the outside action harmless to repeat. We come back to this important limit later.

Transactions stand on the shoulders of a simpler idea that removes duplicates from a single partition.

Idempotent is a mathematical word that means “doing it twice has the same effect as doing it once.” Pressing an elevator button twice does not call two elevators. An idempotent producer works the same way: even if it sends the same message twice, the partition keeps only one copy.

Where idempotence stops

Idempotence protects a producer only for one partition and only as long as the producer keeps the same identity. If the producer crashes and restarts, it usually gets a brand new producer ID, so the broker cannot recognize it as the same producer. And idempotence does nothing for a group of writes that must succeed together across many partitions. To fix these gaps, Kafka added the transaction coordinator, which gives a producer a stable identity that survives restarts and makes group writes atomic.

A precise but friendly definition, with a picture of the most common use.

A Kafka transaction is a bundle of writes, possibly to several topics and partitions, that becomes visible to readers all at once or not at all. The producer opens the bundle, adds writes, and then either commits (makes everything visible) or aborts (makes everything invisible). Additionally, the bundle can include consumer bookmark updates, so that “I have finished reading up to here” is saved atomically with the output messages.

The classic pattern: consume, transform, produce

If the program crashes anywhere before the commit, the outputs stay invisible and the bookmark stays where it was. After restart, the program reads the same input again and redoes the work, but because the earlier attempt was never committed, no visible duplicates exist. This is the heart of Kafka’s exactly-once processing.

The referee of every Kafka transaction.

The transaction coordinator is a component that runs inside every Kafka broker. For any given transaction, exactly one broker plays the coordinator. Its job is to remember which transactions are open, which partitions they touch, and what the final decision is. It makes sure that the decision is written down safely, and then it makes sure the decision is carried out on every partition involved.

Which broker is the coordinator?

Any broker could be a coordinator, but for a given transactional ID only one broker is. That ID is a name chosen by the application, such as “payment-processor-7”. Kafka turns the name into a number, and the number decides which partition of the internal transaction log is responsible for it. The broker that currently leads that partition of the log is that producer’s coordinator. This is a neat trick: leadership of a log partition doubles as the coordinator election.

A coordinator that forgets is useless, so its memory lives in a replicated Kafka topic.

The coordinator writes every important change of every transaction into an internal topic called the transaction state log. Its name is __transaction_state. By default it has fifty partitions and each partition is stored on three brokers, with at least two copies required to be in sync before a write is confirmed. The topic is compacted, which means Kafka eventually keeps only the newest record for each transactional ID, like a notebook where old entries for the same person are crossed out.

Why the number of partitions is fixed

Because the mapping from name to partition depends on the partition count, the count must not change after the cluster begins using transactions. If it changed, names would suddenly point to different cabinets and the coordinators would lose track of open transactions. This is why the count is set once, at the beginning, and treated as permanent.

The whole picture: who talks to whom, and why.

flowchart TB
  P["Transactional Producer"]
  subgraph BR["Kafka Broker Fleet"]
    TC["Broker B - Transaction Coordinator"]
    TL[("Transaction log partition - led by Broker B")]
    L1["Broker A - leader of orders partition 0"]
    L2["Broker C - leader of stock partition 3"]
    GC["Broker D - group coordinator for offsets"]
  end
  CON["Read-committed Consumer"]
  P -->|"1 find coordinator and get identity"| TC
  P -->|"2 register partitions"| TC
  TC -->|"saves state"| TL
  P -->|"3 send messages"| L1
  P -->|"3 send messages"| L2
  P -->|"4 send consumer offsets"| GC
  P -->|"5 commit or abort"| TC
  TC -->|"6 write commit or abort markers"| L1
  TC -->|"6 write commit or abort markers"| L2
  TC -->|"6 write markers"| GC
  L1 -->|"only committed data"| CON
  L2 -->|"only committed data"| CON
    

Read the figure from the left. The producer first talks to the coordinator to get an identity and to register the partitions it plans to write to. It then sends its messages directly to the partition leaders, without going through the coordinator, so the coordinator is not a bottleneck for data. When the producer is ready, it tells the coordinator to commit or abort. The coordinator writes the decision to its log and then sends small markers to every partition involved. Consumers that ask for committed data only see messages after they are marked committed.

The components in plain words

The duplicate-catching machinery that transactions rely on every second.

Each partition leader remembers, for each producer ID, the last few sequence numbers it accepted. When a new batch of messages arrives, the leader compares its first sequence number with what it expects next. There are only three possible situations, and each has a clear response.

Why several batches can be in flight

To keep the network busy, a producer may send several batches without waiting for each reply. Kafka allows a small number of such in-flight batches per connection while still preserving order, because the sequence numbers let the leader detect any reordering after a failed send. This is why the producer’s order-preserving retries are safe when idempotence is on.

How it connects to transactions

Transactions use the same producer ID, epoch, and sequence numbers that idempotence uses. What transactions add is the coordinator, which makes the producer ID survive restarts through the stable transactional ID, and which decides the final fate of all the numbered messages in a transaction. Idempotence is the foundation, and the coordinator is the roof.

These three labels are easy to mix up. Here is a simple way to keep them apart.

Why a stable name is required

When a producer crashes, a new instance starts and uses the same transactional ID. The coordinator sees a familiar name, gives it the same producer ID, and raises the epoch. In doing so, it also cleans up: if the earlier instance left a transaction half-finished, the coordinator aborts it before letting the new instance continue. This is how a restart avoids leaving stale work behind.

Every transaction moves through a small set of named states, and the coordinator records each move.

stateDiagram-v2
  [*] --> Empty
  Empty --> Ongoing : first partition added
  Ongoing --> Ongoing : more partitions or offsets added
  Ongoing --> PrepareCommit : producer asks to commit
  Ongoing --> PrepareAbort : producer asks to abort or timeout expires
  PrepareCommit --> CompleteCommit : markers written to all partitions
  PrepareAbort --> CompleteAbort : markers written to all partitions
  CompleteCommit --> Empty : ready for next transaction
  CompleteAbort --> Empty : ready for next transaction
  Empty --> Dead : transactional ID expires from idleness
    

A closer look at the small record that the coordinator keeps for every transactional ID.

For each transactional ID, the coordinator keeps one small record, both in memory and in the transaction log. Think of it as an index card. It is not big, but it holds everything needed to recover the transaction from any point.

Every change is a new card

The coordinator never edits a card in place. Each change, such as adding a partition or moving to a Prepare state, is written as a new version of the card at the end of the transaction log. Because the log is compacted, older versions of the same card are cleaned up over time. When a new coordinator loads its partition, it reads the cards in order, and the last version of each card wins.

Why the state names matter for recovery

After a crash, the state on the card tells the new coordinator exactly what remains to be done. An Ongoing card means “wait for the producer, or abort on timeout.” A Prepare card means “the decision is made, finish delivering markers.” A Complete card means “nothing to do.” This simple rule turns recovery from a puzzle into a lookup.

One complete transaction from the first request to the final marker.

sequenceDiagram
  participant P as Producer
  participant TC as Transaction Coordinator
  participant TL as Transaction Log
  participant PL as Partition Leaders
  P->>TC: FindCoordinator for my transactional ID
  TC-->>P: I am the coordinator
  P->>TC: InitProducerId
  TC->>TL: Save producer ID and new epoch
  TC-->>P: Producer ID and epoch
  Note over P: beginTransaction is a local step
  P->>TC: AddPartitionsToTxn for orders-0 and stock-3
  TC->>TL: Save state Ongoing with partitions
  TC-->>P: OK
  P->>PL: Produce messages marked as transactional
  PL-->>P: Acknowledged
  P->>TC: EndTxn commit
  TC->>TL: Save PrepareCommit
  TC-->>P: Commit accepted
  TC->>PL: WriteTxnMarkers commit
  PL-->>TC: Markers written
  TC->>TL: Save CompleteCommit
    

The first conversation sets up everything that follows.

When the application starts, it creates a producer with a transactional ID and calls the initialization command. Behind the scenes, the client asks any broker to find the coordinator for that ID. The broker computes the partition of the transaction log that owns the name and answers with the address of that partition’s leader.

The producer then asks that broker for an identity. The coordinator looks in its memory for this transactional ID. If it has never seen the name, it creates a fresh producer ID and starts the epoch at zero. If it has seen the name before, it keeps the producer ID and increases the epoch by one. It writes the result into the transaction log, waits until the copies are safe, and replies.

Cleaning up the past

If the previous instance was in the middle of a transaction, the coordinator must finish it before answering. If the old transaction had already reached the “prepare commit” state, the coordinator completes the commit. If it was only ongoing, the coordinator aborts it. Only when the slate is clean does the new producer get its new epoch. This is how a restarted program never sees half-finished work from its earlier self.

The coordinator must know every partition a transaction touches, or it cannot finish the job later.

Suppose the producer wants to write to the “orders” topic, partition 0, for the first time within the current transaction. Before sending a single message, the producer tells the coordinator “I will write to orders-0.” The coordinator changes the transaction’s state to Ongoing if needed, records the partition in its list, saves the update in the log, and replies. Only then does the producer send the messages.

The coordinator decides first, and delivers second. That order is the secret of safety.

flowchart TD
  A["Producer asks to commit"] --> B["Coordinator saves PrepareCommit in the transaction log"]
  B --> C{"Saved on enough copies?"}
  C -->|"No"| B
  C -->|"Yes"| D["Point of no return - transaction WILL commit"]
  D --> E["Reply to producer - commit accepted"]
  D --> F["Send commit markers to every registered partition"]
  F --> G{"All partitions confirmed?"}
  G -->|"No - retry"| F
  G -->|"Yes"| H["Save CompleteCommit in the transaction log"]
  H --> I["Transaction finished - ready for next one"]
    

Why the order matters

If the coordinator delivered markers first and wrote the decision later, a crash in between could leave some partitions with “commit” and the coordinator with no memory of deciding. By writing first, the decision cannot get lost, and delivering the markers becomes a job that can always be repeated safely.

When work must be cancelled, the coordinator follows the same careful steps.

A transaction can be aborted by the producer itself, for example when an error occurs in the business logic. It can also be aborted by the coordinator, for example when the transaction stays open longer than allowed, or when a new instance of the same producer takes over. The path mirrors the commit: the coordinator saves “PrepareAbort,” delivers abort markers to every registered partition, and finally saves “CompleteAbort.”

The tiny records that tell every reader whether a transaction won or lost.

A transaction marker is a special record placed in a partition’s log at the end of a transaction. It contains no business data. It only says “the transaction from producer ID so-and-so has been committed” or “…aborted.” Because it is a real record in the log, it occupies an offset number, which is why you may notice that offsets sometimes skip a number in transactional topics. Readers never see the markers as messages; they use them as signposts.

Why markers live in each partition

A reader only talks to partition leaders, not to the coordinator. So the outcome must be written where the reader can see it: in the partition itself. The coordinator’s log holds the master truth, and the markers are the copies of the truth distributed to each place that needs it.

The other half of the promise: readers must choose to respect transactions.

Consumers have a setting called the isolation level. In the default level, called read-uncommitted, a consumer sees every message as soon as it is written, including messages of open transactions and even messages of transactions that will be aborted later. In the strict level, called read-committed, the consumer sees only messages of committed transactions, plus ordinary messages that were not part of any transaction.

The cost of being strict

Because strict readers cannot go past the last stable offset, a single long-running open transaction can hold back everything after it in that partition, even messages from other, already committed transactions. This is why transactions should be short. A transaction that stays open for minutes creates minutes of delay for every strict reader of the partitions it touched.

Let us watch one partition through a short story with real numbers.

Imagine one partition that is empty. Two transactional producers, Asha and Ravi, write to it at about the same time. We will follow the offsets one by one and see what a strict reader and a normal reader each see at every moment.

This is where transactions deliver true exactly-once processing inside Kafka.

In an ordinary program, the consumer saves its bookmark in one step and the producer writes outputs in another. A crash between the steps causes either lost work or duplicated work. In a transactional program, the producer tells the transaction “please include these bookmark updates,” and the bookmark and outputs commit together or not at all.

What if an old instance of your program wakes up after a new one has already taken over?

Programs sometimes freeze for a long time because of a network pause, a heavy garbage collection, or a stuck machine. Meanwhile, the orchestration system may decide the program is dead and start a replacement. Later the frozen program wakes up and continues as if nothing happened. Such a revived old instance is called a zombie. If two copies write the same outputs, duplicates and confusion follow.

Why this is such a strong protection

Fencing does not depend on clocks or on guessing who is alive. It depends on a number that only goes up and that every part of the system checks. This idea, used across many distributed systems, is one of the most reliable ways to prevent two active copies from causing damage.

Every open transaction is a small promise to finish soon. The coordinator enforces that promise.

These are typical defaults and may vary by version, so check the documentation for yours. The important idea is that the coordinator does not trust producers to finish. If the producer disappears with a transaction open, the coordinator aborts it after the timeout, which releases the readers who were waiting behind it.

Choosing the timeout

Set the transaction timeout comfortably longer than your longest normal transaction, but not much longer. Too short, and healthy but slow transactions get aborted. Too long, and a crashed producer can hold back strict readers for a long time. If your transactions normally take a second, a timeout of several seconds to a minute is usually plenty.

Forgetting idle IDs

If a transactional ID is unused for the full expiration time, the coordinator forgets it to save space. When the ID returns later, it is treated as new. This matters for rare batch jobs that run less often than the expiration time. They should not depend on remembered state between runs.

A referee who can collapse must have a replacement who knows exactly where the game stood.

Because the coordinator’s memory lives in a replicated log, losing a coordinator broker never loses the transactions. When the broker that leads a partition of the transaction log fails, the cluster’s own leader election picks a new leader for that partition from the copies that are fully caught up. In modern Kafka, the active controller of the KRaft controller quorum performs this choice and records it in the metadata log, exactly like any other partition.

Why loading takes a moment

Rebuilding memory from the log is not instant. For a coordinator responsible for a large number of transactional IDs, loading can take noticeable time, during which requests for those IDs receive a “coordinator loading” answer and the producer retries. Good clients handle this automatically. Keeping the number of transactional IDs reasonable and the log compacted keeps loading fast.

A design is only as good as the way it behaves when things go wrong.

A transaction that neither finishes nor gets cleaned up can freeze readers for a long time.

A hanging transaction is a transaction whose data messages are in a partition, but whose marker never arrived. Strict readers cannot move past it, because the last stable offset sits in front of it. Normally the coordinator’s timeout prevents this. In rare cases, such as certain software bugs, unusual failure sequences, or very old client versions, a transaction can be left hanging even though the coordinator no longer remembers it.

Most teams never call the transaction commands directly. They use them through bigger tools.

How the commit timer affects everything

With exactly-once processing turned on, Streams commits often, by default about every hundred milliseconds instead of every thirty seconds. Each commit is a full transaction. A shorter interval reduces delay for strict readers but increases the number of transactions per second, so it costs more coordinator work. A longer interval does the opposite. This one setting is the main dial for trading latency against overhead.

Kafka transactions protect what happens inside Kafka. Everything outside needs extra care.

Suppose a program reads an order from Kafka and also sends a confirmation email. The email service is not part of the Kafka transaction. If the program sends the email and then crashes before committing, the restart repeats the work and sends the email again. Kafka cannot take back an email.

Transactions add safety, and safety costs a little work. Here is where the cost goes.

Finding a balance

Very small transactions, such as one message each, waste effort: the coordinator work dwarfs the data. Very large transactions hold back strict readers and risk timeouts. A good rule is to group as many messages as fit within your acceptable delay, often somewhere between tens of milliseconds and a few seconds.

How the design scales

The coordinator’s work is spread across the fifty partitions of the transaction log, and therefore across many brokers. The data itself never passes through the coordinator; it goes straight to partition leaders. So adding brokers and partitions increases capacity without making one central machine a bottleneck. The main things that grow with scale are the number of transactional IDs and the number of partitions touched per transaction.

How much memory, disk, and attention does the coordinator need?

The coordinator keeps a small entry in memory for every transactional ID it manages, containing the producer ID, the epoch, the state, and the current list of partitions. Even a large number of IDs uses modest memory. The transaction log on disk is compacted, so it holds roughly one latest entry per ID, plus recent history not yet cleaned.

A special case: many short-lived IDs

An anti-pattern is to create a new transactional ID for every request or every job run. The coordinator then accumulates a huge number of entries that each live for a week before expiring. This wastes memory, slows failover, and adds no benefit. Design a small, stable set of IDs instead.

An honest look at what transactions give you and what they cost.

Kafka’s transaction feature has been refined many times since it first appeared.

The settings that matter, grouped by who owns them and described by their meaning.

Small cluster warning

Small development clusters with fewer than three brokers cannot honor the default replication of the transaction log. Some people lower the settings in development to make things start. That is fine for learning, but never carry those reduced settings to production, because the safety of the whole feature depends on them.

A producer that can run transactions can influence what many readers see, so its rights deserve care.

Kafka’s access rules treat the transactional ID as a resource, just like a topic. A producer needs permission to use its own transactional ID, permission to write to the topics it touches, and, when it saves consumer bookmarks inside the transaction, permission to read the consumer group. Idempotent writes also require a cluster-level right on older versions.

What to watch so you notice trouble before your users do.

Monitoring transactions comes down to four questions: Are transactions completing? Are they fast? Are any stuck? Are the coordinators healthy? Kafka exposes measurements for each, though exact names differ between versions, so treat this as a guide to the ideas.

Logs that tell the story

Coordinator logs mention state changes, timeouts, and epoch bumps for transactional IDs. Producer logs mention fencing, coordinator changes, and retries. Read them together in time order. A pattern such as “coordinator moved, loading, producer retried, committed” is normal. A pattern such as “fenced, restarted, fenced, restarted” points to two copies of the same program.

A short map of what you will use, described by what each thing does.

A daily and weekly routine

A gentle path from a simple at-least-once program to a transactional one.

Habits that keep transactions calm and predictable.

Useful design patterns

Three popular mistakes and the better ways to work.

Common mistakes in short

• Sharing one transactional ID between two running programs
• Setting the transaction log’s in-sync requirement to one
• Changing the transaction log’s partition count after use
• Committing single-message transactions at very high rates
• Ignoring aborted-transaction rates until disk use grows
• Assuming exactly-once extends to email, payments, or databases

From symptom to likely cause to first check.

The same ideas appear in many well-known systems.

Everything you learned, told as one small story.

QuickBite is a food-delivery company. When a customer pays, a service called payment-processor reads a “payment-requested” event, checks the customer’s balance, and writes three things: a “payment-confirmed” event, a “ledger-entry” event, and a “notification-request” event. It also saves its bookmark on the input topic. All four must succeed together.

A quick reference for the words used in this guide.

Quick answers to the questions beginners ask most often.

The whole story in a few short lines.