A single loop sits at the heart of stream processing. Understanding its steps is the foundation for everything else.

The four steps

A read-process-write application, sometimes called consume-transform-produce, repeats the same four steps forever. First it reads a batch of records from one or more input topics using a consumer. Second it processes them: filtering, enriching, aggregating, or converting each one. Third it writes the results to one or more output topics using a producer. Fourth it commits its consumed offsets, which is the bookmark that says “everything before this point has been dealt with”.

flowchart LR
    IN["Input topic partitions"] -->|"1. poll records"| APP["Application: process"]
    APP -->|"2. produce results"| OUT["Output topic partitions"]
    APP -->|"3. commit offsets"| OFF["Offsets topic"]
    OFF -->|"remembers position for restart"| IN
        

Why this pattern is everywhere

Crashes are not rare accidents. In a long-running cluster, they are a routine part of life.

What the offset commit really means

A committed offset is a promise to your future self. If the application crashes and restarts, or if another instance takes over its work, it will resume reading exactly at the committed offset. Records before that point are considered finished, and records after it will be read again. So the timing of the commit relative to the write decides whether a crash produces duplicates or losses.

Commit first, write second

If the application commits the offset before the results are written, a crash between the two steps loses data. On restart, the application resumes after the record it never finished, so the output for that record never appears. This is called at-most-once behavior: nothing is repeated, but something may vanish.

Write first, commit second

If the application writes the results and only then commits, a crash between those steps causes repetition. On restart, the application resumes at the old offset and processes the same records again, producing the same output a second time. This is at-least-once behavior: nothing is lost, but something may appear twice.

sequenceDiagram
    participant App as Application
    participant Out as Output topic
    participant Off as Offsets
    Note over App: Order A: write then commit
    App->>Out: Write result for records 10 to 19
    Note over App: Crash here
    App->>Off: Commit offset 20 (never happens)
    Note over App: Restart resumes at offset 10
    App->>Out: Write results for 10 to 19 again
    Note over Out: Duplicates in the output
        

sequenceDiagram
    participant App as Application
    participant Out as Output topic
    participant Off as Offsets
    Note over App: Order B: commit then write
    App->>Off: Commit offset 20
    Note over App: Crash here
    App->>Out: Write result for records 10 to 19 (never happens)
    Note over App: Restart resumes at offset 20
    Note over Out: Results for 10 to 19 are missing forever
        

Other sources of trouble

Crashes are not the only trigger. A slow application can be declared dead by the group coordinator, causing its partitions to be handed to another instance while it is still working. The original instance, now a “zombie”, may then write output and try to commit, even though somebody else owns that work. Network glitches can make a producer retry a send, creating duplicates on their own. Any complete solution must handle all of these.

Every term below is defined once here and then used consistently for the rest of the tutorial.

The whole design fits into one sentence: treat the output records and the input offsets as a single transaction.

The central trick

Look again at the loop. The output records live in the output topic. The consumed offsets live in an internal topic, because Kafka stores consumer offsets as ordinary records in a log. Both are just writes to Kafka logs. That observation is the key. If Kafka can make several writes to several logs succeed or fail together, then the output and the offset commit can succeed or fail together too.

flowchart TD
    A["Begin transaction"] --> B["Read records from input"]
    B --> C["Process records"]
    C --> D["Write results to output partitions, marked as part of the transaction"]
    D --> E["Add consumed offsets to the same transaction"]
    E --> F{"Everything succeeded?"}
    F -->|"Yes"| G["Commit: outputs become visible and offsets advance together"]
    F -->|"No, crash or error"| H["Abort: outputs stay hidden and offsets do not move"]
    H --> I["Restart from old offsets and redo the work"]
        

The two outcomes

After a crash, exactly one of two things is true. Either the transaction committed, in which case the outputs are visible and the offsets have advanced, so the restarted application starts after those records and does not repeat them. Or the transaction did not commit, in which case the outputs are invisible to committed-only readers and the offsets have not moved, so the restarted application redoes the same records from the start. In neither case is there a duplicate visible to readers, and in neither case is there a loss.

What has to be true for this to work

Several parts of Kafka cooperate to make an atomic read-process-write possible.

flowchart LR
    subgraph App["Your application instance"]
        CON["Consumer: read committed, manual offsets"]
        LOGIC["Processing logic"]
        PRO["Transactional producer"]
    end
    subgraph Brokers["Kafka cluster"]
        TC["Transaction coordinator"]
        TL["Transaction log topic"]
        IN["Input topic partitions"]
        OUT["Output topic partitions"]
        OFF["Consumer offsets topic"]
        GC["Group coordinator"]
    end
    IN --> CON
    CON --> LOGIC
    LOGIC --> PRO
    PRO -->|"begin, add partitions, add offsets, end"| TC
    TC --> TL
    PRO -->|"transactional writes"| OUT
    PRO -->|"offsets as part of transaction"| OFF
    CON --- GC
    TC -->|"write markers"| OUT
    TC -->|"write markers"| OFF
        

The roles, one by one

Transactions build on the idempotent producer’s identity, adding a name that survives restarts.

From a throwaway identity to a lasting one

A plain idempotent producer receives a fresh producer ID every time it starts, and its deduplication only covers that one session. A transactional producer sets a name, the transactional ID. When it starts, it contacts the transaction coordinator responsible for that name, and the coordinator returns the same producer ID as before, together with a higher epoch. The producer ID and epoch then travel with every batch the producer sends.

What fencing does for a read-process-write loop

Suppose one instance of your application freezes for a long pause. The cluster decides it is gone, and a replacement instance starts, using the same transactional ID. The coordinator gives the replacement a higher epoch. Now the frozen instance wakes up and tries to write. Brokers see its lower epoch and reject the write. The old instance learns it has been fenced and must stop. Without this, two instances could both write results for the same input records.

sequenceDiagram
    participant A as Old instance (epoch 4)
    participant TC as Transaction coordinator
    participant B as New instance
    participant Br as Partition leader
    Note over A: Long pause, cluster assumes it is gone
    B->>TC: Initialize with the same transactional ID
    TC-->>B: Same producer ID, epoch 5, abort any open transaction of epoch 4
    Note over A: Wakes up
    A->>Br: Write with epoch 4
    Br-->>A: Rejected, epoch too old
    Note over A: Fenced, must shut down
    B->>Br: Write with epoch 5
    Br-->>B: Accepted
        

Choosing the transactional ID

The name must be stable and unique per logical unit of work. Historically, a common rule was one transactional ID per input partition, so that when a partition moves between instances, the new owner uses the same name and fences the old owner. Newer designs, explained later, rely on the consumer group’s own membership information for fencing, which lets one producer serve many partitions. In either case, a random name on every start defeats the purpose, since the coordinator would treat each start as a completely new identity and nothing would be fenced or cleaned up.

Follow one complete transaction from the first poll to the final marker.

sequenceDiagram
    participant C as Consumer
    participant P as Transactional producer
    participant TC as Coordinator
    participant O as Output partition leader
    participant G as Offsets partition leader
    C->>P: Batch of input records
    P->>P: Begin transaction
    P->>TC: Add output partition to transaction
    TC->>TC: Record partition in transaction log
    P->>O: Send result records (flagged transactional)
    P->>TC: Add offsets partition to transaction
    P->>G: Send consumed offsets as pending
    P->>TC: End transaction: commit
    TC->>TC: Log prepare-commit
    TC->>O: Write commit marker
    TC->>G: Write commit marker
    TC->>TC: Log complete-commit
    TC-->>P: Commit acknowledged
        

What the application code actually does

From the programmer’s side, the shape is short: initialize once, then in a loop, poll, begin, send results, send offsets, commit. If any step throws, the code aborts the transaction and rewinds the consumer to the last committed positions so the same input is read again. The rest of the machinery is invisible. This simplicity is deliberate, but it hides a lot of coordination, which is why understanding the internals pays off during incidents.

Where the time goes in one transaction

It helps to picture a transaction as a small schedule of waits. The application spends most of its time polling and processing, which is ordinary work that any consumer does. Then come the coordination steps: registering each new partition, sending the offsets, and finally the commit request. The commit call returns only after the coordinator has recorded its decision and the markers are on their way, so the caller pays for a replicated write to the transaction log and, in many versions, for the marker writes as well. After the commit returns, committed-only readers still need a moment before they can read the new records, because the last stable offset advances only when the markers land in the partition.

Seeing the schedule laid out clarifies two common tuning choices. Reducing the number of distinct partitions a transaction touches shrinks both the registration and the marker phases. Keeping the coordinator brokers lightly loaded shrinks the commit phase. The processing phase, which is usually the biggest slice, is a matter for ordinary application profiling.

This single step is what separates a true read-process-write guarantee from a mere batch of transactional writes.

Why offsets must not be committed separately

If the consumer committed its own offsets in the usual way, that commit would be a separate action, outside the transaction. We would be back to the two-step problem: output in one place, bookmark in another, and a crash between them. To avoid this, the consumer’s automatic committing is switched off, and the application hands its offsets to the producer, which includes them in the transaction.

What happens to those offsets

The offsets are written to the internal offsets topic, like any offset commit, but flagged as belonging to an open transaction. Until the commit marker arrives, the group coordinator treats them as pending, so a restarted or replacement consumer still sees the old committed positions. When the commit marker is written, the pending offsets become the committed ones. If an abort marker is written instead, the pending offsets are discarded and the old positions remain.

flowchart TD
    A["Offsets for records 10 to 19 sent inside transaction"] --> B["Stored as pending in offsets topic"]
    B --> C{"Marker arrives"}
    C -->|"Commit marker"| D["Pending offsets become committed, group position moves to 20"]
    C -->|"Abort marker"| E["Pending offsets discarded, group position stays at 10"]
    D --> F["Restart resumes at 20, no duplicates"]
    E --> G["Restart resumes at 10, work is redone, aborted output stays hidden"]
        

Including the group membership information

Modern clients send the offsets together with the consumer group’s metadata: the group name, the current generation number, and the member identity. The group coordinator checks these when it receives the offsets. If the generation is old, because a rebalance has happened and the partitions have moved to someone else, the request is rejected. This is the mechanism that fences zombies at the level of the consumer group, and it is what allows one producer to serve many partitions safely.

The coordinator settles the fate of every transaction with a careful two-step protocol.

The problem it solves

A transaction may span many partitions, each led by a different broker. The commit must not be half done: no partition may show the outputs as committed while another still shows them as pending forever. The coordinator therefore uses the classic idea of two phases. Phase one is “decide and record the decision durably”. Phase two is “tell everyone”. Because the decision is recorded first, a crash during phase two can always be resumed.

The states of a transaction

flowchart LR
    E["Empty"] -->|"first partition added"| O["Ongoing"]
    O -->|"producer commits"| PC["Prepare commit"]
    O -->|"producer aborts or timeout"| PA["Prepare abort"]
    PC -->|"all commit markers written"| CC["Complete commit"]
    PA -->|"all abort markers written"| CA["Complete abort"]
    CC --> E
    CA --> E
        

Why a crash in phase two is harmless

Suppose the coordinator has recorded “prepare commit” and written markers to two of three partitions when it crashes. Another broker takes over as coordinator, reads the transaction log, and sees that this transaction is in the prepare-commit state. It knows the decision was to commit, so it finishes the job: it writes the missing marker and then records “complete commit”. Writing a marker twice is harmless, because markers are idempotent facts. The decision never flips, so the transaction never ends half committed and half aborted.

Transactions leave visible footprints inside the partition logs. Knowing them makes debugging far easier.

Data batches flagged as transactional

Every record batch that belongs to a transaction carries a flag in its header saying “this is transactional”, plus the producer ID and epoch. The broker writes it to the log like any other batch, immediately. There is no hidden staging area. Transactional data lands in the partition right away, and the question of whether readers may see it is answered later by the marker.

Control batches

When the transaction ends, the coordinator asks each partition leader to append a small special batch called a control batch. It holds a single control record of type commit or abort, again carrying the producer ID and epoch. Applications never see control records, since the consumer library filters them out, but they occupy offsets in the log.

flowchart LR
    A["Offset 100: normal record"] --> B["Offset 101 to 103: transactional batch, producer 42"]
    B --> C["Offset 104: normal record"]
    C --> D["Offset 105 to 106: transactional batch, producer 42"]
    D --> E["Offset 107: commit marker for producer 42"]
    E --> F["Offset 108: normal record"]
        

Aborted data stays on disk

An aborted transaction does not delete anything. Its records remain in the log until normal retention removes them, and brokers keep a small index that lists the ranges of aborted transactions so readers can skip them. This design keeps the write path fast, because aborts are cheap: writing a single marker is enough, with no need to rewrite or erase data.

The guarantee is only as strong as the readers that respect it. Here is how a consumer filters the log.

Two isolation levels

The last stable offset

A committed-only reader may only read up to the last stable offset, often shortened to LSO. This is the position just before the earliest transaction that is still open in that partition. Anything beyond it might belong to an unfinished transaction, and the broker will not hand it over yet. It is different from the high watermark, which only reflects replication. The LSO is never above the high watermark, and equals it when no transactions are open.

flowchart TD
    A["Consumer requests data from partition"] --> B["Broker returns records up to last stable offset"]
    B --> C["Response also lists aborted transaction ranges in the fetched span"]
    C --> D{"Consumer reads next batch"}
    D -->|"Batch belongs to an aborted transaction"| E["Skip batch"]
    D -->|"Control record"| F["Consume the marker and continue"]
    D -->|"Committed or normal batch"| G["Deliver to application"]
        

How aborted data is skipped

When the broker answers a fetch, it can also include the list of aborted transactions within the returned range, taken from its aborted-transaction index. The consumer library keeps track of which producer IDs currently have an aborted range covering the position it is reading. When it meets a batch from such a producer inside such a range, it drops the batch silently, and continues. The application never sees the records.

The cost: waiting for slow transactions

Because the LSO stops at the earliest open transaction, one slow or stuck transaction on a partition delays everything behind it for committed-only readers, even records written by other, healthy producers. Downstream lag grows, although the partition itself is healthy. This is why short transactions matter, and why hanging transactions are treated as incidents.

Precise promises are easier to trust than vague ones. Here is what you can count on.

sequenceDiagram
    participant PA as Producer A
    participant PB as Producer B
    participant L as Partition log
    participant R as Committed-only reader
    PA->>L: Offset 10 (transaction A)
    PB->>L: Offset 11 (transaction B)
    PB->>L: Commit marker B at offset 12
    Note over R: Cannot read past offset 9, transaction A still open
    PA->>L: Commit marker A at offset 13
    Note over R: Now reads offsets 10 and 11 in log order
        

Isolation is read committed, not serializable

Kafka transactions give you atomic visibility of writes, and protection against reading uncommitted data. They do not lock records or prevent two different applications from processing the same business entity concurrently. If two independent producers write conflicting facts about the same key in different transactions, both may commit. Business-level conflicts must be prevented by partitioning the work so that each key has a single writer, which the standard keyed-partition design does naturally.

A transaction that cannot finish must be undone cleanly. This chapter shows what that looks like in the loop.

When an abort happens

What the application must do after an abort

Aborting the transaction is only half the job. The consumer has already read past those records in memory, so if the application simply continued, it would skip them. After an abort, the application must rewind each input partition to its last committed offset, so the same records are read again in the next loop. Then, when the retry succeeds, the output appears exactly once for committed-only readers, since the earlier attempt’s output is hidden behind an abort marker.

flowchart TD
    A["Error during transaction"] --> B["Abort transaction"]
    B --> C["Coordinator writes abort markers to each involved partition"]
    C --> D["Pending offsets discarded, aborted output hidden"]
    D --> E["Consumer seeks each partition back to last committed offset"]
    E --> F["Next loop re-reads the same records"]
    F --> G{"Retry succeeds?"}
    G -->|"Yes"| H["Commit: output visible once, offsets advance"]
    G -->|"No, same error again"| I["Poison record: needs skipping or a dead letter path"]
        

Poison records and infinite loops

If one particular input record always causes a failure, then abort-and-retry becomes an endless loop, with the same record failing again and again while the partition makes no progress. A robust design catches errors for individual records and routes bad records to a separate dead letter topic, as part of the same transaction, so the loop can move past them. The dead letter write is transactional too, so it appears once, and the offset advances past the bad record in the same commit.

Partitions move between instances. The guarantee must survive every move.

Why rebalances are dangerous

In a consumer group, input partitions are distributed among instances. When an instance joins, leaves, or is declared dead, the group rebalances and partitions move. During that moment, the previous owner may still be running: it may be mid-transaction, or frozen, or slow. If both the old and the new owner processed the same partition at once, both would write output for the same input, and the results would be duplicated.

Fence by identity, or fence by group generation

sequenceDiagram
    participant A as Instance A (old owner)
    participant GC as Group coordinator
    participant B as Instance B (new owner)
    participant TC as Transaction coordinator
    Note over A: Slow, misses heartbeats
    GC->>GC: Rebalance, generation 7 to 8, partition moves to B
    B->>GC: Read committed offsets for the partition
    GC-->>B: Waits if A has pending offsets in an open transaction
    A->>GC: Send offsets with generation 7 in transaction
    GC-->>A: Rejected, generation is old
    Note over A: Transaction cannot commit, A aborts
    B->>TC: Begin own transaction, work proceeds on the partition
        

Cooperative rebalancing and transactions

Modern group protocols can move only some partitions at a time, so healthy instances keep working through a rebalance. Transactions fit this model: when a partition is taken away, the instance must finish or abort any open transaction covering it before continuing, and it must not send offsets for partitions it no longer owns. The safest simple behavior, and the one frameworks use, is to abort the current transaction whenever the assignment changes, and start fresh.

A transaction that never ends is the most disruptive thing that can happen to a committed-only reader.

The transaction timeout

Every transaction has a time budget, set by the producer, and capped by a broker-side maximum. If the budget runs out while the transaction is still ongoing, the coordinator aborts it on its own. This protects readers: without a timeout, a producer that crashed mid-transaction and never came back would hold the last stable offset back forever. The timeout should be longer than the longest normal processing time for one batch plus a safety margin, but not so long that a real crash blocks readers for many minutes.

Hanging transactions

A hanging transaction is one that has data in a partition but no marker, and whose coordinator does not know about it, or cannot complete it. In older versions, rare races between late writes and completed transactions could produce such a state. The last stable offset then stays stuck, and downstream readers lag steadily, even though every broker looks healthy. Recent protocol improvements close most of these races by having the producer and the coordinator agree more strictly on which partitions belong to which transaction, but operators should still know how to spot and fix the problem.

flowchart TD
    A["Downstream lag grows while partition is healthy"] --> B{"Is last stable offset far behind high watermark?"}
    B -->|"No"| C["Look elsewhere: slow consumer or processing"]
    B -->|"Yes"| D["List open transactions and producers on the partition"]
    D --> E{"Is the oldest open transaction still live and progressing?"}
    E -->|"Yes"| F["Wait or shorten transactions in the producing application"]
    E -->|"No, orphaned"| G["Use the transactions tool to find hanging transactions and abort them"]
    G --> H["Marker written, last stable offset advances, lag recovers"]
        

Marketing phrases blur edges. Engineers need the exact statement.

What Kafka promises

For a read-process-write application that reads from Kafka topics, writes only to Kafka topics, uses a transactional producer, sends its consumed offsets through that producer, and reads with committed-only isolation downstream, the following holds. The effect of processing each input record on the output topics is visible to committed-only readers exactly once, even if instances crash, restart, are fenced, or if brokers fail over. The consumed offsets advance if and only if the corresponding output is committed.

What the promise depends on

Common misreadings of the guarantee

The second myth deserves extra attention, because it has practical consequences. If your processing logic has side effects that are not idempotent, such as incrementing an in-memory counter that is later exposed through a web page, then a redone attempt will apply the side effect again. Keep the logic inside the transaction free of such effects, or confine them to state that is itself rebuilt from committed data after every abort.

The transaction boundary is a fence around Kafka. Everything on the other side of the fence needs its own protection.

Since the fence is real, engineers have developed reliable ways to cross it.

Pattern one: the idempotent sink

The receiving system accepts the same write repeatedly without changing the result. A database table with a unique key derived from the input record’s topic, partition, and offset, or from a business identifier, turns a repeated insert into a harmless no-op or an update to the same values. A search index that stores documents by identifier behaves the same way. This is the simplest, most robust pattern, and should be your first choice when the target supports it.

Pattern two: store the offset with the result

When the target is a transactional database, write the result and the input offset in one database transaction, and on restart, read the stored offset from the database and seek the consumer to it. The database becomes the authority on progress. This inverts the usual arrangement, but it gives atomicity between the result and the position without needing Kafka transactions at all.

Pattern three: the transactional outbox

A service must change its database and also publish an event. Doing both directly risks doing one and failing the other. Instead, the service writes the event into an outbox table in the same database transaction as the business change. A relay process later reads the outbox and publishes to Kafka, marking rows as sent. Duplicates are still possible if the relay crashes after publishing, so it uses an idempotent producer and gives each event a unique identifier that consumers can use to spot repeats.

flowchart LR
    A["Service handles request"] --> B["One database transaction"]
    B --> C["Update business tables"]
    B --> D["Insert event into outbox table"]
    D --> E["Relay reads new outbox rows"]
    E --> F["Idempotent producer publishes to Kafka"]
    F --> G["Relay marks rows as sent"]
    F --> H["Consumers deduplicate by event identifier"]
        

Pattern four: command events and dedicated actors

Rather than performing an external side effect inside the loop, write a command event, such as “send this email with this identifier”, to a topic. A small dedicated service reads those commands and performs the effect using the identifier as an idempotency key with the external system. This keeps the main loop purely Kafka-to-Kafka, exactly once, and isolates the difficult external behavior into one place that can be designed and tested carefully.

When the processing step remembers things, the memory must be part of the guarantee too.

The state problem

Many pipelines keep state: a running count per customer, the latest value per key, a window of recent events. If the state is stored in a local database on the application’s disk, a crash and a redo would apply the same update to the state twice, or lose it. Yet the whole point of the transaction was that processing effects happen exactly once.

The change-log solution

Stream processing libraries keep local state for speed, but every change to it is also written to a compacted change-log topic in Kafka. That write is part of the same transaction as the output and the offsets. After a crash, the state is rebuilt by replaying the committed part of the change-log. Aborted updates are hidden by their abort marker, so the rebuilt state contains exactly the effects of committed work.

flowchart TD
    A["Read input record"] --> B["Update local state store"]
    B --> C["Write state change to change-log topic"]
    C --> D["Write output record"]
    D --> E["Add input offsets"]
    E --> F{"Commit transaction"}
    F -->|"Commit"| G["State, output, and offsets all durable together"]
    F -->|"Abort"| H["Change-log and output hidden, local state restored from committed change-log"]
        

The library setting

In Kafka Streams, one configuration value selects the processing guarantee. The modern exactly-once option, often called version two, uses one transactional producer per stream thread, relies on the consumer group’s metadata for fencing, and requires reasonably recent brokers. An older option used one producer per task and is deprecated. The at-least-once option is the default and skips the transaction overhead.

Connectors

Data integration tools sit at the fence. A source connector reading from an external system can use transactions to write its records and its own progress markers together, with support in recent releases. A sink connector writing to an external system generally offers at-least-once delivery, and relies on idempotent sinks for exactly-once effects, following the patterns of the previous chapter.

Hand-written loop or library?

Writing the loop yourself gives full control, but it also makes you responsible for every subtle duty: aborting on rebalance, rewinding after an abort, treating fencing as fatal, closing producers cleanly, and rebuilding state. A stream processing library has already solved these problems, and has been tested against thousands of failure combinations. Unless your logic is very simple or your needs are unusual, the library is usually the safer choice.

A small, concrete story makes the abstract rules easy to remember.

The setting

An enrichment service reads order events from partition 3 of an input topic and writes enriched orders to partition 1 of an output topic. Its consumer group has committed offset 200 for the input partition. It uses a transactional producer with a stable transactional ID. The service polls and receives records at offsets 200 to 204.

Attempt one: a crash mid-transaction

The service begins a transaction, enriches the five records, and sends five output records, which land in the output partition at offsets 700 to 704. It then sends the offset 205 as the position to commit, which is stored as pending. Before it can call commit, the process is killed.

Attempt two: the replacement instance

A replacement instance starts with the same transactional ID. The coordinator sees an old open transaction, aborts it, and writes an abort marker into the output partition at offset 705 and into the offsets partition. The replacement reads the committed group position, which is still 200. It polls the same five records, processes them, and begins a new transaction. This time the transactional output lands at offsets 706 to 710, the pending offset 205 is stored again, and the commit succeeds. Commit markers are written, the group position becomes 205, and the output at 706 to 710 becomes visible.

sequenceDiagram
    participant A as Instance A
    participant TC as Coordinator
    participant O as Output partition
    participant B as Instance B
    A->>O: Enriched records at 700 to 704 (transaction 1)
    A->>TC: Offsets pending, position 205
    Note over A: Crash, no commit request sent
    B->>TC: Initialize with same transactional ID
    TC->>O: Abort marker at 705
    TC-->>B: New epoch, old transaction aborted
    B->>O: Enriched records at 706 to 710 (transaction 2)
    B->>TC: Offsets, position 205, then commit
    TC->>O: Commit marker at 711
    Note over O: Committed-only readers see 706 to 710 once
        

What a downstream reader observes

A committed-only reader of the output partition reads the offsets in order. It skips 700 to 704, since they belong to an aborted transaction, consumes the control record at 705 silently, delivers the five records at 706 to 710, and consumes the commit marker at 711. From the application’s viewpoint, five enriched records appeared exactly once. The offsets jumped from 699 straight to 706 in what the application receives, which is the visible sign of the abort. A default reader, in contrast, would have delivered the records at 700 to 704 as well, producing ten records instead of five.

A handful of settings on three components decide whether the guarantee really holds.

Things that quietly break the guarantee

Three environments, three postures

The same application often runs in several environments, and the safe posture differs in each. A laptop or a continuous integration job wants speed of startup and tiny footprint, a staging cluster wants to behave like production so that failures are discovered early, and production wants maximum durability. Mixing these postures is a frequent source of surprises, so write them down explicitly and keep them in version control.

The danger lies in the first row leaking into the third. A single-replica transaction log is convenient on a laptop but means that losing one broker in production could lose the state of every open transaction. Treat any environment file that mentions a transaction log replication of one as suspect, and require a code review before it can ever be applied to a shared cluster.

Atomicity costs something. The cost is smaller than most people fear, and it is controllable.

Where the extra cost comes from

Amortizing with batch size

The fixed cost of a transaction is spread over all the records in it. A transaction that carries a single record pays the full coordination cost for one record. A transaction that carries a thousand records pays it once for all thousand. This is why the commit interval is the main tuning knob: a longer interval means larger transactions, higher throughput, and higher end-to-end latency, and a shorter interval means the reverse.

flowchart LR
    A["Shorter commit interval"] --> B["Smaller transactions"]
    B --> C["More coordinator work per record"]
    B --> D["Lower latency for readers"]
    E["Longer commit interval"] --> F["Larger transactions"]
    F --> G["Less overhead per record"]
    F --> H["Higher latency, more work redone after an abort"]
        

The cost of redoing work

Larger transactions carry another cost: after a failure, everything in the open transaction is redone. A ten-second transaction that fails after nine seconds wastes nine seconds of processing. Choose an interval that balances overhead against the acceptable amount of repeated work after a failure.

Partition fan-out

A transaction that writes to hundreds of partitions writes hundreds of markers and registers hundreds of partitions with the coordinator. If your pipeline fans out widely, consider whether every output really belongs in one transaction, or whether the output can be split into several loops with narrower fan-out.

Sizing the coordinator side

The transaction coordinator role is spread across the brokers that lead partitions of the transaction log. Each transactional ID is assigned to one of those partitions by hashing its name, so the load of many producers spreads out on its own, provided there are enough transactional IDs and enough log partitions. A cluster with very few transactional IDs concentrates all coordination on a handful of brokers, while a cluster with a huge number of short-lived IDs fills the transaction log with state that must later expire.

As a rule of thumb, watch the size and replication health of the transaction log topic, the number of active transactional IDs, and the request latency of the coordinator role. If any of them drifts steadily upward, investigate the producers before adding brokers.

Transactions add new resources that need access control, and new ways to interfere with other applications.

Data visibility and privacy

Aborted records stay on disk until retention removes them. If a transaction wrote sensitive data and then aborted, the data still exists in the log for the retention period. Access to the topic is therefore access to the aborted data as well, and privacy or deletion requirements must account for it. Compacted or short-retention topics reduce the window.

A healthy transactional pipeline is quiet. These signals tell you when it stops being healthy.

Tools that ship with Kafka

flowchart TD
    A["Alert: downstream lag rising"] --> B{"Is application input lag also rising?"}
    B -->|"Yes"| C["Processing is too slow: scale out or tune the batch"]
    B -->|"No"| D{"Is last stable offset stuck below high watermark?"}
    D -->|"Yes"| E["Find the oldest open transaction, check its owner, abort if orphaned"]
    D -->|"No"| F{"Are aborts and fencing errors increasing?"}
    F -->|"Yes"| G["Check rebalances, timeouts, and deployment activity"]
    F -->|"No"| H["Check downstream consumers and network"]
        

Walk through the typical failures one at a time, and see what each leaves behind.

flowchart TD
    A["Failure occurs"] --> B{"Where in the transaction?"}
    B -->|"Before commit requested"| C["Transaction will abort by timeout or takeover: output hidden, offsets unchanged"]
    B -->|"After prepare-commit recorded"| D["Commit is certain: coordinator completes markers"]
    B -->|"During marker writing"| E["New coordinator resumes: missing markers written, then complete"]
    C --> F["Restart at old offsets, redo work"]
    D --> G["Restart at new offsets, no repeat"]
    E --> G
        

Slow-motion failures that look like something else

Not every problem announces itself with a crash. Some of the most confusing incidents in transactional pipelines are slow degradations whose symptoms appear far from their cause. Learning to recognize them saves hours.

Some designs repeatedly work well. Others repeatedly fail in ways that only show up under load or failure.

Patterns worth copying

Anti-patterns to avoid

A checklist for design reviews, code reviews, and production readiness reviews.

A production readiness review

The same loop appears in very different industries, each with its own reasons to demand correctness.

You can, and for many workloads that is the right choice. It gives at-least-once behavior. If duplicates are expensive, as with money or counters, a transaction removes them at moderate cost.

No. Transactions in Kafka control the visibility of writes. They do not lock input records or prevent other applications from reading them. Business conflicts are avoided by giving each key a single processing owner.

They stay in the log until retention deletes them, but committed-only readers skip them, using the abort marker and the broker’s index of aborted ranges.

Control records for commit and abort markers occupy offsets, and aborted data is skipped. Both make the delivered offsets non-consecutive, which is normal.

No. The high watermark reflects replication, while the last stable offset also accounts for open transactions. It is never higher than the high watermark, and equals it when nothing is open.

For committed-only readers, yes. Readers cannot pass the earliest open transaction on a partition, so records written after it by other producers wait too.

Not with modern clients and brokers. Fencing is based on the consumer group generation, so one producer per thread is enough. Older designs needed one producer per input partition.

Treat the call as outside the guarantee. Use an idempotency key so repeats are recognized, or emit a command event and let a dedicated service perform the call.

Somewhat, mostly through added latency and coordination work per transaction. With sensible batch sizes the throughput cost is usually modest, but you should measure your own workload.

Catch the error for that record and write it to a dead letter topic inside the same transaction, so the loop can move past it. Otherwise abort-and-retry loops forever on the same record.

Another broker takes over, reads the replicated transaction log, and finishes the job from the recorded state. A recorded commit decision is never reversed.

Not when configured correctly. It is a replicated internal topic. Use three replicas and a minimum of two in-sync replicas so the loss of one broker does not stop transactions or lose state.

The standard pattern relies on group offsets and group metadata. For consumers that manage positions themselves, the safe alternative is to store the position together with results in the target system.

It delays data until transactions complete. Consumers that can tolerate aborted or duplicate data and want the lowest latency may choose the default, but they give up the guarantee.

Yes. Change-log topics used for state are compacted and transactional at the same time. Aborted records are ignored by readers, and compaction removes old values in the background.

Run a fault-injection test: send uniquely numbered inputs, kill and restart instances and brokers during processing, then read the output with committed-only isolation and confirm each input appears once.

A batch. The fixed cost of coordination is spread across all records in the transaction, so larger batches raise throughput. Choose the batch so the added latency and the amount of work redone after an abort are both acceptable.

It reads them to learn where transactions end, uses them to decide which data to deliver or skip, and then discards them. They never reach the application, but they do take up offsets.

Yes, but readers of the output must handle both. Committed-only readers deliver ordinary non-transactional data as soon as it is stable, so they work with a mixed topic. Plan the switch carefully, and check that no reader still uses default isolation.

Any open transaction stays open until its timeout expires, and then the coordinator aborts it. During that time it can delay committed-only readers. Shut applications down gracefully, so they can commit or abort before exiting.

Adding partitions changes how keys map to partitions, which can break per-key ordering and state assumptions for stateful loops. Plan such changes carefully, ideally with a drain and restart, and test them in staging.

So the group coordinator can verify that the sender is still a current member of the group at the right generation. A stale member is rejected, and that check is what fences zombies without needing one producer per partition.

Per output partition, records appear in the order they were written, which follows the processing order for a single instance. Across different output partitions there is no ordering guarantee, as is true for Kafka in general.

Fencing errors, meaning a newer holder of the identity exists, are fatal for that instance and it should stop. Temporary abortable errors allow an abort, a rewind, and a retry with the same producer. Check the documentation of your client version for the precise list.

A read-process-write loop touches an input log, an output log, and an offsets log, and a crash between any two touches breeds either duplicates or losses. No ordering of the steps can fix this. Kafka’s answer is to make the output records and the consumed offsets a single transaction, coordinated by a broker that records its decision durably before telling any partition, and to let committed-only readers respect the outcome. Fencing through epochs and consumer group generations stops zombie instances, timeouts and tooling deal with stuck transactions, and stream processing libraries wrap the whole pattern, including state, into one setting. The promise is precise and bounded: exactly-once effects inside Kafka, and idempotent designs at every boundary outside it.