Before we can talk about saving a position, we need to understand what a position actually is.

Apache Kafka is a system for moving streams of events between applications. An event might be a customer placing an order, a sensor reporting a temperature, or a user clicking a button. Events are grouped into named categories called topics. You can think of a topic as a labelled shelf in a very large library, where every book on that shelf is about the same subject.

A single topic can be very large, so Kafka splits it into smaller pieces called partitions. Each partition is an ordered, append-only log. New events are always added to the end, and once written, an event is never changed in place. Because each partition is its own separate log, Kafka can spread the partitions across many servers, which is how it handles enormous amounts of data.

What exactly is an offset?

Every event inside a partition receives a sequence number, starting at zero and increasing by one for each new event. That number is the offset. It is unique only within its partition, so the first event of partition 0 and the first event of partition 1 both have offset 0. To point at any single event in Kafka, you need three pieces of information: the topic, the partition, and the offset.

Why consumer groups matter here

Consumers rarely work alone. A consumer group is a set of consumers that share the same group name and cooperate to read a topic. Kafka gives each partition to exactly one consumer in the group at a time, so the work is divided without two members reading the same partition simultaneously. If a topic has six partitions and the group has three consumers, each consumer typically reads two partitions.

This is important because the saved position is remembered per group, not per individual consumer. If one consumer crashes, another member of the same group takes over its partitions and continues from the group’s last saved position. That handover only works if the position was saved correctly, which brings us to committing.

Committing is the act of telling Kafka, “I am done with everything before this point.”

When a consumer reads events, it keeps a position in memory that moves forward with every batch it receives. This in-memory position is private to the running process. If the process crashes, restarts, or is replaced by another machine, that memory is gone. A new consumer would have no idea where to begin.

To solve this, Kafka lets consumers save their position to the cluster itself. This save operation is called committing an offset. The saved value is called the committed offset. When any consumer in the group later needs to know where to start for a given partition, it asks the cluster for the committed offset and resumes from there.

The next-message rule

Here is a detail that trips up many beginners. The committed offset is not the offset of the last event you processed. It is the offset of the next event you want to read. If you finished processing the event at offset 41, the value you commit is 42. Think of it as the bookmark pointing to the first page you have not yet read, not the last page you finished.

Where is the committed offset stored?

Kafka stores committed offsets inside itself, in a special internal topic named __consumer_offsets. Each commit is written as a small record whose key is the combination of group name, topic and partition, and whose value is the committed offset along with a little metadata such as a timestamp. Because it is an ordinary replicated Kafka log, the saved positions enjoy the same durability as your data. Older versions of Kafka used ZooKeeper for this purpose, but modern versions have used the internal topic for many years.

The default behaviour is convenient, but convenience hides a gap between what was read and what was truly finished.

By default, a Kafka consumer commits offsets on its own. A setting called enable.auto.commit is turned on, and another setting, auto.commit.interval.ms, defaults to five seconds. Each time the application asks for more events, the client checks whether the interval has passed, and if so, it quietly commits the positions returned by the previous request.

For a quick experiment or a low-stakes pipeline, this is fine. But there is a subtle problem. The client has no idea what your application did with the events. It only knows that it handed them to you. Whether you wrote them to a database, called a payment service, or crashed halfway through is invisible to the client.

Two ways automatic commits can hurt

The first problem is data loss. Suppose the consumer receives 500 events and the auto-commit timer fires when you ask for the next batch, committing the whole batch. If your application crashes while it is still in the middle of processing that batch, the replacement consumer starts after the committed position. The events you never finished are skipped forever.

The second problem is duplicate processing. Suppose you have processed 400 of 500 events, but the timer has not fired yet, and then the application crashes. The replacement consumer starts from the last commit, which is before this batch. All 400 finished events are processed a second time. Without safeguards, a customer might be charged twice or receive the same email twice.

sequenceDiagram
    participant K as Kafka Partition
    participant C as Consumer
    participant B as Auto Commit Timer
    K->>C: Deliver batch of events 100 to 199
    B->>K: Timer fires, commit position 200
    C->>C: Still processing event 130
    C--xC: Application crashes
    Note over K,C: New consumer resumes at 200
    Note over K,C: Events 130 to 199 are lost
        

Manual offset commits hand the decision of “when is this really done” back to your application code.

With manual offset commits, you switch off automatic committing and take responsibility for telling Kafka when a position should be saved. Your code reads a batch of events, does the real work, and only when the work has succeeded does it call the commit operation. The consumer client never commits on its own behalf. The saved position therefore always reflects what your application has decided is complete.

The idea is simple, but its consequences are large. You now choose the moment of the commit, the granularity of the commit, whether the commit blocks or runs in the background, and what happens when a commit fails. Each choice trades speed against safety, and understanding those trade-offs is the heart of this tutorial.

How to switch it on

Turning on manual control needs one configuration change: set enable.auto.commit to false. From that moment the client stops committing by itself. Your application must then call one of the commit operations provided by the consumer library. Frameworks built on top of Kafka usually expose this as an acknowledgement mode, where your handler explicitly acknowledges each record or batch once it is finished.

Once you control the commit, the next question is whether your code should wait for the answer.

The Kafka consumer offers two flavours of manual commit. They save the same kind of information but behave very differently when you call them, and each has situations where it shines.

Synchronous commit

A synchronous commit sends the request to the cluster and waits until the cluster answers. If the answer is a temporary problem, such as a brief network glitch or a leadership change, the client retries automatically until the commit succeeds or a permanent problem occurs. Only then does control return to your code. If something unrecoverable happens, an exception is raised so that you know for certain the commit did not succeed.

The strength of this mode is certainty. When the call returns quietly, the position is saved. The weakness is speed. Every commit pauses your loop for a network round trip, which can noticeably reduce throughput if you commit very frequently.

Asynchronous commit

An asynchronous commit sends the request and returns immediately, letting your loop carry on reading and processing while the commit travels in the background. You may attach a callback that runs later to tell you whether the commit worked. This keeps throughput high because the loop is never stalled by the network.

The catch is that the client deliberately does not retry a failed asynchronous commit. The reason is ordering. Imagine commit A for position 100 fails and is retried, while commit B for position 200 has already succeeded. If A’s retry lands afterwards, it would overwrite the newer position with an older one, and after a restart you would reprocess events 100 to 199 needlessly. Skipping the retry avoids that; the next commit will carry a newer position anyway.

The popular combined approach

Many production consumers use both. During the normal loop they commit asynchronously for speed, accepting that an occasional commit may fail because a later one will succeed. Right before the consumer shuts down, or just before it gives up its partitions, they perform one final synchronous commit so that the last position is guaranteed to be saved. This gives near-maximum throughput in steady state and a firm guarantee at the moments where it matters most.

You decide not only when to commit, but how much progress each commit records.

Granularity means the size of the step between two commits. Kafka lets you commit at several levels, and the right level depends on how expensive a duplicate is compared with how expensive a commit is.

Commit after each batch

The most common pattern is to commit once after your application has finished an entire batch returned by the consumer. The client provides a simple form of commit that saves the latest positions of every partition it has handed you so far. It is cheap because there is one request per batch, and it is easy to reason about. The cost is that a crash in the middle of a batch causes the whole batch to be redelivered.

Commit after each event

At the other extreme, you can commit after every single event. This minimises how much work is repeated after a failure, but it multiplies the number of commit requests dramatically. At high volumes this places noticeable load on both your consumer and the brokers hosting the internal offsets topic. It is appropriate only when each event is very expensive to reprocess and the volume is modest.

Commit specific partition positions

The consumer also lets you pass an explicit map of partitions to offsets. This gives fine control. You can commit only the partitions you have completely finished, while others are still being worked on, or commit progress after every fixed number of events. It is especially useful when work is handed to a pool of worker threads that finish at different times, since you can commit only the contiguous prefix of work that is fully done.

The order of “process” and “commit” decides which promise your system can make.

Delivery semantics describe how many times an event affects your system when failures happen. There are three well-known levels. Manual commits are the tool that lets you choose between them deliberately, instead of accidentally.

At-most-once

If you commit first and process afterwards, then a crash between the two steps means the event is marked done but was never handled. The event is delivered at most one time and might be delivered zero times. This is the loss-prone mode. It can be acceptable for very high-volume telemetry where a missing reading does not matter.

At-least-once

If you process first and commit afterwards, then a crash between the two steps means the work was done but the position was not saved. After restart, the event arrives again. The event is delivered at least one time and possibly more. This is the most popular choice for business systems because nothing is lost. The price is that your processing must cope with duplicates.

Exactly-once

Exactly-once means the effect of each event appears once, even when retries happen. Kafka offers this inside its own world through transactions: a processor can write its output events and commit its input offsets in one atomic transaction, so either both happen or neither does. Consumers reading that output must be configured to see only committed transactional data. Outside Kafka, for example when writing to a relational database, you reach the same result by combining at-least-once delivery with idempotent handling or by storing the offset in the same transaction as the business data.

flowchart TD
    A["Read batch from Kafka"] --> B{"Order of steps"}
    B -->|"Commit first, then process"| C["At-most-once"]
    B -->|"Process first, then commit"| D["At-least-once"]
    B -->|"Process and commit in one atomic step"| E["Exactly-once effect"]
    C --> F["Risk: lost events"]
    D --> G["Risk: duplicate events"]
    E --> H["Needs transactions or idempotency"]
        

A commit is not magic. It is a small request handled by a specific broker with a specific job.

Each consumer group is managed by one broker acting as its group coordinator. The coordinator is chosen by taking the group name, mapping it to one partition of the internal offsets topic, and using the broker that leads that partition. The same broker therefore handles both the membership of the group and the storage of its committed offsets.

sequenceDiagram
    participant App as Application Code
    participant Cl as Consumer Client
    participant Co as Group Coordinator Broker
    participant Lg as Offsets Topic
    App->>Cl: Finished batch, request commit
    Cl->>Co: Offset commit with group and generation
    Co->>Co: Validate membership and assignment
    Co->>Lg: Append commit record
    Lg-->>Co: Replicated and durable
    Co-->>Cl: Success response
    Cl-->>App: Return or run callback
        

What happens when a consumer starts

When a consumer joins a group and receives its partitions, it asks the coordinator for the committed offset of each one. If a value exists, reading resumes there. If none exists, for example because the group is brand new or its saved offsets have expired, a setting named auto.offset.reset decides what to do. Its choices are to start from the earliest available event, start from the latest, or raise an error. This choice has serious consequences, which we return to in the best-practices chapter.

Offset expiry

Committed offsets do not live forever for groups that go idle. Once a group has had no active members for a configured retention period, which defaults to seven days in recent versions, the cluster removes its saved positions. A consumer that returns after a long holiday shutdown may find its bookmark gone and fall back to the reset policy, causing either a huge replay or a silent skip to the newest data.

Committing correctly is tied closely to how the consumer proves it is still alive.

A consumer application is built around a loop. In each turn it asks Kafka for a batch of events, processes them, and asks again. That request for events is called polling, and it is more than a data fetch. Polling is also how the client signals that the application is healthy and still making progress.

Three settings govern this. The maximum number of events returned per poll defaults to 500. The maximum allowed time between two polls, called the poll interval, defaults to five minutes. A separate background heartbeat thread tells the coordinator every few seconds that the process is alive, and if no heartbeat arrives within the session timeout, the member is declared dead.

Why this matters for manual commits

With manual commits, your processing time sits between polls. If a batch takes longer than the poll interval to process, the coordinator concludes the consumer is stuck, removes it, and gives its partitions to someone else. When your slow consumer finally tries to commit, its commit is refused because it no longer owns those partitions. The events it processed will be handled again by the new owner, producing duplicates.

The most delicate moment for any manual commit is when the group reshuffles its partitions.

A rebalance is the process by which the group redistributes partitions among its members. It happens when a consumer joins, a consumer leaves or is removed, or the number of partitions changes. During a rebalance, partitions are taken away from some members and given to others.

This is where manual commits earn their keep. Before a partition is taken away, the current owner has a brief opportunity to save its progress. The consumer library lets you register a listener that is notified just before partitions are revoked and just after new ones are assigned. In the revocation notification, a well-behaved application finishes or abandons the in-flight work for those partitions and commits a final synchronous offset. The next owner then starts exactly where the previous one stopped, with almost no duplication.

flowchart LR
    A["Consumer A owns partitions 0 and 1"] --> B["Consumer B joins the group"]
    B --> C["Rebalance begins"]
    C --> D["Consumer A notified: partition 1 will be revoked"]
    D --> E["Consumer A finishes work and commits synchronously"]
    E --> F["Partition 1 assigned to Consumer B"]
    F --> G["Consumer B reads committed offset and resumes"]
        

Eager and cooperative rebalancing

Older rebalancing worked in an eager style: every member gave up every partition, and everything was reassigned from scratch, causing a short pause across the whole group. Modern Kafka supports cooperative rebalancing, where only the partitions that actually need to move are revoked, while the rest keep flowing. For manual commits this is a gift, because fewer partitions are disturbed, and fewer in-flight batches are exposed to the duplicate risk.

The dreaded commit failure

If a consumer tries to commit after it has already lost its partitions to a rebalance, the coordinator rejects the request with a commit failure exception. This is not a bug; it is the cluster protecting the committed position from being overwritten by a stale member. Your application should treat it as a signal: the work from that batch will be repeated by another consumer, so anything you did must tolerate being seen again.

Let us follow a single order event from the moment it is written until its position is safely saved.

flowchart TD
    A["Producer writes order event to partition"] --> B["Consumer polls and receives a batch"]
    B --> C["Application validates and processes each event"]
    C --> D{"All events in batch succeeded?"}
    D -->|"Yes"| E["Commit next offset"]
    D -->|"No, retryable error"| F["Retry with backoff"]
    F --> C
    D -->|"No, poison event"| G["Send to dead letter topic"]
    G --> E
    E --> H["Coordinator stores position in offsets topic"]
    H --> B
        

Notice the gap between steps 3 and 4. That gap is the window in which a crash produces a duplicate. Manual commits do not remove the gap; they make it visible, small, and yours to manage. Everything in the next chapters is about living safely with that gap.

Manual control forces you to decide what “success” means when something goes wrong.

With automatic commits, a failing event is often ignored by accident: the position moves on regardless. With manual commits, a failing event blocks progress until you decide what to do with it, because committing past it would mean giving up on it. This is good, since nothing disappears silently, but it demands a clear policy.

Temporary failures

Some failures are temporary: a database is briefly unreachable, a downstream service times out, a network blip occurs. The right response is to retry, usually with a growing delay between attempts. Do not commit while the retry is in progress. Be mindful of the poll interval, since long retry loops can push you beyond the allowed time between polls.

Poison events

A poison event is one that can never succeed, no matter how many times it is retried. Examples include malformed data, a missing required field, or a value that breaks a business rule. If you retry a poison event forever, the partition is stuck behind it, and every event after it waits. This is sometimes called head-of-line blocking.

The standard cure is a dead letter topic. After a limited number of attempts, or immediately for clearly invalid data, the application writes the failing event, together with the error reason, into a separate topic dedicated to problems. It then commits past the bad event so the healthy flow continues. People or automated tools can inspect the dead letter topic later, fix the cause, and replay the events.

Since at-least-once means events may repeat, the safest systems are the ones where repeats do no damage.

An operation is idempotent when doing it once has the same final effect as doing it many times. Setting a light switch to “on” is idempotent: pressing it ten times leaves the light on. Adding one to a counter is not idempotent: pressing ten times gives ten.

If your processing is idempotent, then the duplicate window between processing and committing stops being a problem. A replayed event simply reproduces the same result. This is the most practical route to correct behaviour in systems that cannot use Kafka transactions, such as those writing into an ordinary database or an external service.

Practical ways to achieve idempotency

Sometimes the safest place for the bookmark is right next to the data it describes.

Kafka’s own commit store is excellent, but it lives in a different system from your database. That means saving your business result and saving your position are two separate actions that cannot be made atomic. There is always a tiny window between them.

An alternative is to store the offset yourself, in the same database and the same transaction as the business data. When your application writes the processed result, it also writes the partition and next offset in one transaction. Either both are saved or neither is. On startup, the consumer reads the saved positions from the database and explicitly tells Kafka to continue from them, ignoring Kafka’s own committed values.

Every gain in control comes with a cost in effort. An honest comparison helps you choose.

A commit that cannot be seen or protected is a risk waiting to happen.

Security considerations

Committed offsets are stored in the internal offsets topic, and access to them is controlled by permissions on the consumer group resource. A consumer needs read permission on the topic it consumes and also read permission on its group. Without the group permission, commits are refused. Restricting group permissions by name prevents one team’s application from reading or overwriting another team’s saved positions.

Positions are quite valuable to an attacker or a careless operator. Resetting a group’s offsets backwards forces a mass replay that might resend emails or repeat payments. Resetting them forwards silently skips data. Limit who is allowed to reset offsets, log every reset, and prefer tooling that requires an explicit confirmation step. Also remember that data in transit between consumers and brokers should be encrypted, since commit requests reveal your group and topic names.

What to monitor

The single most useful indicator is consumer lag: the difference between the newest offset in a partition and the committed offset of the group. A steadily growing lag means the consumers are not keeping up, or that commits have stopped even though processing continues. Monitoring should track lag per partition, not just a total, since one stuck partition can hide behind healthy averages.

Logging that helps in emergencies

Log each commit failure with the group, partition and offset involved, and log each rebalance with the partitions gained and lost. When a duplicate or a gap is reported by the business, these logs let you reconstruct exactly which positions were saved and when. Without them, investigations turn into guesswork.

Some approaches have proven themselves in many teams, while others reliably cause incidents.

Patterns worth adopting

Anti-patterns to avoid

A short checklist of habits that separate reliable consumers from fragile ones.

Frequent mistakes, at a glance

The same idea shows up wherever an event stream must be trusted.

Quick answers to the questions that come up most often.

No. Committing only saves a position for the group. The events remain in the partition until the topic’s retention rules remove them, and other groups can still read them independently.

Yes. Each group keeps its own bookmark. This is why several applications can consume the same topic at different speeds without interfering with each other.

No. They guarantee that the position is saved only when you say so. A crash between finishing work and committing still leads to a repeat. Prevent harm by making handlers idempotent or by using transactions.

It gives you certainty about the result and retries temporary errors, so it is safer for a single call. But an application that combines asynchronous commits in the loop with a final synchronous commit is normally just as safe overall, and much faster.

The consumer keeps reading forward in memory, but nothing is saved. After any restart, it begins at the last committed position, or follows the reset policy if none exists, and you re-read everything since.

As often as the cost of repeating work justifies. Per batch is a solid default. Commit more often when repeats are expensive, and less often when repeats are cheap and throughput matters most.

Usually because the time between polls exceeds the allowed interval, often due to slow handlers or oversized batches, or because heartbeats are missed from long pauses in the process. Shrink batches, speed up handlers or adjust timeouts carefully.

Acknowledging is the framework’s friendly name for the same idea. When your handler acknowledges a record or batch in manual mode, the framework performs the offset commit on your behalf at the appropriate moment.

Yes, using administrative tools or by seeking within the consumer. This is a legitimate way to replay events after fixing a bug, but it should be controlled carefully because it will re-trigger all side effects of the replayed events.

In a transactional consume-transform-produce pipeline, offsets are committed as part of the transaction rather than through the ordinary commit calls, and automatic committing must be off. So you still take control of offsets, just in a different, atomic way.

An offset is simply a bookmark inside a partition, and committing is the act of saving that bookmark to the cluster so any consumer in the group can continue from it. Automatic committing saves the bookmark on a timer, unaware of what your application actually accomplished, which makes both loss and duplication possible. Manual offset commits switch that responsibility to your code, so the bookmark moves only when you have honestly finished the work before it.

Using manual commits well means understanding a handful of ideas together: commit the next offset, not the last one; process before you commit; combine fast asynchronous commits with a firm synchronous commit at shutdown and on revocation; keep processing time inside the poll interval; and design your handlers so that a repeated event is harmless. Add sensible failure paths with retries and dead letter topics, watch lag and commit health, and protect who may reset positions. Do these things, and the small gap between “finished” and “saved” becomes a managed, understood part of your system rather than a hidden source of surprises.