What Is Offset Tracking in Kafka?
A beginner-friendly guide to how Kafka remembers exactly where every reader left off, so nothing is lost and nothing has to start over from scratch.
Imagine a long audiobook series with hundreds of episodes, and a listener who commutes to work every day, always picking up exactly where the last episode ended. For this to work smoothly, something must remember, precisely, which minute of which episode the listener last heard, even if they switch devices, close the app for a week, or accidentally go back to replay a scene. In Kafka, this remembering job has a name: offset tracking. In this guide we will build up the whole picture slowly, defining every new word the first time it appears.
1The Problem Offset Tracking Solves
Before defining offset tracking precisely, it helps to see the failure it prevents.
Reading without remembering is fragile
A consumer is any program that reads events from a Kafka topic, a named stream of related events. Now imagine a consumer that reads thousands of events every minute but has no way to remember its progress. If that consumer crashed, even for a moment, it would have absolutely no idea where it had gotten to. Should it start over from the very beginning, reprocessing everything it had already handled? Should it just start from whatever is newest, silently skipping everything in between? Neither option is acceptable for any system that actually matters.
This is exactly the gap that offset tracking closes. It gives every consumer a precise, durable memory of its own progress, so restarts, crashes, and handovers between machines can all happen smoothly, without guessing.
Think of a marathon runner wearing a timing chip that records exactly which checkpoint they last passed. If they trip and fall, officials do not need to guess how far they got; the chip’s last recorded checkpoint tells them precisely. Offset tracking is that timing chip for a Kafka consumer.
2So, What Exactly Is Offset Tracking?
A precise definition, then the plain-English version.
A short definition
Offset tracking is the overall process by which Kafka records and remembers how far each consumer, or more precisely each consumer group, has progressed through a topic’s partitions. It relies on offsets, which are simple numbers marking an event’s exact position within a partition, starting at zero and increasing by one for every new event written. Offset tracking is what turns a raw position number into a durable, reusable bookmark.
In everyday terms, offset tracking is Kafka’s memory system for reading progress. It answers the question “exactly how far has this reader gotten?” reliably, even across restarts, crashes, and changes in which machine is doing the reading.
Two closely related but different things
It helps to separate two related ideas clearly. An offset is simply the number itself, identifying one specific position in a partition. Offset tracking is the broader system, of storing, updating, and retrieving those numbers on behalf of every consumer group, so progress is never lost.
Offset tracking is tracked separately for every combination of consumer group and partition. Two different groups reading the same topic keep entirely separate progress records, never affecting one another.
3Where Does Kafka Actually Store These Offsets?
A surprising and elegant answer: Kafka stores this using itself.
A special internal topic
Modern Kafka stores committed offsets inside a special, internal topic named “__consumer_offsets”, created automatically the first time it is needed. Every time a consumer group commits its progress, that commit is written as an event into this internal topic, keyed by a combination of the consumer group’s name, the topic, and the partition it refers to.
This is a clever design choice, because it means offset storage benefits from all the same durability and replication guarantees that protect ordinary Kafka data. There is no separate, external database needed just to remember reading progress.
Rather than building a brand-new filing cabinet just to store bookmarks, imagine a library that simply keeps a special shelf, using its own normal shelving system, dedicated to recording which page every regular reader was last on. The same trusted system handles both the books and the bookmarks.
flowchart LR
C[Consumer Group: analytics] -- commits offset --> T["__consumer_offsets topic"]
T -- replicated across brokers --> B1[Broker 1]
T -- replicated across brokers --> B2[Broker 2]
T -- replicated across brokers --> B3[Broker 3]
In much older versions of Kafka, offsets were instead stored inside ZooKeeper, an external coordination tool Kafka once depended on. Storing offsets as a Kafka topic instead was a major improvement, since it removed extra load on that external system and made offset commits far faster.
4Two Kinds of Offsets Worth Telling Apart
The word “offset” quietly refers to a few slightly different things depending on context.
Current Position
The offset of the next event a consumer will read, tracked temporarily in the consumer’s own memory as it processes a batch of events.
Committed Offset
The durable bookmark saved into Kafka’s internal offset storage, representing progress that has been made official and safe to resume from.
Log End Offset
The offset just past the very last event currently written to a partition, marking where the next new event will land.
Lag
The gap between the log end offset and a consumer group’s committed offset, showing how far behind that group currently is.
Why the distinction matters
A consumer’s current position can move forward constantly as it processes events, but that progress is not truly safe until it has actually been committed. If a crash happens between processing an event and committing its offset, the committed offset, the one Kafka actually remembers, will still reflect the last safe point, not the very latest processing that happened moments before the crash.
5How a Commit Actually Happens
Following one offset from processing to being safely remembered.
The consumer polls for events
A consumer actively asks Kafka for the next available events from the partitions it has been assigned.
The events are processed
The consumer’s own application logic handles each event, such as saving it to a database or triggering some business action.
An offset commit request is sent
The consumer, either automatically or through explicit code, sends a request to record its new progress.
The commit is written to the offsets topic
Kafka appends this commit as a new event into the internal “__consumer_offsets” topic, just like any other write.
The commit is confirmed
Once safely written and replicated, the commit is considered durable, and the consumer’s progress is now officially remembered.
sequenceDiagram
participant Con as Consumer
participant Br as Broker
participant OT as Offsets Topic
Con->>Br: Poll for events
Br-->>Con: Return batch of events
Con->>Con: Process events
Con->>Br: Commit offset
Br->>OT: Append commit record
OT-->>Br: Acknowledge
Br-->>Con: Commit confirmed
6Automatic Versus Manual Commits
Teams get to choose how and when offset tracking actually happens.
Automatic commits
With automatic committing enabled, the consumer’s client library periodically commits the current position in the background, on a fixed schedule, without any explicit action from the application’s own code. This is simple to set up and works well for many everyday situations where an occasional duplicate or slightly imprecise commit timing is acceptable.
Manual commits
With manual committing, the application code explicitly decides exactly when to commit, typically right after it has fully and successfully processed a batch of events. This gives much tighter control over exactly what “safely processed” means for that specific system, at the cost of a little extra code and responsibility.
| Commit style | Who decides when | Best fit |
|---|---|---|
| Automatic | The client library, on a timer | Simple pipelines where occasional duplicates are fine |
| Manual | The application’s own code | Systems needing precise control over what counts as done |
Automatic committing does not guarantee an event has actually finished being processed before its offset is committed. If a crash happens in between, that event could be skipped on restart, since Kafka only knows about the last committed offset, not what was silently in progress.
7How Offset Tracking Connects to Delivery Guarantees
The exact timing of a commit, relative to processing, decides what a consumer can honestly promise.
Committing before processing finishes
If an offset is committed before an event has actually been fully processed, a crash right afterward means that event’s processing is lost, since Kafka’s records now claim it was already handled. This pattern generally leads to at-most-once behaviour, where an event might occasionally be skipped but never processed twice.
Committing after processing finishes
If an offset is only committed after an event has been fully processed, a crash before the commit happens means that event will be read and processed again once the consumer restarts, since Kafka still believes it was never marked done. This pattern generally leads to at-least-once behaviour, where an event is never lost but might occasionally be handled twice.
Imagine crossing off a task on a to-do list. Crossing it off before you actually do it risks forgetting to do it at all if you get interrupted. Crossing it off only after finishing means you might occasionally redo something you had actually just finished, if you were interrupted right at that exact moment, but you would never simply forget it.
Most teams commit after processing, accepting the small risk of occasional duplicate processing, and design their systems to handle duplicates safely, a property called idempotence, rather than risk silently losing data.
8Resetting and Seeking: Deliberately Moving the Bookmark
Sometimes a team wants to intentionally change where a consumer group resumes from.
Why you might want to reset an offset
Occasionally, a team needs to deliberately move a consumer group’s tracked position, rather than simply continuing from wherever it happens to be. This might be needed after fixing a bug that caused certain events to be processed incorrectly, or when a brand-new consumer group wants to intentionally reprocess a topic’s full history from the very beginning.
Common reset targets
A consumer group’s offsets can typically be reset to the very earliest available offset in a partition, effectively replaying all retained history, to the very latest offset, skipping straight to only new events going forward, or to a specific point in time, letting a team say “resume as if we were reading from exactly this moment onward.”
Example: Recovering from a Bug
A team discovers that a consumer had a bug causing it to silently miscalculate order totals for the past six hours. After fixing the bug, they reset that consumer group’s offsets back to the timestamp when the bug was introduced, letting it safely reprocess and correct exactly the affected events.
Resetting offsets while a consumer group is actively running can cause confusing, inconsistent behaviour. It is generally safest to make sure the group’s consumers are stopped before deliberately resetting their tracked position.
9What Happens When There Is No Committed Offset Yet
A brand-new consumer group faces a special, one-time decision.
No history to resume from
When a consumer group connects to a partition for the very first time, there is naturally no previously committed offset for Kafka to look up. In this situation, a configured starting-point setting decides what happens: the group can start from the earliest available offset still stored in the partition, or it can start only from the latest offset, meaning only new events published from that moment onward.
This choice matters only once per partition, per group
Once a consumer group has committed even a single offset for a given partition, this starting-point setting no longer applies to that group and partition going forward. Every future restart simply resumes from the last committed offset, regardless of the original starting-point choice, since Kafka now has a real bookmark to rely on.
Think of joining a book club partway through the year. On your very first meeting, you decide whether to read every past chapter from the beginning or just start with whatever chapter the group is currently discussing. Once you have attended a single meeting, your own personal bookmark takes over from there, regardless of that initial choice.
10Offset Retention: Bookmarks Do Not Last Forever
Even committed offsets are eventually cleaned up if left unused.
Expiring old, unused commits
If a consumer group stops committing offsets entirely, perhaps because it was shut down permanently and never restarted, Kafka does not keep that group’s committed offsets forever. After a configured retention period with no new activity, the group’s old offset records are eventually removed from the internal offsets topic, much like ordinary event data eventually ages out of a regular topic.
Why this matters
If a long-dormant consumer group eventually comes back after its offsets have already expired, it will effectively be treated as brand new for that partition, falling back to the earliest-or-latest starting-point decision described earlier, rather than resuming from where it had actually left off long ago.
This retention period is a deliberate design choice, preventing Kafka from accumulating an ever-growing pile of offset records for consumer groups that have been abandoned and will never return.
11Monitoring Offset Tracking and Lag
Offset tracking is not just an internal mechanism; it is also a vital source of visibility.
Consumer Lag
The gap between the latest available offset and a group’s committed offset, showing how far behind that group currently is.
Commit Frequency
How often a consumer group is successfully committing its progress, helping reveal whether commits are happening as expected.
Commit Failures
How often offset commit attempts are failing, which can point to connectivity issues or an unstable consumer.
Stalled Groups
Consumer groups whose committed offset has not moved for an unusually long time, suggesting processing has silently stopped.
Watching offset tracking metrics is like a race organiser watching each runner’s checkpoint times. A runner whose checkpoint has not updated in a long time, or who is falling further and further behind the leaders, is worth checking on before something goes seriously wrong.
If a team can only watch one offset-related metric closely, consumer lag remains the single most valuable early warning sign of a struggling or stalled consumer group.
12Design Patterns and Anti-patterns
Good habits to copy and bad habits to avoid around offset tracking.
Helpful patterns
Commit After Processing
Committing only once processing is genuinely complete avoids silently skipping events, favouring safe reprocessing over silent loss.
Idempotent Processing
Designing processing logic that produces the same correct result even if an event is handled twice, making at-least-once delivery safe in practice.
Deliberate Offset Resets for Recovery
Using controlled, intentional offset resets as a recovery tool after bugs are fixed, rather than as a routine, everyday operation.
Habits to avoid
Problem
Relying purely on automatic commits for processing where losing even a single event is unacceptable, such as billing or payment events.
Why It’s Harmful
Automatic commits are not aware of whether processing has truly finished, creating a real risk of silently skipping events after a crash at the wrong moment.
Correct Approach
Use manual commits, timed deliberately after processing is confirmed complete, for any data where correctness genuinely matters.
Problem
Resetting a consumer group’s offsets casually or frequently, without a clear, deliberate reason.
Why It’s Harmful
Careless resets can cause large amounts of unnecessary reprocessing, duplicate side effects, or accidentally skipped data if done incorrectly.
Correct Approach
Treat offset resets as a deliberate, carefully planned recovery action, done with the consumers stopped and a clear target offset in mind.
13Best Practices and Common Mistakes
Practical lessons for using offset tracking well.
Best practices
Choose commit timing based on real risk tolerance
Match your commit strategy to how costly a skipped or duplicated event would actually be for your specific system.
Design for safe reprocessing
Build processing logic that behaves correctly even if the same event is handled more than once.
Monitor lag and commit health continuously
Treat offset-related metrics as core operational signals, not an afterthought only checked when something breaks.
Plan offset resets carefully
Stop the relevant consumers, choose a precise target offset or timestamp, and document why the reset was needed.
Be aware of offset retention limits
Understand that a long-dormant consumer group’s offsets can eventually expire, effectively resetting it to a fresh starting-point decision.
Mistakes beginners often make
Assuming that processing an event automatically means it is safely remembered ignores the fact that only a committed offset is truly durable and recoverable after a crash.
A consumer that appears to be running fine might actually be failing to commit offsets due to connectivity issues, silently building up risk without any obvious symptoms until a crash occurs.
Assuming that fixing or resetting one consumer group’s offsets affects any other group reading the same topic can lead to confusion, since every group’s tracked progress is entirely independent.
14Frequently Asked Questions
In modern Kafka, committed offsets are stored inside a special internal topic named “__consumer_offsets”, benefiting from the same durability and replication as any other Kafka data.
No. Every consumer group tracks its own committed offsets completely independently, even when several different groups are reading from the exact same topic.
If a commit attempt fails, the consumer’s committed offset simply stays at whatever it was previously. Depending on the client’s configuration, the commit may be retried, but until it succeeds, the group’s official progress has not moved forward.
Yes. Kafka provides tools for resetting a consumer group’s offsets to a specific point, whether that is the earliest offset, the latest offset, or a particular timestamp, typically done while the group’s consumers are stopped.
If a consumer group stops committing entirely for longer than a configured retention period, its old offset records are eventually cleaned up, since Kafka assumes a group that has been silent that long may never return.
Committing offsets does add a small amount of overhead, since it involves writing to the internal offsets topic, but this cost is generally small and well worth the reliability it provides.
Not necessarily. The current position can move forward continuously as a consumer processes events, but it only becomes a durable, recoverable committed offset once an actual commit has been made and confirmed.
15Summary and Key Takeaways
Offset tracking is the system that gives every Kafka consumer group a reliable memory of exactly how far it has progressed through a topic’s partitions. Behind the scenes, Kafka stores these committed offsets durably inside its own special internal topic, benefiting from the same reliability that protects ordinary event data. The precise timing of when a consumer commits its progress, whether automatic or manual, directly shapes whether that system behaves as at-most-once or at-least-once, and thoughtful choices here matter enormously for correctness. Tools for resetting and monitoring offsets give teams the control and visibility they need to recover from mistakes and catch struggling consumers early. Understanding offset tracking deeply is essential for building Kafka-based systems that behave predictably, even when machines crash or restart.
Key Takeaways
- Offset tracking is Kafka’s memory of reading progress — recorded separately for every consumer group and partition.
- Committed offsets live in Kafka’s own internal topic — gaining the same durability as ordinary event data.
- Current position and committed offset are different things — only a committed offset is truly safe and recoverable.
- Commit timing shapes delivery guarantees — committing before or after processing decides between at-most-once and at-least-once behaviour.
- Offsets can be deliberately reset or replayed — a powerful recovery tool when used carefully and intentionally.
- Old, unused offsets eventually expire — so long-dormant consumer groups may lose their exact resume point.
- Monitoring lag and commit health matters — catching a struggling consumer early before it falls dangerously behind.