What Is auto.offset.reset in Kafka?
A chapter-by-chapter guide to the Kafka consumer setting that decides where reading begins when no bookmark exists, and why choosing wrongly can silently skip or replay millions of messages.
Imagine you have just joined a streaming service that has a very long-running show with thousands of episodes. On your first evening the app asks a simple question: do you want to start from episode one, or just watch whatever is airing right now? Apache Kafka faces exactly this question every time a new reader shows up, and the answer it uses is a small consumer setting called auto.offset.reset. It looks like one harmless line in a configuration file, yet it decides whether your application processes a full history, only fresh events, or refuses to start until a human makes a decision.
In this guide you will learn what the setting is, when it is used, what each of its values does, and how it behaves inside the consumer. We will also look at the surprising situations where it fires without warning, how it interacts with retention and committed offsets, and how real teams choose a value for production. No prior Kafka knowledge is assumed, and every new word is explained the first time it appears.
1Why This Setting Exists at All
To understand a fallback rule, you first need to understand what it falls back from.
A quick picture of Kafka
Apache Kafka is a system for moving streams of events between applications. An event is a small record that says something happened, such as “order 4471 was placed” or “sensor 12 reported 31 degrees”. Programs called producers write these events into Kafka, and programs called consumers read them out. Kafka stores the events on disk in an ordered sequence, so a consumer can read them now, or a few minutes later, or even days later.
The key difference from an ordinary message queue is that reading an event does not remove it. The event stays in Kafka until a retention rule deletes it, which by default happens after about seven days. Because events stay, many different applications can read the same stream independently, each at its own speed, without disturbing each other.
Think of Kafka as a very long roll of paper tape on which a machine keeps printing new lines at the end. Anyone can walk up and read the tape, but nobody tears off the part they have read. Each reader simply keeps a finger on the line where they stopped. If a reader has never read this tape before, someone has to tell them where to put their finger for the first time. That instruction is what auto.offset.reset provides.
The question every new reader raises
When a consumer starts, it needs to know which event to read first. If it has read this stream before, the answer is easy: continue right after the last event it finished. Kafka keeps that last position on the consumer’s behalf. But what if the consumer has never read before, or the saved position points to an event that no longer exists? Kafka cannot guess. It needs a rule, and the rule is chosen by you through auto.offset.reset.
Words you will see throughout this guide
Topic
A named stream of events, like “orders” or “payments”. Think of it as a labelled channel.
Partition
A topic is split into partitions, and each partition is its own ordered log of events.
Offset
The sequence number of an event inside one partition, starting at zero and only ever growing.
Producer
An application that writes events into a topic.
Consumer
An application that reads events from a topic and does something useful with them.
Consumer Group
A team of consumers that share the work of reading one topic, identified by a shared group name.
Broker
A Kafka server that stores partitions and answers requests from producers and consumers.
Committed Offset
The saved bookmark that says how far a consumer group has progressed in a partition.
The setting name is a mouthful, but it breaks into three easy pieces: “auto” because Kafka applies it automatically, “offset” because it is about a position, and “reset” because the position is being set fresh instead of resumed.
2The Kafka Log and Its Offsets
Every value of the setting is really a choice between two ends of the same log.
A partition is an append-only log
Inside a partition, Kafka only ever adds new events at the end. It never inserts in the middle and never edits an existing event. Each event receives the next offset number, so the first event is offset 0, the second is offset 1, and so on. These numbers are permanent labels. Even after old events are deleted, the remaining events keep their original numbers, and a deleted number is never reused for a different event.
Offsets belong to a single partition. Partition 0 and partition 1 of the same topic each have their own offset 0. This matters later, because a consumer that reads a topic with several partitions keeps a separate position for every one of them, and the reset rule is applied to each partition separately.
A partition is like a bank passbook that only ever gets new lines printed at the bottom. Line numbers keep counting upward forever. If the bank shreds the first hundred lines to save paper, the next line is still numbered 101, not 1. The numbers never restart.
The important markers on a log
Log Start Offset
The number of the oldest event still stored. It moves forward as retention deletes old data.
Log End Offset
The number that the next new event will receive. It moves forward every time a producer writes.
High Watermark
The last offset that has been safely copied to enough replicas. Ordinary consumers can only read up to here.
Segment
A partition is stored as a series of files called segments. Retention removes whole segments at a time.
flowchart LR
A["Log start offset - oldest event kept"] --> B["Events waiting to be read"]
B --> C["High watermark - last safely copied event"]
C --> D["Log end offset - where the next event lands"]
Now the meaning of the two most popular reset values becomes clear. The value earliest means “put my finger on the log start offset”, and the value latest means “put my finger at the end, where new events will appear”. Everything else in this guide is a consequence of that simple pair.
When a consumer is told to jump to the end, it lands at the high watermark, not at some unreplicated tail. This protects readers from seeing events that could still be lost if a broker fails. A consumer configured to read only committed transactional data lands at an even more careful marker called the last stable offset.
3Bookmarks: How Kafka Remembers Progress
The reset setting only matters when the bookmark is missing, so let us see how bookmarks work.
What a committed offset is
As a consumer group processes events, it periodically tells Kafka “we have finished everything before offset N in this partition”. That message is called committing an offset, and the saved number is the committed offset. It always points to the next event the group should read, which is one more than the last event it finished. If the group stops and restarts, it asks Kafka for this number and carries on from there without repeating or skipping anything.
Committed offsets are stored per group, per topic and per partition. Two different groups reading the same topic each have their own independent bookmarks. That is why a billing service and an analytics service can read the same “orders” topic at completely different speeds.
Picture a shared library book with a separate bookmark for each family member. Your bookmark sits at page 120, your sister’s sits at page 45, and neither of you moves the other’s. The group name is the family member’s name, and the committed offset is the page number.
Where the bookmarks live
Kafka stores committed offsets in a special internal topic named __consumer_offsets. It is an ordinary Kafka topic, split into 50 partitions by default and copied to several brokers for safety. Each record in it says, in effect, “group G, topic T, partition P is at offset N”. Because only the latest bookmark per group and partition matters, the topic is compacted, which means old bookmarks are cleaned up and only the most recent one is kept.
For each consumer group, one broker plays the role of the group coordinator. It is chosen by hashing the group name onto one of those internal partitions, and whichever broker leads that partition becomes the coordinator. The coordinator tracks which consumers are in the group, hands out partitions, and answers questions about committed offsets.
Automatic versus manual commits
Automatic commit
The default. The consumer saves its position in the background every five seconds while it keeps polling. It is simple, but the save can happen before your code has finished processing, which risks losing work if the application crashes at the wrong moment.
Manual commit after processing
Your code decides when to save, usually right after a batch is fully handled. This gives stronger guarantees that nothing is skipped, at the cost of some extra care and the possibility of reprocessing a few events after a crash.
External storage
Some applications store their own positions in a database alongside their results, so the result and the position are saved together. In that design Kafka’s bookmark is ignored and the application seeks to its own saved position at startup.
The committed offset is the next event to read, not the last event read. When someone says “the group is at offset 500”, they mean events up to 499 are done and 500 is next. Mixing this up leads to off-by-one mistakes when comparing positions and calculating lag.
4What auto.offset.reset Really Means
It is a fallback plan for a missing or broken bookmark, nothing more and nothing less.
The plain definition
auto.offset.reset is a configuration option of the Kafka consumer. It tells the consumer what to do when it needs a starting position for a partition but Kafka has no valid saved position to offer. In that moment the consumer consults this setting and picks the start of the log, the end of the log, or gives up with an error, depending on the value you chose.
The setting is read by the consumer client running inside your application, not by the brokers. That means it is chosen by the team that owns the consuming application, and two applications reading the same topic can safely use different values. It is a client-side decision about how to begin, not a property of the topic.
When the setting is used, and when it is ignored
The setting is used in exactly two situations. First, when there is no initial offset for the group and partition, which is the normal case for a brand-new consumer group or a brand-new partition. Second, when the saved offset no longer exists on the broker, for example because retention deleted those events or because the log was truncated. In any other situation, the consumer simply resumes from the committed offset and the setting has no effect at all.
This is the single most misunderstood fact about the setting. Changing the value from latest to earliest does nothing for a group that already has a valid committed offset. The group will continue from its bookmark exactly as before, and people who expected a replay are left wondering why nothing happened.
Many beginners believe that setting earliest makes a consumer always read from the beginning. It does not. It only decides where a group starts when it has no valid bookmark. After the first successful commit, the bookmark takes over.
Think of a spare key hidden under a flowerpot. You only reach for it when your own key is missing or does not fit the lock. If your own key works, the spare stays where it is. The reset setting is the spare key of Kafka consumers.
5The Values: earliest, latest and none
Three classic choices, plus a newer time-based option, each with a different personality.
earliest
Start from the oldest event still stored in the partition. Nothing that still exists is skipped, so you get the full available history.
latest
Start from the end of the log and see only events written after that moment. This is the default value.
none
Do not choose for me. If no valid bookmark exists, throw an exception and let the application or operator decide.
Duration based
Newer Kafka releases, starting with version 4.0, add a way to say “go back by this much time from now”. Check your client version’s documentation before relying on it.
Everyday analogies for each value
earliest is “start the series from episode one”
You are willing to watch everything the service still has. You will see the whole backstory, but it will take a while to catch up, and you will process old events that happened long before you joined.
latest is “just show me what is airing now”
You do not care about the past. You will see only what happens from this moment forward, which is quick to start, but anything that aired before you tuned in is gone from your point of view.
none is “do not seat me until you find my reservation”
You would rather be turned away at the door than be seated at a random table. The consumer refuses to guess and raises an error, which forces a human or a piece of code to pick a position deliberately.
Side by side
| Value | Where reading begins | Risk | Best fit |
|---|---|---|---|
| earliest | Oldest stored event | Large backlog, repeated processing | Pipelines that must not miss data |
| latest | End of the log | Silently skipping events | Live views where old data is worthless |
| none | Nowhere, error is raised | Application stops until handled | Strict systems that need a human decision |
The value must be written exactly as the client expects. A typo such as “earlist” does not fall back to a default. The consumer refuses to start and reports a configuration error, which is helpful because it fails loudly instead of quietly choosing wrongly.
6Two Situations That Trigger the Reset
The reset never fires randomly. It appears only in these two families of situations.
Situation one: there is no initial offset
This is the everyday case. A group name that Kafka has never seen before starts reading a topic, so there is no bookmark to fetch. It also applies when an existing group begins reading a topic it never subscribed to before, and to partitions that were added later to a topic. Each partition without a bookmark triggers the rule independently.
A brand-new application goes live
A new microservice called “loyalty-points” starts reading the “orders” topic with a fresh group name. With earliest it will process every stored order. With latest it will only see orders placed after it starts.
A developer tries things from the command line
The console consumer tool that ships with Kafka creates a random throwaway group name on each run, so every run is a “new group”. That is why it prints nothing old by default, and why adding its from-beginning flag, which simply sets the reset value to earliest, suddenly shows history.
A partition is added to a running topic
An operator raises the partition count from 6 to 8. Existing consumers discover the two new partitions after their next metadata refresh and have no bookmark for them, so the reset rule decides where they begin reading those partitions.
Situation two: the saved offset is out of range
Here a bookmark exists, but it points to a position the broker can no longer serve. Kafka answers the consumer’s fetch with an “offset out of range” error, and the consumer then falls back to the reset rule for that partition. There are several ways to land here.
Retention deleted the events
The consumer was down for longer than the retention period. Its bookmark points at offset 501, but the oldest stored event is now 900. Offset 501 is below the log start, so the position is invalid.
The bookmark itself expired
Kafka forgets the committed offsets of a group that has been completely empty for longer than the offsets retention period, which is seven days by default. When the group returns, it looks like a brand-new group and the first situation applies.
The topic was recreated or restored
If a topic is deleted and recreated, or a cluster is rebuilt from a backup, the new log may be shorter than the saved bookmark suggests. A bookmark beyond the end of the log is also out of range.
The log was truncated after a leader change
In rare failure cases, especially when an out-of-sync replica is allowed to become leader, the new leader may have fewer events than the old one. A consumer that was ahead of the new log end finds its position no longer valid.
The second situation is the dangerous one, because nobody changes anything. A weekend outage, a paused job or a forgotten test environment can quietly push the consumer into a reset the next time it runs. Whatever value you chose months ago now decides what happens to the data it missed.
7Inside the Consumer: How the Reset Really Works
Follow one consumer from startup to its first record and you will see exactly where the setting is consulted.
The consumer subscribes and polls
The application names the topics it wants and calls poll. Nothing is read yet. The first poll starts the machinery that finds the group coordinator and joins the group.
The group assigns partitions
The coordinator shares out the topic’s partitions among the group’s members. Each consumer now knows which partitions it is responsible for.
The consumer asks for committed offsets
For its assigned partitions, the consumer sends an offset-fetch request to the coordinator. The answer for each partition is either a number or “nothing saved”.
Partitions with a bookmark get a position
Where a committed offset was returned, the consumer records it as its current position. The reset setting is not touched for these partitions, at least not yet.
Partitions without a bookmark are marked for reset
For the rest, the consumer notes that a reset is needed and remembers the configured strategy. With none, this is the moment the consumer raises an error to the application instead.
The consumer asks the partition leader for a number
To resolve earliest or latest into an actual offset, the consumer asks the broker that leads the partition for its oldest or newest offset. Internally these requests use special marker timestamps that mean “the beginning” and “the end”.
Fetching begins
With a concrete position for every partition, the consumer starts fetching records. Records flow back, your code processes them, and commits gradually replace the reset decision with a normal bookmark.
sequenceDiagram
participant C as Consumer
participant G as Group Coordinator
participant L as Partition Leader
C->>G: Join the group
G-->>C: Here are your partitions
C->>G: Ask for committed offsets
G-->>C: Offset found or nothing saved
C->>L: If nothing saved, ask for earliest or latest number
L-->>C: Here is the position number
C->>L: Fetch records from that position
L-->>C: Batch of records
The second path: a fetch that fails
There is also a runtime path. Suppose the consumer had a valid position, but while it was away the broker deleted older data. When it sends a fetch for that position, the broker replies that the offset is out of range. The consumer then handles this exactly like a missing bookmark: it applies the configured strategy to that one partition, resolves a new position from the leader, and continues. With none, an out-of-range exception is thrown to the application instead.
The reset is resolved lazily, meaning when the consumer first needs the position during polling, not at the instant your code subscribes. The word “latest” therefore means the end of the log at that later moment, which has real consequences you will meet in Chapter 10.
8The Lifecycle of a Consumer Start, End to End
One decision tree captures every possible starting outcome for a single partition.
flowchart TD
A["Consumer receives a partition"] --> B{"Committed offset exists?"}
B -- "Yes" --> C{"Is it still inside the log?"}
B -- "No" --> D{"Value of auto.offset.reset"}
C -- "Yes" --> E["Resume from committed offset"]
C -- "No" --> D
D -- "earliest" --> F["Start at log start offset"]
D -- "latest" --> G["Start at the end of the log"]
D -- "none" --> H["Raise an exception"]
Reading the diagram in words
Every assigned partition asks the same first question: is there a committed offset? If yes, and the position is still inside the stored log, the consumer resumes there and the setting is irrelevant. In every other case the setting becomes the tiebreaker. Because the decision is made per partition, one consumer can resume five of its partitions from bookmarks while resetting a sixth.
A full outcome table
| Situation | earliest | latest | none |
|---|---|---|---|
| Valid committed offset | Resumes from bookmark | Resumes from bookmark | Resumes from bookmark |
| New group, no bookmark | Reads full stored history | Reads only new events | Exception at poll |
| Bookmark older than log start | Jumps to oldest stored event | Jumps to the end, skipping the gap | Out-of-range exception |
| Bookmark beyond log end | Jumps to oldest stored event | Jumps to the end | Out-of-range exception |
| New partition discovered | Reads it from the start | Reads it from the point of discovery | Exception at poll |
What happens after the first commit
Once the consumer processes records and commits, a real bookmark now exists in __consumer_offsets. From that moment the group is a normal, resuming group, and the reset setting goes back to sleep. This explains a puzzle that surprises many teams: a setting that mattered enormously on day one seems to do nothing on day two.
A new employee is told on the first morning whether to start with the old training archive or only with this week’s tasks. Once they have finished and signed off their first tasks, their manager just says “carry on where you left off”. The morning instruction no longer applies unless their records are lost.
9earliest in Depth
The completeness-first choice, and the price of being complete.
What earliest gives you
With earliest, a group with no valid bookmark begins at the oldest event still stored in each partition. Nothing that exists is skipped. This is the natural choice for systems whose correctness depends on seeing every event, such as data loading jobs, audit trails, billing calculations and search index builders. It also lets you onboard a new consumer application and have it catch up on history without any special tooling.
It is important to remember that “earliest” means the earliest event that is still available, not the earliest event ever written. If the topic retains seven days of data, a new group starts with those seven days. Anything older is already gone, and no setting can bring it back.
Advantages
- No stored events are skipped for a new group.
- New applications can build up their state from history automatically.
- Recovery after long outages loses nothing that retention still holds.
- Works well with compacted topics to rebuild the latest value for every key.
- Makes testing repeatable because the same data is read each time with a fresh group.
Disadvantages / Trade-offs
- A large topic can create a huge backlog that takes hours to process.
- Downstream systems may be flooded with old events they did not expect.
- Consumers must tolerate duplicates, because events may be handled twice after resets.
- Old events may trigger stale side effects such as emails or alerts for past activity.
Where earliest quietly bites
Consider a team that deploys a consumer with a fresh group name on every release, perhaps by adding a version number or a random suffix. With earliest, every deployment replays the entire topic. The application appears to work, but each release causes a burst of duplicate emails, repeated database writes and a sharp spike in load. The remedy is to keep the group name stable and treat it as the permanent identity of the application.
Choosing earliest is like reading a newspaper archive from its very first issue whenever you subscribe. You will never miss a story, but you should plan for a long evening and not be surprised by news that is years old.
Before deploying with earliest on a big topic, estimate the catch-up time. Divide the number of stored events by the rate at which your application can process them. If the answer is many hours, plan for it, and consider whether downstream systems can handle the burst.
10latest in Depth
The default value, the quickest start, and the source of the most confusing missing-data bugs.
What latest gives you
With latest, a group with no valid bookmark begins at the end of each partition and sees only events written afterwards. Start-up is instant because there is no backlog to work through. This suits systems that show or react to the current state of the world, such as live dashboards, real-time monitoring, and alerting on fresh signals, where handling stale events would be pointless or even harmful.
It is also the default. If you never set the option, you get latest, which means many production applications are running with this behaviour simply because nobody made a decision.
Advantages
- Very fast start with no backlog to process.
- Avoids reacting to stale events for real-time use cases.
- Lower risk of flooding downstream systems on first start.
- A sensible choice for disposable consumers used only for inspection.
Disadvantages / Trade-offs
- Events written before the consumer’s position is resolved are never seen by that group.
- After a long outage, the gap of missed events is silently skipped.
- New partitions can lose their first events before consumers discover them.
- The missing data leaves no error and is easy to overlook for weeks.
Gotcha one: the start-up race
Because the reset is resolved lazily, there is a gap between “the code subscribed” and “the consumer knows its position”. If a producer sends events during that gap, a latest consumer will start after them and never see them. This is the classic reason an automated test passes on a developer’s laptop when run slowly but fails randomly in a build pipeline. The producer wrote first, the consumer joined second, and the events fell into the gap.
The fix is to make the ordering explicit. Wait until the consumer has actually been assigned its partitions before producing, or use earliest in test environments so that timing stops mattering.
Gotcha two: partitions added later
When a topic gains new partitions, running consumers only learn about them at their next metadata refresh, which by default happens every five minutes. With latest, they begin at the end of the new partition at the moment of discovery. Any events producers wrote in the meantime are missed by that group. Teams that add partitions to live topics and care about completeness commonly choose earliest for exactly this reason.
Gotcha three: long downtime
If a consumer group is stopped for longer than the offsets retention period, its bookmarks vanish. When it restarts it behaves like a new group, and with latest it jumps straight to the end. Everything that arrived during the outage is skipped, with no warning beyond a lag graph that suddenly looks healthy.
Leaving the value unset is a choice to accept latest. For any consumer whose output matters, write the value explicitly in configuration so that the intent is visible to every reader of the code.
Choosing latest is like walking into a live radio broadcast midway. You hear whatever is being said from now on, and you have no way of knowing what was said in the minutes before you turned the dial.
11none in Depth
The cautious choice: refuse to guess, and make the risk visible.
What none gives you
With none, the consumer never invents a starting position. If a partition has no committed offset, polling raises an exception that says no offset was found for that partition. If a committed offset is out of range, polling raises a different exception describing the invalid position. In both cases the application is forced to handle the situation on purpose.
This is valuable when a wrong guess is more expensive than a stopped service. A payment reconciliation job, for example, should never quietly skip a day of transactions, nor should it silently reprocess a month of them. Failing loudly turns an invisible data problem into a visible operational event.
The consumer raises an exception
Polling fails for the partitions with no usable position. The exception names the partitions involved, which tells you exactly where the gap is.
The application or operator decides
Someone chooses the correct starting point based on business rules: the beginning, a specific timestamp, or a saved position from an external record.
The position is set explicitly
The application uses the client’s seek operations, or an operator uses the group management tool to set offsets while the group is stopped.
Normal consumption resumes
After the first commit, the group has a healthy bookmark again and behaves like any other.
Advantages
- No silent data skipping and no silent full replay.
- Forces explicit ownership of the starting position.
- Turns retention and expiry surprises into alerts.
- Fits regulated domains that need every start to be justified.
Disadvantages / Trade-offs
- The first ever run always fails unless offsets are prepared in advance.
- Needs extra code or runbooks to handle the exception properly.
- An unattended outage can turn into a longer stoppage awaiting a human.
- Not suitable for casual tools and quick experiments.
Teams using none often run a one-time bootstrap step that creates the group’s first bookmark, for example by setting offsets to a chosen timestamp, before the real application ever starts. After that, restarts are uneventful.
Choosing none is like an airline gate agent who will not let you board without a boarding pass. It is slightly inconvenient when you forget yours, but nobody ends up on the wrong plane.
12Retention, Expiry and Out-of-Range Surprises
Two different clocks decide when a bookmark stops being usable.
Clock one: how long events are kept
Kafka deletes old events according to retention rules on each topic, by default after seven days, and optionally by total size. Deletion happens by whole segment files, so the log start offset jumps forward in steps rather than one event at a time. Once events are deleted, any bookmark pointing into the deleted range becomes invalid. Nothing tells the consumer while this is happening. It finds out only when it next asks for that position.
Clock two: how long bookmarks are kept
Bookmarks have their own lifetime, set on the broker by the offsets retention setting, which defaults to seven days. The clock starts only when a group has no active members. While at least one consumer in the group is alive, its bookmarks are safe. Once the group is empty for longer than the limit, Kafka removes its offsets, and the next start looks like a first start.
flowchart LR
A["Day 0 - consumer stops after finishing offset 500"] --> B["Day 3 - retention deletes offsets 0 to 800"]
B --> C["Day 4 - consumer restarts and asks for 501"]
C --> D["Broker replies that 501 is out of range"]
D --> E["Reset policy decides the new position"]
Comparing the two clocks
| Clock | What it protects | Typical default | What happens when it runs out |
|---|---|---|---|
| Event retention | The stored events themselves | 7 days | Old offsets become out of range |
| Offsets retention | The saved bookmark of an empty group | 7 days | Group looks brand new |
Problem
A team sets topic retention to two days to save disk space, but their consumer is sometimes offline for long weekends and holidays.
Why It’s Harmful
Every long outage pushes the consumer past the retention limit. The bookmark falls out of range and the reset rule takes over. With latest, the gap is silently lost. With earliest, the group replays whatever remains, which may be a partial and confusing slice of history.
Correct Approach
Set retention comfortably longer than the longest outage plus the time needed to catch up. Alert on consumer lag well before it approaches the retention window, and choose the reset value knowing what an out-of-range event will do.
A useful rule is that the maximum tolerable downtime must be smaller than the shorter of the two clocks. If your consumers can be offline for ten days, neither seven-day default is safe without changes.
13How to Replay or Skip Data on Purpose
Since the setting is only a fallback, deliberate repositioning needs other tools.
Sooner or later every team needs to re-read history after fixing a bug, or jump ahead to ignore a poisoned backlog. Because auto.offset.reset only applies when there is no valid bookmark, you need a way to change or remove the bookmark itself. Kafka offers several.
Use a new group name
A fresh name has no bookmark, so the reset rule applies. Simple, but it leaves the old group’s data behind and changes the application’s identity.
Reset with the group tool
Kafka ships a command-line tool for consumer groups with a reset-offsets option. It can move a group to the beginning, the end, a specific offset, a timestamp, or a shift by a number of events. The group must have no active members while it runs.
Seek inside the application
The client offers operations to jump to the beginning, to the end, or to a specific offset for chosen partitions. Another operation translates a timestamp into an offset so you can replay “everything since 9 a.m.”.
Assign partitions manually
Instead of joining a group, the application says exactly which partitions it wants and manages positions itself. This gives full control but removes automatic sharing of work.
flowchart TD
A["Need to change where a group reads?"] --> B{"Is the group still running?"}
B -- "Yes" --> C["Stop it, or seek from inside the application"]
B -- "No" --> D["Use the reset offsets option of the group tool"]
A --> E["Or start with a fresh group name"]
A safe replay routine
Stop every member of the group
Offset changes on an active group are rejected or overwritten by the running consumers, so make sure the group is empty.
Preview the change first
The group tool can show what it would do without applying it. Read the planned positions and check they match your intention.
Apply the reset
Run the reset for the chosen topic and target position, then confirm the new bookmarks with a group description.
Restart and watch
Start the consumers and watch lag and downstream effects closely. A replay creates duplicates by nature, so confirm that the application is safe to run on repeated events.
Replaying events triggers real side effects again unless the consumer is idempotent, which means handling the same event twice gives the same final result. Emails, payments and notifications need explicit protection before any replay.
14Related Settings That Work With It
The reset value is only one of several dials that together decide where and when reading happens.
| Setting | Where | What it does | Link to the reset rule |
|---|---|---|---|
| group.id | Consumer | The name that identifies the consumer group and its bookmarks | A new name means no bookmark, so the reset rule applies |
| enable.auto.commit | Consumer | Whether bookmarks are saved automatically in the background | Without any commits, every restart falls back to the reset rule |
| auto.commit.interval.ms | Consumer | How often automatic commits happen, five seconds by default | Shorter intervals reduce repeated work after a crash |
| group.instance.id | Consumer | Gives a member a stable identity to avoid needless rebalances | Helps restarts keep partitions, but bookmarks still expire as usual |
| isolation.level | Consumer | Whether uncommitted transactional data is visible | Changes where latest lands, at the last stable offset |
| metadata.max.age.ms | Consumer | How often topic layout is refreshed, five minutes by default | Controls how soon new partitions are discovered |
| max.poll.interval.ms | Consumer | Longest allowed gap between polls | Exceeding it removes the consumer and can cause rebalance churn |
| retention.ms | Topic | How long events are kept, seven days by default | Decides when bookmarks become out of range |
| offsets.retention.minutes | Broker | How long an empty group’s bookmarks survive, seven days by default | After this, the group looks brand new |
Auto commit and the reset rule together
A subtle trap is disabling automatic commits and forgetting to commit manually. The application runs fine, records flow, and nothing looks wrong. But no bookmark is ever written, so each restart is treated as a first start. With earliest, the application reprocesses everything on every restart. With latest, it skips everything that arrived while it was down. Both are painful, and the root cause is nowhere near the reset setting.
Group identity is the anchor
Everything in this chapter hangs on the group name. It is the key under which bookmarks are stored, so renaming it, or generating it randomly, throws away all progress. Treat the group name like a database name: choose it carefully, keep it stable, and change it only when you deliberately want a fresh start.
Keep the group name, the commit strategy and the reset value together in one reviewed configuration file. When they are edited in different places, contradictions creep in that only appear during an outage.
15Frameworks and Ecosystem Behaviour
Many tools sit on top of the Kafka consumer, and each has its own habits around starting positions.
The tools below build on the same idea but sometimes expose it under a different name or override the default. Always confirm the exact behaviour in the documentation of the version you run, because defaults can change between releases.
Kafka Streams
Streams applications use their application identifier as the group name. Streams overrides the consumer default and starts from the earliest data for a new application, since stream processing usually needs full history. It also offers a dedicated application reset tool that clears bookmarks and internal state together, and lets individual input sources choose their own starting behaviour.
Kafka Connect
Sink connectors read from Kafka and write to other systems. Each connector uses a group named after it, and connectors typically begin from the earliest data so that no events are missed in the destination. Individual connectors can override consumer settings when a different start is needed.
Spring for Apache Kafka
Spring Boot exposes the same option through its configuration properties under the consumer section. It does not change the meaning, so the default remains latest unless you set it.
ksqlDB
Queries in ksqlDB start from the latest data by default, which surprises many newcomers who run a query and see no rows. Setting the reset property to earliest for the session makes the query process history as well.
Spark Structured Streaming and Apache Flink
These engines keep their own progress inside their checkpoints and have their own starting-position options, so they do not simply follow the consumer’s reset setting. In Spark you choose the starting offsets through a source option, and in Flink you choose an offsets initializer for the Kafka source. Once a job has a checkpoint, it restores from that checkpoint rather than from the Kafka bookmark.
Non-Java clients
Clients for Python, Go, .NET and other languages usually wrap a shared native library that follows the same model. The setting is present under the same name, and some versions accept alternative words such as “smallest” and “largest” for the same two positions.
Never assume that a framework’s default matches the plain consumer’s default. An application that moves from a hand-written consumer to a framework may silently change from “skip old data” to “replay all data” or the reverse. State the value explicitly when you migrate.
16Security, Monitoring and Metrics
Resets change what data an application sees, so they deserve the same care as any access decision.
Security considerations
The reset rule itself runs inside the client, but the actions around it involve permissions. Reading a topic requires read permission on that topic and on the consumer group. Repositioning a group with the management tool requires permission on both as well. If your cluster uses access control lists, review who can modify group offsets, because moving a group backwards can re-trigger business actions, and moving it forwards can hide data from a compliance process.
A consumer set to earliest can read every stored event the moment its group and topic permissions allow. That is usually intended, but it means a newly granted permission also grants access to the whole retained history, including data from before the team was onboarded. Confirm that this matches your data-handling rules before enabling access.
What to monitor
Consumer lag
The gap between the log end offset and the group’s position. Sudden drops to near zero without matching processing can signal a jump to latest.
Lag spikes
A sudden jump upward after a restart can signal a jump to earliest and a huge new backlog.
Commit rate and age
If commits stop, bookmarks go stale and the next restart falls back to the reset rule.
Group state and members
A group that is empty for many days is approaching bookmark expiry and needs attention.
Out-of-range errors in logs
Log messages about resetting an offset name the partition and the new position. Search for them after outages.
Exceptions with none
Alert immediately when the no-offset or out-of-range exceptions appear, since they mean a human decision is needed.
Logging and audit
The consumer writes an informational log line whenever it resets a position, naming the partition and the chosen offset. Make sure your log collection keeps these lines, and consider raising an alert on them in production. An operator-driven reset should also be recorded with who did it, why, and which positions changed, so that a later data investigation can reconstruct what was replayed or skipped.
Alert on time-based lag, meaning how many minutes behind the group is, and compare it to the retention period. An alert that fires at half the retention time gives your team a fair chance to fix problems before bookmarks go out of range.
17Design Patterns and Anti-patterns
Proven ways to live with the reset rule, and habits that repeatedly cause incidents.
Patterns worth adopting
Idempotent consumer
Design processing so that seeing an event twice changes nothing. This makes earliest, replays and crash recovery safe.
Earliest plus de-duplication
Choose earliest to avoid gaps and protect against repeats by storing processed event identifiers or using upserts keyed by business identity.
None plus alert plus runbook
Fail loudly, alert the owning team, and follow a written procedure for choosing the correct position.
Dedicated replay group
Run a separate group with its own name and earliest, so that reprocessing never disturbs the production group’s bookmarks.
Commit after processing
Save bookmarks only after the work is complete, so a crash causes repeats rather than losses.
Explicit configuration
Write the reset value in configuration for every consumer, with a comment on why it was chosen.
Anti-patterns to avoid
Problem
The team leaves the value unset in production and assumes Kafka will “read everything” or “read only new data” without checking which it actually does.
Why It’s Harmful
The default is latest. A new group or an expired bookmark quietly skips data, and the missing events are noticed only when a report or customer complains.
Correct Approach
Set the value explicitly for every consumer, and document the reasoning next to it so that reviewers can challenge it.
Problem
A developer changes the value from latest to earliest in production to force a replay, and redeploys expecting old data to reappear.
Why It’s Harmful
The existing bookmark is valid, so nothing changes. The developer then renames the group in frustration, which makes the application replay the entire topic and re-triggers side effects across many systems.
Correct Approach
Reposition the existing group deliberately with the group tool or seek operations, after confirming the consumer is idempotent and downstream systems can cope.
Problem
Every deployment generates a random group name so that “we always start clean”.
Why It’s Harmful
Progress is discarded on each release. With earliest the whole topic replays each time, and with latest the events between the old and new instances are lost. It also fills the cluster with abandoned groups.
Correct Approach
Keep one stable group name per application and let bookmarks carry the state across releases.
18Best Practices and Common Mistakes
A practical checklist you can apply the next time you configure a consumer.
How to choose a value
Ask what a missed event costs
If missing events causes financial, legal or customer harm, lean toward earliest or none. If old events are worthless, latest may be right.
Ask what a repeated event costs
If reprocessing is harmful and cannot be made safe, earliest is dangerous and none with careful handling is safer.
Estimate the catch-up burden
Compare topic size with processing speed. If a full replay would take longer than is acceptable, plan for it or choose differently.
Check the clocks
Make sure retention and offsets retention are longer than your worst realistic outage.
Write it down
Record the chosen value and the reason in configuration and in the service’s documentation.
Quick guide by scenario
| Scenario | Suggested value | Reason |
|---|---|---|
| Data pipeline into a warehouse | earliest | Completeness matters more than start-up speed |
| Live operations dashboard | latest | Only the current picture is useful |
| Payment reconciliation | none | Neither skipping nor replaying is acceptable without a decision |
| New microservice needing history | earliest | It must build its state from past events |
| Temporary debugging consumer | latest | Only watching current traffic |
| Topics that gain partitions often | earliest | Avoids losing the first events of new partitions |
Believing the setting replays data on every restart. It only applies when no valid bookmark exists.
Testing with latest and producing before the consumer has joined. Events fall into the start-up gap and the test fails at random.
Ignoring retention when planning downtime. A maintenance window longer than retention makes the bookmark invalid on return.
Editing group offsets while consumers are running. Active members overwrite the change or the tool refuses to apply it.
Rehearse an outage in a test environment. Stop the consumer, produce data, wait past a shortened retention period, and restart. Seeing the chosen value act on real data teaches more than any document.
19Real-World and Industry Examples
How different kinds of systems pick a starting position, and why.
Kafka began at LinkedIn to move activity events between systems, and it is now used across industries such as retail, finance, telecommunications, travel and media. Large technology companies rely on it for event pipelines, operational data and analytics. The examples below are illustrative patterns rather than descriptions of any particular company’s internal setup, but they reflect how these choices are commonly made.
Online retail order processing
Orders, payments and shipments flow through Kafka topics. The fulfilment service uses earliest with careful de-duplication so that a rebuilt or newly scaled service never misses an order. Idempotent updates keyed by order number make repeats harmless.
Fraud detection
A scoring service starts with earliest when a new model is deployed, so it can warm up on recent history, then keeps a stable group for normal operation. The reset choice is paired with strict alerts on lag, because a missed window of transactions could hide fraud.
Live metrics and log monitoring
A dashboard that plots the last few minutes of application metrics uses latest. Old readings would only distort the picture, and a fast start matters more than history.
IoT telemetry from devices
Millions of sensors send readings. A long-term storage consumer uses earliest to avoid gaps in historical data, while a separate alerting consumer uses latest because an alert about a temperature from three days ago is meaningless.
Search index rebuild from change events
A team feeds database changes into Kafka and needs to rebuild a search index. They start a dedicated replay group with earliest, let it run to completion against a compacted topic, and then switch traffic to the rebuilt index.
Financial reconciliation
A nightly job uses none. If its bookmark is missing or expired, it stops and pages an engineer, who sets the correct business date before the job continues. The pause is an intended safety feature.
Onboarding a new team
A new analytics team subscribes to an existing topic. Because they need last week’s data for baseline reports, they create their own group with earliest, which does not disturb any existing group’s bookmarks.
20Frequently Asked Questions
The default is latest. If you do not set the option, a group with no valid bookmark begins at the end of each partition and sees only new events. Some frameworks change this default, so confirm the behaviour of the tool you actually use.
No. A running group with a valid bookmark ignores the setting. To replay, you must reposition the group with the group tool or seek operations, or start with a fresh group name.
It is applied per partition, at the moment that partition needs a starting position. Within one consumer, some partitions can resume from bookmarks while others are reset.
Most often the group is new and the value is latest, so the consumer starts at the end and waits for fresh events. Set earliest, or produce new events, and it will show data.
The consumer begins at the oldest event that still exists. Deleted events cannot be recovered by any consumer setting.
No initial offset means there is no bookmark at all, as with a new group. Out of range means a bookmark exists, but it points to a position that the broker no longer has or never had. The reset rule handles both.
Yes. The setting belongs to each consumer’s own configuration, and each group keeps its own bookmarks, so they can start and behave independently.
While the group has active members, its bookmarks are kept. After the group becomes empty, they are retained for the broker’s offsets retention period, which is seven days by default, and then removed.
Because a visible failure is often better than a silent mistake. It forces a person or program to choose the right starting point, which is important where skipping or repeating data has serious consequences.
No. It is a consumer-side setting only. Producers and brokers do not read it, although broker and topic retention settings influence when it is triggered.
Reposition the group to a timestamp using the reset-offsets option of the group tool while the group is stopped, or use the client’s timestamp lookup and seek operations inside the application. The reset setting alone cannot do this.
Not automatically. It avoids gaps, but it can create large backlogs and repeated side effects. Choose based on what missing and repeating events cost for that specific application, and make the consumer idempotent whichever value you pick.
21Summary and Key Takeaways
The auto.offset.reset setting is small, but it answers one of the most important questions in any Kafka application: where do I start reading when I have no reliable bookmark? It is used only when a group has no initial offset or when its saved offset is no longer valid, and it never overrides a healthy bookmark. Its three classic values give three personalities: earliest for completeness, latest for freshness, and none for caution. Because the decision can be triggered silently by retention, expiry, new partitions and long outages, the right habit is to choose it deliberately, write it down, and support it with idempotent processing, sensible retention and good monitoring.
Key Takeaways
- It is a fallback, not a replay switch — the setting applies only when a partition has no committed offset or the committed offset is out of range.
- Three values, three personalities — earliest starts at the oldest stored event, latest starts at the end of the log, and none raises an exception instead of guessing.
- The default is latest — leaving it unset means silently skipping history for new groups and expired bookmarks, so set it explicitly.
- Two clocks decide when bookmarks fail — event retention and offsets retention both default to seven days, and your worst outage must fit inside them.
- Applied per partition, lazily — the position is resolved when the consumer first needs it, which explains start-up races and missed events on new partitions.
- Use the right tool to reposition — replay or skip with the group tool, seek operations or a fresh group name, never by merely editing the setting.
- Design for duplicates and watch lag — idempotent consumers, stable group names, explicit configuration and lag alerts keep every choice safe in production.