What Is a Consumer in Kafka?
A beginner-friendly guide to the programs that read data out of Kafka, how they keep track of their place, and how they share work as a team.
Imagine a public library that never closes and never removes a book from its shelves. Readers can walk in at any hour, pick up exactly where they left off in a long series, borrow the same book as someone else without taking it away from them, and read at whatever pace suits them. Nobody needs to ask permission from the library before starting, and nobody disturbs any other reader by being there. This is remarkably close to how a Kafka consumer behaves when it reads data. In this guide we will build up the whole picture slowly, defining every new word the first time it appears.
1Why Kafka Needs a Reading Side at All
Before defining a consumer precisely, it helps to see the role it fills.
Data has to go somewhere useful
Kafka is built to move large streams of events, such as customer orders, sensor readings, or website clicks, from wherever they happen into whatever systems need to react to them. A program that creates and sends this data into Kafka is called a producer. But data sitting inside Kafka, never read by anything, would be pointless. Something needs to actually pull that data back out and put it to use, whether that means updating a dashboard, charging a payment, or triggering a shipment.
This is the job of a consumer: the program on the reading side of Kafka, responsible for fetching events and doing something useful with them.
Think of a newspaper that gets printed and stacked at newsstands every morning. The printing press is the producer, doing its job the moment the papers are ready. But the newspaper only becomes useful once readers actually pick up a copy and read it. A Kafka consumer plays exactly that role for the events sitting inside Kafka.
Reading without disturbing anyone else
A key part of what makes a consumer special is that reading an event does not remove it or change it in any way. Many completely different consumers can read the exact same event, at completely different times, without ever interfering with one another. This is quite different from many older messaging systems, where reading a message often meant taking it away from everyone else.
2So, What Exactly Is a Consumer?
A precise definition, then the plain-English version.
A short definition
A consumer is any program that connects to a Kafka cluster, subscribes to one or more topics, and reads the events stored in them. A topic is simply a named stream of related events. Once subscribed, a consumer fetches events, typically processes each one in some meaningful way, and keeps track of how far it has read, so it always knows where to continue from next.
In everyday terms, a consumer is a reader that has told Kafka “I am interested in this particular category of events”, and Kafka responds by making those events available for it to pull, whenever it is ready.
A consumer decides its own pace
Unlike some systems where data is pushed onto a receiver whether it is ready or not, Kafka consumers generally pull events when they choose to, a design often called a pull model. This means a consumer that is temporarily busy or slow is never overwhelmed by a flood of incoming data; it simply asks for more whenever it has finished with what it already has.
A single program can be both a producer and a consumer at the same time, publishing events to some topics while subscribing to and reading from others. This is common in larger systems built from many cooperating services.
3The Core Ideas Behind How Consumers Work
A handful of terms explain most of what a consumer does. Learn these and the rest becomes easy.
Subscription
The act of a consumer telling Kafka which topics it wants to receive events from, before it can start reading anything.
Offset
A number marking an event’s exact position within a partition, used by a consumer to remember precisely where it has read up to.
Poll
The act of a consumer actively asking Kafka for the next batch of available events, the core step of the pull model.
Consumer Group
A named team of consumers working together, sharing the job of reading a topic’s partitions between them.
Committed Offset
A bookmark saved by a consumer, recording how far it has successfully processed, so it can resume correctly after a restart.
Putting these ideas together
A consumer first subscribes to a topic, then repeatedly polls Kafka for new events, processes whatever it receives, and periodically commits its offset as a bookmark of its progress. This loop of poll, process, and commit is the basic rhythm every Kafka consumer follows, no matter how simple or complex the surrounding application is.
flowchart LR
S[Subscribe to topic] --> P[Poll for new events]
P --> W[Process events]
W --> C[Commit offset]
C --> P
4Offsets: How a Consumer Remembers Its Place
This one idea is what makes reliable, resumable reading possible.
A running position number
Every event inside a Kafka partition has an offset, a number starting at zero and increasing by one for each new event written. As a consumer reads through a partition, it is always reading at some particular offset, and it can look at that number to know exactly how far it has progressed.
Why this matters so much
Because a consumer can save its current offset as a bookmark, it never has to guess where it left off. If it crashes and restarts an hour later, it can simply resume from its last saved offset, picking up exactly where it stopped, without reprocessing everything from the very beginning or accidentally skipping anything.
An offset works much like a bookmark placed inside a long novel. You can close the book at any point, walk away for days, and pick it back up exactly where you left off. You could also deliberately move the bookmark back a few chapters to reread something, which a consumer can do too, if it needs to reprocess older events.
flowchart LR
O0["Offset 0 read"] --> O1["Offset 1 read"] --> O2["Offset 2 read"] --> O3["Offset 3: bookmark here"] --> O4["Offset 4: not yet read"]
5Committing Offsets: Saving the Bookmark
Remembering a position is only useful if it is saved somewhere durable.
What committing actually means
Committing an offset means telling Kafka to durably record how far a consumer has progressed, so that record survives even if the consumer itself restarts or a different consumer later takes over its work. Kafka stores these committed offsets safely, associated with the specific consumer group that committed them, so each group can track its own independent progress.
Choosing when to commit
A consumer can commit its offset automatically at regular intervals, which is simple but carries a small risk of processing an event twice, or occasionally missing marking one as done, depending on exact timing. Alternatively, a consumer can commit manually, right after it has finished fully processing a batch of events, giving it tighter control over exactly when progress is considered safely recorded.
| Commit style | How it works | Trade-off |
|---|---|---|
| Automatic | Committed periodically in the background | Simple, but timing is less precise |
| Manual | Committed explicitly by the application’s own code | More control, but requires careful handling |
Committing too early, before an event has actually finished being processed, can cause that event to be silently skipped if the consumer crashes right afterward. Committing too late can cause an event to be processed again unnecessarily. Getting the timing right matters.
6Consumer Groups: Sharing the Work
One consumer alone can only do so much. Groups let many work together.
What a consumer group is
A consumer group is a named collection of consumers that work together as a team to read a topic. Kafka divides the topic’s partitions among the members of the group, so that each partition is being actively read by exactly one member of that group at any given time. This lets the total reading workload be shared and processed in parallel, rather than resting on a single consumer alone.
An example of sharing partitions
If a topic has six partitions and a consumer group has three members, Kafka typically assigns two partitions to each consumer. If the group later grows to six members, each consumer might then handle just one partition, spreading the work out even further as demand grows.
Picture a large pile of exam papers to be graded. A team of teachers splits the pile into smaller stacks, with each teacher grading their own stack independently. Together they finish far faster than one teacher working alone, and each paper is still graded exactly once by the team.
flowchart TD
T[Topic with 6 partitions]
T --> P0[Partition 0]
T --> P1[Partition 1]
T --> P2[Partition 2]
T --> P3[Partition 3]
T --> P4[Partition 4]
T --> P5[Partition 5]
P0 --> C1[Consumer 1]
P1 --> C1
P2 --> C2[Consumer 2]
P3 --> C2
P4 --> C3[Consumer 3]
P5 --> C3
7Different Groups, Independent Readers
The other half of what makes consumer groups so powerful.
Every group gets the full picture
Different consumer groups reading the same topic are completely independent of each other. Each group receives every event published to that topic, tracked using its own separate set of committed offsets. One group being slow, fast, or temporarily offline has no effect whatsoever on any other group reading the same data.
A concrete example
Imagine an “orders” topic being read by three separate consumer groups at the same company: one group handling payment processing, another updating a live analytics dashboard, and a third archiving every order into long-term storage. Each group works through the exact same stream of events, entirely independently, at whatever pace suits its own purpose.
Think of a popular public notice board. One group of people reads it for job postings, another for community events, and another for lost pet notices. Everyone reads the exact same board, but each group pays attention to it for their own reasons, on their own schedule, without interfering with anyone else reading it too.
flowchart TD
T[Topic: orders]
T --> G1[Consumer Group: Payments]
T --> G2[Consumer Group: Analytics]
T --> G3[Consumer Group: Archiving]
8Rebalancing: What Happens When a Group Changes
Consumer groups are not static; Kafka reorganises them automatically as membership changes.
Why rebalancing happens
Whenever a consumer joins or leaves a group, whether intentionally, by scaling the group up, or unexpectedly, because a consumer crashed, Kafka needs to redistribute the group’s partitions among the remaining or new set of members. This process is called a rebalance, and it ensures every partition still has exactly one active reader within the group at all times.
A brief pause is normal
During a rebalance, there is typically a short pause in processing while Kafka works out the new assignment of partitions to consumers. For most systems, this pause is brief and expected, but frequent, unexpected rebalances, especially caused by consumers repeatedly crashing or becoming unresponsive, can noticeably slow down overall processing and deserve investigation.
Imagine a delivery company reassigning routes among its drivers whenever someone calls in sick or a new driver joins. There is a brief moment of reorganising who covers which route, but once settled, deliveries continue smoothly with the new arrangement.
A consumer that regularly takes too long to process events, or crashes often, can trigger frequent rebalances, which hurts the whole group’s throughput. Keeping consumers healthy and responsive is important for avoiding this problem.
9Reading From the Beginning, or Only What’s New
A consumer has real choices about where in a topic’s history it wants to start.
Starting positions
When a new consumer group first starts reading a topic, it typically has to decide where to begin. It can choose to start from the very earliest available event still stored in the topic, effectively replaying the topic’s full history from the start, or it can choose to start only from events published from this moment onward, ignoring anything that happened before it connected.
Why this choice matters
A system doing historical analysis or building up state from scratch often wants to start from the earliest available event, so nothing important is missed. A system only interested in live, ongoing activity, such as a real-time alerting dashboard, might prefer to start only from the latest point onward, ignoring old history that is no longer relevant to its purpose.
Example: Rebuilding a Cache from History
A brand-new service that needs to build up an internal cache of every customer’s current status might deliberately start reading from the earliest available event, replaying the full history to reconstruct an accurate, up-to-date picture before switching to live processing.
This starting-point choice only really matters the first time a consumer group begins reading a topic. Once it has committed offsets, it will simply resume from its last committed position on every future restart, regardless of this initial setting.
10Delivery Guarantees for Consumers
Understanding exactly what a consumer can promise about processing each event.
Three familiar promises
Combined with how and when offsets are committed, a consumer can end up with one of three delivery behaviours. At-most-once means an event might occasionally be skipped if a crash happens before it is processed, but it will never be processed more than once. At-least-once means an event is never skipped, but it might occasionally be processed twice if a crash happens after processing but before the offset is committed. Exactly-once means an event is neither skipped nor duplicated, though it usually requires extra coordination to achieve reliably.
| Guarantee | What can happen | Typical use |
|---|---|---|
| At most once | May skip an event, never duplicates | Non-critical metrics where an occasional miss is acceptable |
| At least once | Never skips, may occasionally duplicate | Most everyday pipelines, especially with safe reprocessing |
| Exactly once | Neither skips nor duplicates | Financial totals and other precise counting |
Many teams choose at-least-once processing and design their consumers to handle receiving the same event twice safely, a property called idempotence. This is often much simpler to achieve reliably than a full exactly-once setup, and works well for most everyday needs.
11Consumer Lag: The Most Important Health Signal
A single measurement tells you more about a consumer’s health than almost anything else.
What lag actually measures
Consumer lag is the difference between the latest offset available in a partition and the offset a consumer has actually committed. In plain terms, it measures how far behind a consumer currently is compared to the newest data waiting to be read. A lag of zero means the consumer is fully caught up with everything published so far.
Why rising lag is a warning sign
If lag is steadily climbing over time, it means new events are arriving faster than the consumer can process them, and the gap between what has happened and what has actually been handled keeps growing. Left unchecked, this can lead to seriously delayed processing, or even data being removed by the topic’s retention settings before a struggling consumer ever gets a chance to read it.
Think of a growing pile of unread letters in a mailbox. If new letters keep arriving faster than you can open and read them, the pile keeps growing, and eventually some very old letters might become irrelevant or get thrown away before you ever get to them.
If a team can only monitor one thing about their consumers, consumer lag is usually the best choice. It is the clearest, earliest sign that a consumer is falling behind the real world.
12Advantages and Trade-offs
Understanding both sides helps in designing consumers well.
Advantages
- Reading never removes or disturbs data for other consumers
- Consumer groups allow work to be shared and scaled easily
- Offsets allow safe, precise resuming after any restart or crash
- Independent groups can each get their own full view of a topic
- Old events can be replayed when needed, within the retention period
Disadvantages / Trade-offs
- Rebalances can cause brief pauses when group membership changes
- Getting offset commit timing right requires care to avoid skipped or duplicated processing
- A struggling consumer can accumulate lag that is not always obvious without monitoring
- Achieving true exactly-once processing adds real complexity
13Real-World Uses of Consumers
Seeing a few concrete examples brings the whole idea to life.
Payment Processing
A payment service consumes order events, checks payment details, and confirms or declines each transaction, all as part of a larger, decoupled system.
Real-Time Dashboards
A live analytics dashboard consumes a stream of events to update charts and counters the instant new activity happens, without needing to constantly query a database directly.
Fraud Detection
A fraud detection system consumes transaction events as they occur, applying rules or models in real time to flag suspicious activity before it causes harm.
Search Index Updates
A search service consumes product change events and updates its search index accordingly, keeping search results fresh without repeatedly polling the original database.
Long-Term Archiving
An archiving consumer reads every event from a topic and writes it into long-term storage, preserving a full historical record for compliance or future analysis.
14Best Practices and Common Mistakes
Practical lessons that save teams from painful surprises.
Best practices
Match consumer count to partition count sensibly
Having more active consumers in a group than partitions leaves some consumers permanently idle, since a partition is read by only one group member at a time.
Commit offsets thoughtfully
Choose a commit strategy and timing that matches how much duplicate or skipped processing your system can safely tolerate.
Design processing to handle duplicates safely
Building consumers that can process the same event twice without causing harm makes the whole system far more resilient to failures.
Monitor consumer lag continuously
Treat lag as a core health metric, catching struggling consumers early before they fall dangerously far behind.
Keep processing per event reasonably fast
Slow processing within a consumer contributes directly to lag and can also trigger unwanted rebalances if it takes too long to respond.
Mistakes beginners often make
Reading an event from Kafka never removes it. Beginners sometimes design systems around this false assumption, only to be surprised when other consumers, or the same one after a rewind, read it again.
Without regular monitoring, a slowly growing lag can go unnoticed for a long time, until it suddenly causes serious delays or missed data once retention limits are reached.
Adding more consumers to a group than there are partitions does not speed anything up; the extra consumers simply sit idle, doing no useful work.
15Frequently Asked Questions
No. Reading is non-destructive. The event stays in the topic, available to be read again by the same consumer or any other, until it is eventually removed according to the topic’s retention settings.
Yes. Any number of independent consumer groups can subscribe to and read the same topic, each getting its own full, independent view of the data at its own pace.
Kafka detects that it has stopped responding and triggers a rebalance, reassigning its partitions to the remaining active consumers in the group, so reading continues with minimal disruption.
Yes, as long as those events are still within the topic’s retention period. A consumer can intentionally move its offset backward to reprocess history, which is useful for fixing mistakes or testing new logic.
Yes, brief fluctuations are normal, especially during traffic spikes. What matters more is whether lag consistently trends upward over time without recovering, which signals a genuine, ongoing problem.
In practice, yes. Consumers within the same group are expected to perform the same kind of processing, since they are collectively sharing one logical job across the topic’s partitions.
A single consumer can only read as fast as one process allows, even if a topic has many partitions. A consumer group spreads that same work across multiple processes, letting them read and process partitions in parallel.
16Summary and Key Takeaways
A consumer is the program on the reading side of Kafka, responsible for subscribing to topics, pulling events at its own pace, and processing them without ever removing them for anyone else. Offsets let a consumer track exactly where it has read up to, and committing those offsets turns that position into a durable bookmark that survives restarts. Consumer groups let many consumers share the work of reading a topic’s partitions in parallel, while completely independent groups can each read the same topic on their own terms. Understanding lag, delivery guarantees, and rebalancing helps teams build consumers that stay healthy, keep up with real-world data, and recover gracefully whenever something goes wrong.
Key Takeaways
- A consumer reads events from Kafka topics — pulling data at its own pace without ever removing it.
- Offsets track a consumer’s exact position — letting it resume precisely after any restart.
- Committing turns a position into a durable bookmark — timing this well avoids skipping or duplicating processing.
- Consumer groups share the reading workload — dividing a topic’s partitions among their members.
- Different groups are fully independent — each gets its own complete, separately tracked view of a topic.
- Rebalancing keeps groups organised — automatically reassigning partitions whenever membership changes.
- Consumer lag is the key health signal — showing how far behind a consumer is from the latest available data.