Storing events is only half of the story. Somebody has to read them, and reading at scale is harder than it sounds.

A quick refresher

Apache Kafka stores streams of events, such as orders, payments, clicks, or sensor readings. Events are grouped into named categories called topics, and every topic is split into slices called partitions. Each partition is an ordered, append-only log, and every event in it has a position number called an offset. The applications that read these events are called consumers.

What goes wrong with one reader

Imagine a topic receiving fifty thousand events every second. A single consumer program might be able to process only ten thousand per second. The gap grows every moment, and the consumer falls further and further behind. You could start a second copy of the program, but then a new question appears: how do the two copies avoid reading and processing the same event twice, and what happens when one of them crashes?

Without some organising idea, the two consumers would either both read everything, wasting effort and causing duplicate work, or fight over which events belong to whom. Kafka solves this with a simple and elegant concept: the consumer group.

A consumer group is a set of consumers that cooperate to read a topic, sharing its partitions so that each partition is read by only one member at a time.

The simple definition

Every consumer chooses a name for its group, called the group identifier. All consumers that use the same group identifier belong to the same group. Kafka treats the group as one logical reader. Inside the group, the partitions of the subscribed topics are divided among the members. Each partition goes to exactly one member, and one member may hold several partitions.

The three big promises

flowchart LR
  T["Topic: orders - 4 partitions"] --> P0["Partition 0"]
  T --> P1["Partition 1"]
  T --> P2["Partition 2"]
  T --> P3["Partition 3"]
  P0 --> A["Consumer A"]
  P1 --> A
  P2 --> B["Consumer B"]
  P3 --> B
  subgraph G["Consumer Group: email-service"]
    A
    B
  end
        

The second superpower of consumer groups is that different groups do not disturb one another.

Every group gets the full stream

Inside one group, the partitions are divided. Between different groups, nothing is divided. Each group receives every event of the topic and keeps its own bookmark. The email service, the analytics team, and the fraud checker can each have their own group reading the same topic, and each sees every order. A slow group never slows the others, because their bookmarks are independent.

Two behaviours in one design

This design gives Kafka the strengths of two older messaging styles. Within a group, the members compete for partitions, which behaves like a work queue where each job goes to one worker. Across groups, every group gets a copy, which behaves like a broadcast where every subscriber hears the news. Kafka offers both with one mechanism.

flowchart LR
  T["Topic: orders"] --> G1["Group: email-service"]
  T --> G2["Group: analytics"]
  T --> G3["Group: fraud-check"]
  G1 --> R1["Sends receipts"]
  G2 --> R2["Builds dashboards"]
  G3 --> R3["Flags suspicious orders"]
        

The number of partitions and the number of consumers together decide who does what.

The sharing rule

Because a partition can be read by only one member of a group at a time, the number of partitions sets the ceiling on how many members can be busy. If a topic has six partitions, at most six members of one group can be reading. A seventh member would have nothing to do and would simply wait as a standby.

Why only one reader per partition?

Kafka keeps the events of a partition in strict order. If two members read the same partition at once, they would process events in overlapping and unpredictable ways, and the order would be lost. Giving each partition a single reader keeps order safe and makes each member’s work simple to reason about.

A group must remember how far it has read, or every restart would begin from scratch.

What is a committed offset?

Each event in a partition has an offset, a number that shows its position. As a group reads, it periodically saves the offset it has reached, called committing an offset. The saved number means “everything before this position has been handled, and the next event to read is this one.” The bookmark is saved per group, per partition.

Where the bookmarks live

Kafka stores these bookmarks in itself, in a special internal topic dedicated to offsets. Because that topic is replicated like any other, the bookmarks survive broker failures. When a consumer starts, it asks Kafka for the last committed offset of its partitions and continues from there. If no bookmark exists yet, a setting decides whether the group starts from the earliest available event or only from new events that arrive from now on.

Two positions to keep apart

flowchart LR
  E0["Offset 0-99 - handled and committed"] --> E1["Offset 100-119 - read, not yet committed"]
  E1 --> E2["Offset 120 and beyond - not yet read"]
        

Somebody must keep track of who is in the group and who owns what. In Kafka, one broker plays that role for each group.

What the coordinator does

For every consumer group, one broker acts as the group coordinator. It keeps the list of active members, receives their regular signals, notices when someone stops responding, starts a reshuffle when the membership changes, and stores the group’s committed offsets. Different groups may have different coordinators, which spreads the management work across the cluster.

Joining a group step by step

sequenceDiagram
  participant C as New Consumer
  participant B as Any Broker
  participant GC as Group Coordinator
  C->>B: Who is the coordinator for my group?
  B-->>C: Broker 2
  C->>GC: Join group and subscribe to topics
  GC-->>C: Partition assignment
  C->>GC: Fetch committed offsets
  GC-->>C: Last saved positions
  C->>GC: Heartbeats while reading
        

How does the group know a member is still healthy? Through small regular signals and one important time limit.

Heartbeats

Each consumer sends a small heartbeat to the coordinator every few seconds. It is like a wave that says “I am still here”. If the coordinator hears nothing for longer than the session timeout, it decides the member is gone, removes it from the group, and starts a reshuffle of partitions.

The processing time limit

Heartbeats alone are not enough. A consumer could be alive yet stuck, for example waiting forever on a slow database. To catch this, Kafka also sets a limit on how long a consumer may go without asking for more events. If a consumer takes too long to process a batch and misses this limit, it is treated as failed and removed from the group, even if its heartbeats are still arriving.

Teams change all the time. A rebalance is how a group redistributes its partitions when that happens.

What triggers a rebalance

The steps of a rebalance

flowchart TD
  A["Group has 3 members and 6 partitions"] --> B["Member 3 crashes"]
  B --> C["Coordinator misses heartbeats and declares it dead"]
  C --> D["Rebalance starts"]
  D --> E["Partitions of Member 3 are given to Members 1 and 2"]
  E --> F["Members continue from last committed offsets"]
        

Not all rebalances are equally disruptive. The way partitions are moved makes a big difference.

The classic stop-the-world style

In the older style, called eager rebalancing, every member gives up all its partitions at the start of the rebalance, and then everything is assigned afresh. During this short period, the whole group stops reading. In a large group, or when rebalances happen often, these pauses add up and lag grows.

The gentle, incremental style

Cooperative rebalancing is smarter. Members keep the partitions that do not need to move and continue reading them. Only the partitions that must change owners are paused and handed over. Most of the group keeps working through the change. Newer versions of Kafka also offer an improved group protocol that moves much of the coordination work to the broker side, making rebalances lighter still, especially in very large groups.

Assignment strategies

Many rebalances are caused by ordinary things like restarts during deployments. Kafka has a feature to make them calmer.

The problem with quick restarts

Normally, when a consumer restarts, Kafka sees the old member leave and a new member join. That triggers two reshuffles for a change that lasts only seconds. In a large group deployed one member at a time, this can cause a long chain of pointless reshuffles.

Giving members a stable name

With static membership, each consumer is given a fixed name of its own. If it disappears and returns within the session timeout with the same name, Kafka recognises it as the same member and returns its previous partitions without any reshuffle. The other members are not disturbed at all. If it fails to return in time, the normal rebalance begins.

When the group saves its bookmark decides what happens after a crash.

Three possible behaviours

Automatic and manual commits

By default, many consumers save their bookmark automatically every few seconds. This is easy, but it can save a position for events that were fetched yet not fully processed, risking loss if the consumer crashes. Careful applications turn this off and commit manually, after the work for a batch is truly finished. This gives at-least-once behaviour, which is the most common and practical choice.

If you could watch only one number for a consumer group, this is the one.

What is lag?

Lag is the gap between the newest event in a partition and the group’s committed bookmark for that partition. It tells you how many events are waiting to be processed. A small, steady lag is normal, because events arrive continuously. A lag that keeps growing means the group cannot keep up.

Reading the signal

When lag grows, there is a sensible order in which to respond.

flowchart TD
  L["Lag is growing"] --> Q1{"Is processing itself slow?"}
  Q1 -->|"Yes"| F1["Speed up processing"]
  Q1 -->|"No"| Q2{"Members fewer than partitions?"}
  Q2 -->|"Yes"| F2["Add more members"]
  Q2 -->|"No"| Q3{"One partition much busier?"}
  Q3 -->|"Yes"| F3["Review keys and hot partitions"]
  Q3 -->|"No"| F4["Plan more partitions"]
        

Sometimes one event is so strange that a consumer cannot process it, and the whole partition gets stuck behind it.

The problem

Consumers read a partition in order. If one event is malformed, or triggers a bug, the consumer fails on it. If the consumer simply retries forever without moving its bookmark, the partition is blocked and all events behind it wait. This troublemaker is called a poison message. Meanwhile, lag grows and other events, perfectly healthy, are delayed.

Sensible ways to cope

Groups are long-lived objects in the cluster, and they need a little housekeeping.

Naming groups well

The group identifier decides which consumers share work and which bookmarks are used. Choose clear, meaningful names that reflect the application or team, such as the email service or the fraud checker. Two unrelated applications accidentally sharing a group name would split the events between them, and each would miss half of the data.

Resetting a group’s position

Operators can move a group’s bookmark: back to the earliest event, forward to the newest, to a particular time, or to a specific offset. This is helpful for reprocessing after a bug fix, or for skipping a burst of unwanted events. The group must be stopped while its offsets are changed, otherwise the running members would overwrite the new bookmarks.

Cleaning up

Groups with no active members eventually lose their stored bookmarks after an expiry period, so a group idle for too long may start from scratch when it returns. Old, unused groups should be removed so they do not clutter monitoring and confuse operators.

Security

Kafka can require applications to prove their identity and can check which identities may read a topic and which may use a given group name. A well-run cluster gives each application its own identity, allows it only its own group names and only the topics it needs, and keeps credentials in a proper secrets system.

Consumer groups are elegant, but they come with rules you must respect.

Most group problems repeat across companies. Learn them once and avoid them forever.

Best practices at a glance

Let us watch consumer groups at work in an online shop during a sale.

It is a set of consumers sharing one group identifier that divide the partitions of a topic among themselves, so each partition is read by only one member at a time, with the group’s progress saved as shared bookmarks.

Yes, and this is a core feature. Each group receives every event and keeps its own bookmarks, so groups never interfere with each other.

As many as you like, but only as many as the number of partitions can be active at once. Extra members wait idle as standbys.

The coordinator notices the missing heartbeats, removes the member, and rebalances. The remaining members take over its partitions and continue from the last committed offsets.

You probably have more members in the group than partitions. Each partition can be read by only one member of a group at a time.

The group resumes from the last committed offset. Events processed after that bookmark but before the crash are read again. Design consumers to be safe against repeats and commit after processing.

A rebalance redistributes partitions after the group changes. In the older eager style, everyone stops while partitions are reassigned. Cooperative strategies, stable member names, and quick processing reduce the pause.

In an internal, replicated Kafka topic used for offsets, so the bookmarks survive restarts and broker failures.

Yes, as long as the events are still within the retention period. Stop the group, move its bookmarks back to an earlier offset or time, and restart it.

Most consumers use one, because groups provide bookmarks and automatic sharing. Special cases exist where an application assigns partitions to itself manually and manages its own position, but this is uncommon and needs careful design.

We began with a team of teachers marking a pile of exam papers, each bundle given to exactly one teacher, with the bundles reshuffled whenever the team changes. A Kafka consumer group is that team. Consumers that share a group identifier divide the partitions of a topic, so each partition has one reader at a time, order is preserved, and the group can grow or shrink while running. A group coordinator on one of the brokers tracks members through heartbeats, hands out assignments, and stores the group’s bookmarks in a replicated internal topic. Different groups read the same topic independently, giving Kafka both work-queue and broadcast behaviour at once. In return, you must respect a few rules: parallelism is capped by the partition count, rebalances cause pauses, processing must stay within its time limit, consumers must tolerate repeated events, and lag must be watched closely. Master those, and consumer groups will carry your reading workload calmly at any scale.