Sharing work among many workers sounds simple until you list what can go wrong.

The problem of sharing a topic

A Kafka topic is divided into partitions, and each partition is an ordered list of messages. A single consumer can read every partition of a topic, but when the volume grows, one consumer becomes too slow. The natural answer is to start several consumers that share the work. Kafka calls such a team a consumer group, and every group is identified by a name called the group identifier.

The core rule of a consumer group is that each partition is read by only one consumer in the group at any moment. This keeps the order of messages inside a partition and prevents two consumers from handling the same message. If there are more consumers than partitions, the extra consumers simply wait without work, ready to take over if another consumer leaves.

What the coordinator has to guarantee

The coordinator must make sure that all consumers in a group agree on who owns which partition. It must notice quickly when a consumer dies so its partitions are not abandoned. It must also keep track of how far the group has read in each partition, so that a new owner can continue exactly where the previous owner stopped. These jobs are the reason a special broker role and a special protocol exist.

Independent groups do not disturb each other

Different groups reading the same topic are completely independent. Each group has its own coordinator relationship, its own membership list, and its own saved positions. A payment audit group and a search indexing group can both read every message of the same topic at their own speed. A rebalance in one group never causes a rebalance in another.

A handful of terms appear in every explanation, so it is worth learning them well before going further.

The two “leaders” that confuse everyone

Kafka uses the word leader in two unrelated ways. A partition leader is a broker that serves reads and writes for one partition. The group leader is a consumer that computes the partition split for its group. The coordinator is a broker too, but it is a separate role from either kind of leader, although by design the coordinator for a group happens to be the leader of one specific internal partition, which we will see in the next chapter.

A day in the life of one member

To make the vocabulary concrete, follow a single consumer from morning to night. It starts and asks any broker where its coordinator lives. It connects there and joins the group, receiving a member identifier. After the handshake, it learns that it owns partitions four and five. It fetches the saved bookmark for each and begins reading directly from the brokers that lead those partitions, not from the coordinator.

While it works, a background thread sends a heartbeat every few seconds, and the main thread polls for messages, processes them, and commits progress. Later a new member arrives, a rebalance begins, and our consumer learns about it in a heartbeat answer. It rejoins, perhaps gives up partition five, and continues with partition four. At the end of the day it closes cleanly, sending a leave message, and its partition goes to someone else. Every word in the glossary above appears in this short story, which is a good way to check that you have understood them.

Every group needs exactly one coordinator, and every consumer must be able to find it without asking a central directory.

A clever trick using an internal topic

Kafka keeps a special internal topic named __consumer_offsets. It stores committed positions and group membership records. By default it has fifty partitions, and its size is controlled by the setting offsets.topic.num.partitions. Kafka calculates a number from the group identifier using a hash, takes the remainder after dividing by the number of partitions of this internal topic, and arrives at one partition number.

The broker that currently leads that internal partition is the group coordinator for that group. This design is elegant because it reuses the existing machinery of partition leadership. There is no extra election just for coordinators, and the coordinator naturally lives next to the data it must manage, since that same partition stores the group’s records.

How a consumer discovers its coordinator

A new consumer starts by contacting any broker from its bootstrap list and sending a request called FindCoordinator, giving its group identifier. The broker computes the hash, looks up the leader of the matching internal partition, and returns that broker’s address. The consumer then opens a connection to that broker and sends all group related requests there.

flowchart LR
    C["Consumer with group id"] -->|"FindCoordinator request"| B1["Any bootstrap broker"]
    B1 -->|"Hash of group id modulo partition count"| P["Internal offsets partition number"]
    P -->|"Leader of that partition"| B2["Coordinator broker for this group"]
    B2 -->|"Address returned"| C
    C -->|"Join, heartbeat, commit"| B2
        

What happens when the coordinator moves

If the coordinator broker fails or its internal partition gets a new leader, the group’s coordinator changes automatically. The new broker reads the internal partition to rebuild the group’s state, including members and committed offsets. During this loading time, requests may receive an answer saying the coordinator is loading. After it finishes, consumers that receive a not-coordinator answer simply ask again with a new FindCoordinator request and continue.

This is a strong reason why the internal topic must be well replicated. The default replication factor for it is three in production, and a single copy development setup is not safe. If the internal topic loses its in-sync members below the minimum, group operations such as offset commits can fail even though ordinary topics still work.

What the internal topic actually stores

The internal offsets topic holds two kinds of records. The first kind is committed offsets: one record per group, topic, and partition, holding the bookmark and some optional extra text that applications may attach. The second kind is group metadata: a record describing the group’s current members, its generation, the chosen assignment strategy, and the assignment handed to each member. Both kinds are written by the coordinator, and both are what the next coordinator reads to bring a group back to life after a failover.

Because the topic is compacted, old records for the same key are cleaned away over time, and a special empty record called a tombstone marks a removed group or expired bookmark. The time a new coordinator needs to load its partitions depends on how much data is still stored there. A cluster with a huge number of groups, or with many old tombstones waiting to be cleaned, takes longer to load after a restart. This is one more reason to delete groups that are no longer used, and to keep the cleaner of the internal topic healthy.

From the coordinator’s point of view, every group is a small machine that is always in exactly one state.

The five states

The coordinator tracks each group as a state machine. Knowing these states makes logs and metrics easy to read, because they simply tell you where in the story the group currently is.

flowchart TD
    E["Empty"] -->|"First member joins"| PR["PreparingRebalance"]
    PR -->|"All members joined or timed out"| CR["CompletingRebalance"]
    CR -->|"Leader sends assignment"| S["Stable"]
    S -->|"Member joins or leaves or times out"| PR
    CR -->|"Member changes during wait"| PR
    S -->|"Last member leaves"| E
    E -->|"Offsets expire or group deleted"| D["Dead"]
        

Reading the states as a story

A healthy group spends nearly all of its life in the stable state. Each visit to the two rebalance states is an interruption, and the longer those visits last, the longer consumers are idle. If you look at how many groups sit in each state across the cluster, a large number stuck in PreparingRebalance or CompletingRebalance is a strong sign of trouble, such as slow members or misconfigured timeouts.

The heart of the protocol is a two-step conversation that every member has in every rebalance.

Step one: JoinGroup

When a consumer starts, or when it learns that a rebalance is happening, it sends a JoinGroup request to the coordinator. The request carries the group identifier, the member identifier if it has one, the session timeout, the rebalance timeout, and a list of the assignment strategies it supports together with its subscription, meaning the topics it wants to read.

A brand new consumer has no member identifier. Modern brokers answer its first request by saying that a member identifier is required and providing one, and the consumer then sends the request again with that identifier. This small extra round trip prevents a consumer that crashes before finishing the join from leaving a ghost member behind.

The coordinator waits, then picks a leader

After the first join request arrives, the coordinator waits until all existing members have re-joined or until the rebalance timeout expires. For a brand new group, it also applies a short initial delay controlled by group.initial.rebalance.delay.ms, which lets a team of consumers starting together join in one round instead of triggering many rebalances. Once the waiting ends, the coordinator increases the generation number, chooses one member as the group leader, and selects a single assignment strategy that all members support.

Each member’s JoinGroup call is then answered. Ordinary members receive the new generation and their member identifier. The leader receives more: the full list of members and their subscriptions, so it can compute the split.

Step two: SyncGroup

Every member now sends a SyncGroup request. Non-leader members send an empty request, simply asking for their share. The leader sends the complete assignment it computed, listing which partitions belong to which member. The coordinator stores this assignment, then answers every waiting SyncGroup request, giving each member its own part. The group is now stable, and consumers begin fetching messages from their assigned partitions.

sequenceDiagram
    participant A as Consumer A
    participant B as Consumer B
    participant Co as Coordinator
    A->>Co: "JoinGroup with subscription"
    B->>Co: "JoinGroup with subscription"
    Co->>Co: "Wait for all members then bump generation"
    Co-->>A: "Join response - you are the leader, here are all members"
    Co-->>B: "Join response - you are a member"
    A->>A: "Compute partition assignment"
    A->>Co: "SyncGroup with full assignment"
    B->>Co: "SyncGroup with empty request"
    Co-->>A: "Your partitions"
    Co-->>B: "Your partitions"
    A->>Co: "Heartbeat"
    B->>Co: "Heartbeat"
        

What each member advertises and how the coordinator chooses

The join request is a small contract. Each member lists the assignment strategies it can run, in order of preference, and attaches its subscription, which names the topics it wants and may include extra bytes of private information used by custom strategies, such as the partitions it currently owns. The coordinator looks across all members and picks the first strategy that every member supports, weighing the preference lists so that the most widely liked choice wins. If no strategy is shared by every member, the join is rejected as inconsistent, and the mismatched member sees an error that clearly says so.

The join request also carries the rebalance timeout, which is how long the member says it needs to rejoin during a rebalance. The coordinator uses the largest such value among the members as the deadline for the join phase. A member that fails to send its join request before that deadline is removed from the group, which is one more way slow or stuck consumers cost the whole group time.

When something goes wrong inside the handshake

The handshake has its own failure cases, and the group has to handle them without getting stuck. If a member disappears between the join and the sync steps, the coordinator notices through the missing sync request or a missing heartbeat, and starts another rebalance from the beginning. If the group leader disappears before delivering the assignment, the coordinator cannot complete the round, so it restarts the process and picks a new leader from the members that remain.

Each restart of the handshake raises the generation number again. Members that were still working on the previous round find their next request rejected because the generation is out of date, and they simply rejoin. This is why several quick rebalances in a row can feel like a stormy sea: every change makes the previous round obsolete, and members spend their time re-entering the handshake instead of reading messages.

The fairness of the split is decided by a pluggable strategy that runs on the group leader, not on the broker.

Why the client computes the split

In the classic protocol, the coordinator deliberately avoids computing the assignment. The reason is flexibility: applications can supply their own strategy without changing the brokers. Kafka Streams, for example, uses a specialized strategy that understands its own tasks and state stores. The coordinator only checks that the members share at least one strategy in common and passes the leader’s result to everyone.

The built-in strategies

Why stickiness matters so much

Many consumers hold expensive local state, such as caches, open files, or in-memory aggregates built from their partitions. Every time a partition moves, the new owner must rebuild that state, which may take minutes. A sticky strategy minimizes these moves, so the price of a rebalance is limited to the partitions that really need a new home.

Mixing strategies safely

Each member advertises a list of strategies in order of preference. The coordinator picks one that every member supports. During a rolling upgrade from an old strategy to the cooperative one, teams first add the new strategy alongside the old one, roll through all members, and only then remove the old one. Skipping this staging can make members disagree and cause the whole group to fall back to the eager behavior or fail to form.

The coordinator learns that a member is healthy only from messages it receives, so timing settings are the backbone of failure detection.

Two clocks, two kinds of failure

A consumer can fail in two very different ways. It can die completely, or the whole machine can vanish. Or it can stay alive but stop making progress, for example because its processing code is stuck. Kafka uses two separate clocks to detect these two cases.

The first clock is the session timeout, set by session.timeout.ms. A background thread inside each consumer sends heartbeats to the coordinator at a fixed interval, set by heartbeat.interval.ms. If the coordinator receives no heartbeat for the full session timeout, it declares the member dead and starts a rebalance. This detects crashed processes and lost machines.

The second clock is the poll interval, set by max.poll.interval.ms. The application must call the poll method regularly to fetch more messages. If too much time passes between two polls, the consumer concludes that its own processing is stuck, leaves the group on its own, and later rejoins. This detects a live but stalled application, even though its background heartbeats still flow.

The background heartbeat thread

Older consumers sent heartbeats from the same thread that processed messages, so slow processing looked like a dead consumer. Modern consumers use a separate thread, which is why the poll interval exists as an independent check. This separation lets you set a short session timeout for fast crash detection while still allowing long processing times between polls.

Leaving politely versus disappearing

A consumer that closes cleanly sends a LeaveGroup request, and the coordinator starts the rebalance right away. This is the fastest and gentlest way to change membership, because the group learns of the departure in an instant instead of waiting for a timeout. That is why applications should always close their consumers properly during shutdown, giving the client library time to send the leave message and to commit final progress.

A consumer that crashes, is killed, or loses its machine sends nothing. The coordinator only finds out when the session timeout runs out. During that whole time, the partitions of the missing member are unread, and lag builds up silently. If the session timeout is long, this quiet period can be long. This is the basic trade-off of failure detection: the longer you wait to declare a member dead, the fewer false alarms you get, but the longer real failures go unnoticed.

Container platforms often stop an application with a signal and a short grace period. If the application does not shut down its consumer within that period, the platform kills it, and the clean leave never happens. Make sure the grace period is long enough for the consumer to close, and that the application really handles the stop signal.

How the coordinator responds to heartbeats

A heartbeat response is more than an acknowledgment. When the coordinator has started a rebalance, it tells each member in its heartbeat answer that a rebalance is in progress. Members learn about the rebalance through this signal and then send their JoinGroup requests. This means that a longer heartbeat interval slows the spread of rebalance news, since a member only learns at its next heartbeat.

Knowing the causes lets you decide which rebalances are healthy and which are avoidable waste.

Healthy versus wasteful rebalances

Some rebalances are necessary: a real instance was added or removed, or a topic really grew. Others are self-inflicted. Missed poll deadlines from slow processing, tiny session timeouts, frequent restarts, and automatic scaling that keeps adding and removing instances create rebalances that add no value. A useful habit is to explain every rebalance in a fleet by naming its trigger, and to fight the ones that have no good reason.

How quickly the group notices each trigger

Different triggers reach the coordinator at very different speeds, and this shapes how long partitions sit unread. A clean leave is noticed instantly. A new member is noticed as soon as its join request arrives. A crashed member is noticed only after the session timeout. A stuck member is noticed after the poll deadline, which is usually much longer than the session timeout. A change in partition count is noticed only when the leader next refreshes its topic metadata, which by default happens every few minutes.

This ordering explains many puzzling delays. If a topic gains partitions and the new partitions seem ignored for a few minutes, the group is simply waiting for the next metadata refresh. If a crashed instance leaves its partitions unread for nearly a minute, that is the session timeout doing exactly what it was told to do. Knowing which clock governs each event lets you decide whether a delay is normal or a symptom of trouble.

The special case of topic patterns

Applications that subscribe with a name pattern receive a rebalance whenever a matching topic appears or disappears. In environments where topics are created and deleted often, this can cause a rebalance every few minutes. Prefer explicit topic lists when the set is stable, and make sure the periodic metadata refresh interval is not so short that it creates needless checking.

How much of the group must stop when membership changes is the biggest factor in rebalance pain.

The eager approach: everyone stops

In the original eager protocol, when any rebalance begins, every member gives up all of its partitions, rejoins the group, and waits for a completely new assignment. Even a member whose partitions will not change at all must stop and restart its work. This is simple and safe, but the cost grows with the size of the group. For a large group with heavy state, the total pause can last a long time.

The cooperative approach: only what must move

The incremental cooperative protocol changes the rule. Members keep the partitions they already own during the first round while they rejoin and report what they hold. The leader computes the new target split and then removes from the plan only those partitions that must move from one member to another. Those members revoke just those partitions. A second round then assigns the freed partitions to their new owners. Members whose partitions did not change never stop reading.

flowchart TD
    subgraph Eager["Eager rebalance"]
        E1["Change detected"] --> E2["All members revoke all partitions"]
        E2 --> E3["Rejoin and wait for new assignment"]
        E3 --> E4["All members resume"]
    end
    subgraph Coop["Cooperative rebalance"]
        C1["Change detected"] --> C2["Members keep partitions and rejoin"]
        C2 --> C3["Only moving partitions are revoked"]
        C3 --> C4["Second round assigns revoked partitions"]
    end
        

The two rounds in detail

Imagine a group of three members, where a fourth member joins. In the first round, every member sends a join request but tells the coordinator which partitions it already owns. The leader then computes the ideal new split for four members. It compares the ideal split with the current ownership and sees that, for example, two partitions should move from the first member to the newcomer. In the answer of this first round, the leader removes those two partitions from the first member’s assignment and gives nobody ownership of them for the moment.

The first member notices that its assignment shrank. It stops reading those two partitions, commits its progress for them, and triggers a second round. In this second round, the two partitions have no owner, so the leader gives them to the newcomer. Everyone else kept reading throughout. The price is that the change took two trips through the handshake, but each trip was short and harmless to members that had nothing to give up.

This two-round design has an important safety idea behind it: a partition is never owned by two members at once. It first becomes unowned and only then is given to the next owner, so there is no moment where two consumers could both read and process the same partition.

The rebalance listener callbacks

Applications can register a listener that receives three kinds of calls. One is called before partitions are taken away, which is the last chance to commit progress and release resources. Another is called after new partitions are given. A third is called when partitions were lost because the member was already removed, such as after a session timeout, in which case committing is no longer possible. In the eager protocol, the first call always covers all partitions; in the cooperative protocol, it covers only the partitions being moved.

Sometimes a consumer leaves only briefly and returns as the same instance, and Kafka can be told to treat this as no change at all.

The idea

Normally, every join gets a fresh member identifier, so a restarted consumer looks like a stranger. With static membership, the application sets a permanent name for each instance using group.instance.id. The coordinator remembers this name. When a consumer with the same name returns within the session timeout, the coordinator recognizes it and simply hands back its previous partitions, with no rebalance for the rest of the group.

A static member also does not send a leave message when it closes cleanly. From the coordinator’s view, the member is merely silent, and it waits for the session timeout before giving up on it. This is exactly what makes quick restarts invisible to the group.

The important trade-off

Static membership makes the session timeout more important. If it is long, restarts within that window are invisible, but a truly dead instance is only noticed after the same long delay, and its partitions stay unread until then. If it is short, the protection against restarts shrinks. Teams choose a value slightly longer than their typical restart or deployment time, for example a few minutes.

A group is only useful if it can resume exactly where it left off, and the coordinator is the keeper of that memory.

What an offset commit is

A committed offset is the group’s bookmark for one partition: the number of the next message it should read. When a consumer commits, it sends an OffsetCommit request to the coordinator, which writes a record into the internal offsets topic. The key of the record is the combination of group, topic, and partition, and the value is the offset. Because the topic is compacted, only the newest bookmark for each key is kept in the long run.

When a consumer receives a partition after a rebalance, it asks the coordinator for the last committed offset for that partition using an OffsetFetch request, and starts reading from there. If no bookmark exists, a setting decides whether to start from the earliest available message or from the newest.

Timing decides the delivery guarantee

Most applications choose at-least-once and make their processing safe to repeat, meaning that handling the same message twice has the same effect as handling it once. This is called idempotent processing, and it is the most practical protection against the duplicate messages that rebalances can cause.

Fencing by generation

Every commit carries the generation number and member identifier of the sender. If a consumer was removed from the group and a rebalance has already happened, its generation is out of date. The coordinator rejects its commit with an error, which prevents an old owner from overwriting the bookmark that the new owner is now advancing. This is why applications sometimes see a commit failure right after a rebalance: it is the fencing mechanism working as intended, not a bug.

sequenceDiagram
    participant Old as Old owner
    participant New as New owner
    participant Co as Coordinator
    Old->>Co: "Commit offset with generation 5"
    Note over Co: "Group moves to generation 6"
    New->>Co: "Fetch committed offset"
    Co-->>New: "Offset from generation 5"
    Old->>Co: "Late commit with generation 5"
    Co-->>Old: "Rejected - generation is out of date"
    New->>Co: "Commit offset with generation 6"
        

Automatic and manual commits

Automatic commits happen on a timer inside the poll call, which is convenient but not tied to the actual completion of work, so a crash can lose or repeat a stretch of messages that the timer did not match. Manual commits let the application decide exactly when progress is safe to save, either by waiting for the result or by sending the commit without waiting and handling failures later. Committing on every single message is safe but slow; committing after each batch is a common balance.

Choosing a commit style in practice

There are three common styles, and each fits a different need. The first is a blocking commit after every batch: the application waits until the coordinator confirms, which is safe and simple but adds a small delay to each batch. The second is a commit that does not wait for the answer, with a callback to report failures: it is faster, but a failed commit is discovered late, and a later successful commit usually makes the earlier failure harmless because bookmarks only move forward. The third is a mixed approach that uses non-waiting commits during normal work and one final blocking commit when the consumer shuts down or when partitions are being revoked.

The final blocking commit during revocation is especially valuable. It is the last chance to save progress before another member takes over, and it shrinks the window of duplicated messages to almost nothing. Applications that commit only on a timer, without this final step, are the ones that most often see bursts of duplicates right after a rebalance.

Committing very often has a cost of its own. Each commit is a write into the internal offsets topic, replicated across brokers, and processed by the single coordinator of the group. Thousands of members each committing many times per second can turn the coordinator into the busiest broker in the cluster. Commit per batch rather than per message unless the work is truly expensive to repeat.

How long offsets live

Bookmarks of a group that has no active members are removed after a retention period controlled by offsets.retention.minutes, which is seven days by default in recent versions. If a consumer group is stopped for longer than this, its saved positions disappear, and on restart it follows its reset policy, which may replay a huge amount of history or skip ahead. For groups that may pause for a long time, such as monthly batch jobs, check that the retention is long enough.

The coordinator is a broker, so it can crash, restart, or simply be asked to do too much.

Failover step by step

When the coordinator broker stops, the internal offsets partition it led elects a new leader from its in-sync replicas. The new leader becomes the coordinator for every group that hashes to that partition. It reads the partition from the beginning of the retained log to rebuild the membership and offset state. While this happens, requests receive a message that loading is still in progress and clients wait and retry.

Consumers that had been talking to the old coordinator get a not-coordinator error on their next request. They send FindCoordinator again, learn the new address, and reconnect. Members that were healthy usually continue in the same generation, but a group may need to rebalance if the recovered state shows changes.

Group size and the cost per member

Every member of a group costs the coordinator a little: memory to keep its record, time to process its heartbeats, and extra work in every rebalance. In a group with a handful of members these costs are invisible. In a group with hundreds or thousands of members, a single rebalance means the coordinator must collect, track, and answer that many join and sync requests, and the leader must receive and process a very large member list. The pause grows with the size of the group, and the chance that at least one member misbehaves during a round grows as well.

For this reason, very large groups benefit the most from cooperative and server-driven protocols, and from static membership. They also benefit from having fewer, larger instances doing more work each, provided the partition count still allows the needed parallelism. If a group truly needs enormous parallelism, consider splitting the work across several topics with separate groups, so that a problem in one does not disturb the others.

Coordinator load and hot brokers

Because groups are placed by hash, several very busy groups may end up with the same coordinator. Commits, heartbeats, and joins for all of them concentrate on one broker, which also leads some other partitions. Signs of an overloaded coordinator include slow commit times seen by clients, growing request queues on that broker, and consumers timing out their heartbeats. A group with thousands of members, or a group that commits extremely often, is a common source of such load.

Ways to relieve the pressure include committing less often, using fewer and larger consumer instances instead of thousands of small ones, and balancing leadership of the internal offsets partitions across brokers. Because the count of internal partitions cannot be changed safely, capacity is added by spreading the existing partitions, not by adding new ones.

Years of experience with rebalance pain led to a redesign in which the broker takes over more of the work.

Why a redesign was needed

The classic protocol has three well-known weaknesses. Its join and sync steps act as a barrier where the slowest member decides how long everyone waits. Its assignment logic lives in every client, so a bug or a version difference in client libraries can affect the whole group. And its members must all re-join even when only one member changed. Cooperative rebalancing softened these problems, but the design still relies on client-side coordination.

The core changes

The new protocol, introduced through a Kafka improvement proposal and made generally available in the Kafka 4.0 line, replaces the join, sync, and heartbeat requests with a single continuing heartbeat request. Each heartbeat carries the member’s subscription and the partitions it currently owns. The coordinator, not a group leader, computes the target assignment on the broker side, using a server-side assignor. There is no group leader among the consumers and no global barrier.

The coordinator then moves each member toward its target step by step. It tells a member which partitions to give up, waits until the member confirms, and only then hands those partitions to their new owners. Members that are not affected keep reading. The protocol tracks a group epoch that changes when the target changes and a member epoch that shows how far each member has moved toward it.

sequenceDiagram
    participant A as Consumer A
    participant B as Consumer B
    participant Co as Coordinator
    A->>Co: "Heartbeat with subscription and owned partitions"
    Co-->>A: "Assigned partitions"
    B->>Co: "Heartbeat - new member joining"
    Co->>Co: "Compute new target assignment on broker"
    Co-->>A: "Heartbeat response - release partition 3"
    A->>Co: "Heartbeat - partition 3 released"
    Co-->>B: "Heartbeat response - you now own partition 3"
    Note over A,B: "Other partitions were never interrupted"
        

Why the server-driven design is more robust

Moving the assignment logic to the broker removes a whole class of problems. In the classic design, a bug in one client version, or two clients running different versions of the same strategy, could produce a wrong or inconsistent split that every member then had to obey. With the split computed in one trusted place, operators upgrade the logic once, and every application benefits without redeploying its consumers.

The design also removes the fragile moment where the entire group waits for one leader to finish its calculation. Since members keep sending heartbeats and the coordinator moves them incrementally, a slow or paused member only delays its own partitions. The rest of the group continues, and the slow member can catch up when it recovers. In a large fleet where something is always slightly unhealthy, this independence is a major improvement.

What changes for operators

Session and heartbeat timing become broker-side group settings rather than client choices, which lets operators enforce sane values across every application. Assignors are also chosen on the server side, with built-in choices such as a uniform strategy and a range strategy. Upgrading needs care: the new coordinator must be enabled on the brokers first, and clients then opt in application by application, so old and new style groups coexist while you migrate. Always confirm details against the documentation of your exact Kafka version, since defaults and supported features evolve between releases.

A rebalance storm is a group that keeps reorganizing over and over, and it is one of the most common consumer group emergencies.

Recognizing the storm

Symptoms include consumer lag that grows in steps, throughput that drops to zero for seconds at a time, repeated log lines about joining and revoking partitions, and members that show up and vanish. Often the same members repeat the pattern, and the group spends much of its time in the two rebalance states.

A step-by-step method

Reading the logs like a detective

The coordinator logs the reason for every rebalance it starts, and these reasons are the single most useful clue. Typical reasons include a member being removed because its heartbeat expired, a member sending an explicit leave request, a new member joining, a member whose join request arrived after the deadline, and a change of subscription. Client logs add the other side of the story, such as a consumer reporting that it left the group because the time between polls exceeded the limit, or that its commit was rejected because the generation was out of date.

Line these messages up by time. If the coordinator says a member was removed for missed heartbeats and the client log for that same member shows a long garbage collection pause or no output at all during that period, the story is clear. If instead the coordinator shows a leave request at the same moment as a container stop message from the platform, the cause is a restart, not a fault. Collecting logs from the coordinator broker and from the clients into one searchable place, with accurate clocks, turns an hour of guessing into a few minutes of reading.

It also helps to count. Tally the rebalance reasons over the last hour. If almost all reasons are the same, for example poll deadline exceeded for a single member, you have found the cause. If the reasons are spread evenly across many kinds, the problem is broader, such as an unstable environment or an overloaded coordinator.

Both the clients and the brokers publish signals, and the best picture comes from reading them together.

Client-side coordinator metrics

Broker-side coordinator metrics

Brokers publish the number of groups in each state: empty, preparing rebalance, completing rebalance, stable, and dead. A stable fleet has almost all groups stable. Also watch request timing for the group-related requests, and the health of the internal offsets partitions, including their under-replication and leadership spread.

Command line tools

Building a useful alert set

A short list of alerts covers most needs. Warn when a group’s rebalance rate stays above its normal level for an extended time, since this is the earliest sign of instability. Warn when a group remains in a rebalancing state for longer than a few times its rebalance timeout, since this suggests a stuck member or an overloaded coordinator. Warn when consumer lag grows steadily while the member count stays constant, which suggests slow processing, and separately when lag grows in steps that match rebalances, which suggests instability.

Add alerts on the health of the internal offsets topic, including any under-replicated or offline partitions, because group operations depend on it. Finally, alert on groups that are empty but hold bookmarks near their expiry, so that a paused job does not lose its position without anyone noticing. Keep each alert tied to a clear first action in a runbook, so that whoever receives it knows where to look.

A short reference of the settings that matter most, with the reasons behind them.

Choosing the values: a worked way of thinking

Start from the slowest normal thing your consumer does. Suppose the longest processing time for one batch is forty seconds during a busy period. The poll deadline must be comfortably above that, for example two or three times larger, so a slow batch never causes a false departure. Next, look at the longest pause the process ever experiences, such as a garbage collection cycle or a brief network stall, and set the session timeout well above it so ordinary hiccups do not remove the member.

Then set the heartbeat interval to about a third of the session timeout, so that several heartbeats can be lost before the coordinator gives up. Finally, decide how long you are willing to leave partitions unread when a machine truly dies. That tolerance is the upper bound for the session timeout, and if the two needs conflict, the answer is usually to make the processing faster or the batches smaller, not to stretch either timeout beyond reason.

Review these numbers whenever the workload changes. A new downstream dependency, a larger message size, or a move to a slower machine type can silently push the real processing time past a setting that used to be safe, and the first sign will be a rebalance storm that nobody expected.

These mistakes appear again and again in real systems, and each one has a simple, safer alternative.

These illustrative stories show how the ideas combine in practice. They describe common patterns and are not tied to any specific company.

flowchart LR
    A["One member repeats the trouble"] --> B["Look at that member: pauses, poll time, resources"]
    C["Bursts of joins and leaves"] --> D["Look at deployments and autoscaling"]
    E["Many groups fail together"] --> F["Look at the coordinator broker and internal topic"]
    G["Duplicates after rebalance"] --> H["Look at commit timing and idempotent processing"]
        

A well-designed consumer group is one that rarely rebalances, and recovers gracefully when it must.

Design for rebalances, not against them

Rebalances are a normal part of a healthy system, because instances come and go. Design consumers so that a rebalance is cheap: keep local state small or rebuildable, commit progress at sensible points, process messages in a way that can be repeated safely, and handle partition revoke and assign events carefully. A consumer that survives being interrupted at any moment is far easier to operate than one that assumes a perfectly steady world.

Size the group with care

The number of partitions sets the maximum parallelism. Choose it from the peak throughput you need divided by the speed of one consumer, add headroom, and remember that partitions cannot be reduced later. Match the number of consumer instances to the workload rather than to the maximum, and prefer a moderate number of well-sized instances to a very large number of tiny ones, because every member adds heartbeat and join work for the coordinator.

Security and access

Consumers need permission to read the topics they consume and permission to use the group identifier, since group operations are protected as their own resource type. Grant these permissions narrowly, by application, so one application cannot join or reset another’s group. Resetting offsets is a powerful action that can cause mass reprocessing or data skipping, so restrict it and record who does it. Encrypted connections and authentication protect the join and commit traffic, and they also add processing cost, which is worth including in capacity plans.

Testing rebalance behavior before production

The best time to learn how your consumers behave during a rebalance is before customers depend on them. In a test environment that resembles production, deliberately cause each kind of change and watch what happens: stop one instance abruptly, stop one cleanly, start an extra instance, add partitions to a topic, restart the coordinator broker, and roll a deployment through the whole group. For each event, record how long processing paused, how many duplicates appeared, and whether the application recovered without manual help.

These exercises often reveal surprises. A callback that takes too long delays every rebalance. A cache that takes ten minutes to rebuild makes every partition move expensive. A shutdown hook that never runs leaves partitions unread until the session timeout. Finding these in a rehearsal is far cheaper than finding them during a busy afternoon. Repeat the exercise after major changes, such as a client library upgrade or a switch to a new assignment strategy or protocol.

Write down the group’s contract

For each important group, keep a short note describing its purpose, its owners, its expected number of members, its partition count, its timeouts and the reasons for them, its delivery guarantee, and what should happen if it stops for a long time. This small document answers most questions during an incident and makes handovers between teams smooth. It also encourages the healthy habit of choosing settings deliberately instead of leaving defaults in place by accident.

Other systems that use the same idea

Kafka Streams applications use consumer groups underneath, with a specialized assignor that also moves stateful tasks and their local stores. Kafka Connect uses its own group membership protocol for workers to share connectors and tasks. The same vocabulary of generations, heartbeats, and rebalances applies, so the diagnostic habits in this guide carry over.

Short, direct answers to the questions that come up most often.

No. It is a role played by one of your ordinary brokers, chosen for each group by hashing the group identifier onto a partition of the internal offsets topic.

In the classic protocol, no. A consumer chosen as group leader computes the split, and the coordinator only distributes it. In the newer server-driven protocol, the coordinator does compute the target split.

Most likely a rebalance happened. A member joined or left, a heartbeat or poll deadline was missed, or the coordinator changed. Check the logs for the trigger.

The session timeout detects a member that has stopped sending heartbeats, meaning a crash or lost connection. The poll interval detects a member that is alive but not calling poll often enough, meaning stuck processing.

Progress is saved only at commit time. Messages processed after the last commit but before the partition moved are read again by the new owner. Make processing repeatable to make duplicates harmless.

The commit carried an old generation number. The coordinator rejects it to stop a former owner from overwriting the current owner’s progress. This is expected behavior.

You can, but the extra consumers stay idle. They only serve as standbys that take over if another member leaves.

When instances restart often and you want restarts to be invisible to the group, especially for consumers with heavy local state. Choose a session timeout slightly longer than a typical restart.

It removes the pause for members whose partitions do not move, but moving partitions still pause briefly, and the process takes more than one round.

Saved positions of an idle group expire after the offsets retention period. With no bookmark left, the reset policy decides where to start, which may be the beginning of the retained data.

No. Groups are independent. Many groups failing together usually points at their shared coordinator broker or at the internal offsets topic.

It reduces rebalance disruption and moves control to the broker, which is attractive for large groups. Check your Kafka and client versions, migrate a low-risk group first, and verify behavior before moving critical ones.

The consumer group coordinator protocol is Kafka’s answer to a deceptively hard question: how can many workers share one stream of work, recover from failures, and remember their progress without stepping on each other? A broker acts as the coordinator for each group, found by hashing the group identifier onto the internal offsets topic. Members join, agree on a generation, receive their partitions, prove they are alive with heartbeats, and save their progress through the coordinator. When anything changes, a rebalance reorganizes ownership, and the design choices of eager, cooperative, static, or server-driven behavior decide how painful that reorganization feels.

Once you can name the trigger of every rebalance, read the state of a group, and connect symptoms to causes, consumer groups stop being mysterious. They become a predictable system that you can tune, monitor, and trust.