A short refresher, so that the rest of the guide stands on firm ground.

Apache Kafka moves streams of events between programs. A program that writes events is a producer. A program that reads them is a consumer. The servers in the middle are brokers. Events are organised into named streams called topics, and every topic is split into partitions. A partition is an ordered list of messages, and each message in it has a position number called its offset.

Partitions exist so that work can be shared. One partition is read by one consumer at a time, so a topic with twelve partitions can keep up to twelve consumers busy at once. Each consumer remembers how far it has read by saving an offset, which is called committing the offset. If the consumer restarts, it resumes from the committed position.

A consumer group is a team of consumers that read one or more topics together, splitting the partitions between them.

All consumers that use the same group.id belong to the same consumer group. Kafka divides the topic’s partitions among the members so that every partition is read by exactly one of them. If there are more partitions than members, some members read several partitions. If there are more members than partitions, the extra members sit idle, waiting for a chance to help.

flowchart LR
  T["Topic orders
6 partitions"] --> P0["Partition 0"]
  T --> P1["Partition 1"]
  T --> P2["Partition 2"]
  T --> P3["Partition 3"]
  T --> P4["Partition 4"]
  T --> P5["Partition 5"]
  P0 --> C1["Consumer 1"]
  P1 --> C1
  P2 --> C2["Consumer 2"]
  P3 --> C2
  P4 --> C3["Consumer 3"]
  P5 --> C3

The group coordinator

One broker acts as the group coordinator for each group. It keeps the list of members, receives their heartbeats, stores their committed offsets, and decides when a rebalance must start. Consumers find it by asking any broker, which works it out from the cluster’s metadata.

Two groups never interfere

Different groups read the same topic independently. A payment topic might be read by a billing group and, separately, by a fraud-detection group. Each group has its own members, its own assignment and its own committed offsets. A rebalance in one group has no effect on the other.

A rebalance is the process of recalculating who owns which partition. It is not a fault. It is a normal part of life, but it has a price.

The coordinator starts a rebalance whenever something changes in a way that makes the current assignment wrong or unsafe. Understanding the triggers helps you predict how often rebalances will happen in your own system.

The life of a group

The coordinator tracks each group through a small set of states. A group is Empty when it has no members. When a member joins, it enters PreparingRebalance while the coordinator waits for members to report in. Then it moves to CompletingRebalance while the assignment is worked out and handed out. Once everyone has their share, it becomes Stable. Any new trigger sends it back to PreparingRebalance.

stateDiagram-v2
  [*] --> Empty
  Empty --> PreparingRebalance : first member joins
  PreparingRebalance --> CompletingRebalance : all members have joined
  CompletingRebalance --> Stable : assignment received
  Stable --> PreparingRebalance : membership or subscription change
  Stable --> Empty : last member leaves
  Empty --> [*] : group removed

Before comparing old and new styles, we need to see the basic conversation that every rebalance uses.

The classic protocol is built from a few requests. A JoinGroup request says “I want to be in the group, and here is what I subscribe to”. A SyncGroup request says “I am ready to receive my assignment”. A Heartbeat request says “I am still alive”. A LeaveGroup request says “I am leaving”. Interestingly, the coordinator does not compute the assignment itself. It picks one member to be the group leader, and that member’s code does the calculation.

sequenceDiagram
  participant M as Member
  participant L as Group leader
  participant C as Coordinator
  M->>C: JoinGroup with my subscription
  L->>C: JoinGroup with my subscription
  Note over C: Waits for all members or the rebalance timeout
  C-->>L: Here are all members and their subscriptions
  C-->>M: You are in, generation 5
  Note over L: Runs the assignor and builds the plan
  L->>C: SyncGroup with the full plan
  M->>C: SyncGroup with no plan
  C-->>L: Your assignment
  C-->>M: Your assignment

Following one consumer from start to shutdown shows where every rebalance concept fits.

Consider a single consumer instance, from the moment its process starts until it is stopped. Every stage below touches something we have discussed, and knowing the order helps you place problems when they appear.

An assignor is the piece of code that turns “these members and these partitions” into “this member gets these partitions”. Kafka ships with several.

The setting partition.assignment.strategy lists the assignors a consumer supports. The coordinator picks one that every member of the group supports. The four built-in choices differ in how they divide the work and in how they behave when the group changes.

What the defaults are

Since Kafka 3.0, the default list is the range assignor first and the cooperative sticky assignor second. With a fresh group this means Range is used, because it is listed first and everyone supports it. The second entry is there to make a smooth upgrade possible: removing the first entry from all members in a rolling restart moves the group to cooperative behaviour without any special steps. Older Kafka versions used only the range assignor by default.

To appreciate the cooperative approach, we first need to feel the pain of the eager one.

In eager rebalancing, whenever a rebalance starts, every member gives up every partition it owns. It stops fetching, runs its cleanup code, commits its offsets, and sends JoinGroup with nothing in hand. Only after the new plan arrives does it receive partitions again, possibly the same ones it had a moment ago. Between the first revocation and the arrival of the plan, the entire group processes nothing.

flowchart TD
  A["Rebalance triggered"] --> B["Every member revokes all its partitions"]
  B --> C["Every member sends JoinGroup"]
  C --> D["Coordinator waits for all members"]
  D --> E["Leader computes the new plan"]
  E --> F["Every member receives its assignment"]
  F --> G["Every member resumes reading"]

What it costs

The last point deserves attention. With forty instances restarting in sequence, a group can go through dozens of rebalances during one deployment. If each takes ten seconds of complete silence, that is several minutes of lost processing spread across the release, even though the deployment itself was “rolling” and supposedly gentle.

Stickiness answers a simple question: when the group changes, why should a consumer lose partitions it could have kept?

A sticky assignor has two goals, in this order. First, keep the assignment balanced, meaning the number of partitions per consumer differs by at most one. Second, subject to that, keep the assignment as similar as possible to the previous one. In other words, it moves the fewest partitions that are needed to reach a fair split.

A worked example

Take six partitions and three consumers. Each consumer owns two partitions. Now a fourth consumer joins. With four consumers, a balanced split gives two consumers two partitions each and two consumers one partition each, since six divided by four is one and a half. The question is how many partitions must move to reach that.

The sticky answer changes exactly one thing: partition P5 moves from Consumer 3 to the new Consumer 4. Everyone else keeps precisely what they had. Other methods can reshuffle many partitions for no good reason, because they build each assignment from scratch without remembering the last one.

Why stickiness matters

Cooperative rebalancing changes the protocol so that members give up only what they must, and keep working on everything else.

The idea, introduced in Kafka 2.4 by a proposal known as KIP-429, is called incremental cooperative rebalancing. Instead of revoking everything at the start, each member keeps its partitions when the rebalance begins. It reports to the group leader exactly which partitions it currently owns. The leader then plans the new assignment and works out which partitions must move from one owner to another.

Here comes the clever rule: a partition is never handed directly from one owner to another in the same round. If a partition must move, the leader first takes it away from the old owner and leaves it unowned for that round. The old owner sees that it lost the partition, cleans up and commits, and then asks to rejoin. In a second round, the free partition is given to its new owner. Two rounds, but only the moved partitions are ever idle.

Why the safety rule matters

Why not hand the partition straight to the new owner in one round? Because two consumers might then read and process the same partition at the same instant: the old one, which has not yet noticed, and the new one. That breaks the rule that each partition has exactly one owner. By forcing a gap, cooperative rebalancing guarantees that the old owner has let go before the new owner starts. The price is one extra round, and that price is paid only by the partitions that move.

Let us follow the six-partition example again, this time with the cooperative protocol, and count exactly who stops and for how long.

The group starts with three consumers, each owning two partitions: Consumer 1 has P0 and P1, Consumer 2 has P2 and P3, and Consumer 3 has P4 and P5. A fourth consumer starts up and joins. The coordinator begins a rebalance, and the group enters its first round.

sequenceDiagram
  participant C3 as Consumer 3
  participant C4 as Consumer 4
  participant L as Leader and coordinator
  Note over C3,C4: Round 1
  C3->>L: JoinGroup, I own P4 and P5
  C4->>L: JoinGroup, I own nothing
  L-->>C3: Assignment is P4 only, give up P5
  L-->>C4: Assignment is empty for now
  Note over C3: Revoke P5, commit its offset, rejoin
  Note over C3,C4: Round 2
  C3->>L: JoinGroup, I own P4
  C4->>L: JoinGroup, I own nothing
  L-->>C4: Assignment is P5
  Note over C4: Start reading P5

Counting the cost

Assume each round takes about three seconds to complete. With the eager protocol, all six partitions are idle for roughly one round plus cleanup, say four seconds, giving 24 partition-seconds of lost processing. With the cooperative protocol, only P5 is idle, for about two rounds, six seconds, giving 6 partition-seconds. The moved partition waits a little longer, but the total loss is a quarter as large, and the five other partitions never stopped at all.

Members do not only join. When one leaves, its partitions must find new owners. The cooperative approach handles this with fewer rounds, since nobody has to give anything up first.

Suppose Consumer 4 owns P5 and now leaves. There are two ways this can happen, and they differ mainly in how long it takes to notice.

Why one round is enough

When the owner of P5 is gone, P5 has no owner. The rule “never hand over an owned partition in the same round” does not apply, because there is nothing to take away. The leader computes the fair split for three consumers, sees that P5 is free, and assigns it directly. The remaining consumers keep what they already have. The result: one round, no revocations, and no one but the departed member’s partitions is affected.

What survivors experience

For a survivor, the rebalance is almost invisible. It sends JoinGroup with its owned partitions, receives an assignment that includes everything it already had plus, possibly, a new partition, and continues. Its assign callback fires only for the newly added partition. Under the eager protocol, in contrast, every survivor would have stopped, revoked everything, and received partitions again.

Now that the behaviour is clear, let us look at how the group leader actually computes a sticky, cooperative plan.

The work happens in the leader’s process, inside the assignor code. It runs each time a rebalance completes its join phase. The input is the list of members, their subscriptions, the partition lists of the subscribed topics, and, for the cooperative case, what each member says it currently owns.

A second example with a remainder

Ten partitions and four consumers give a quota of two each, with two consumers taking three, since ten divided by four is two remainder two. Suppose the current owners hold 3, 3, 2 and 2 partitions. Already balanced, so nothing needs to move. Now suppose the current split is 5, 3, 2, 0 because of earlier changes. The first consumer is over its quota of three by two, and the last is under its quota of two by two. The leader moves exactly two partitions from the first to the last, and keeps all the rest where they are.

A rebalance changes who owns what. To stop confused or slow members from acting on old ownership, Kafka numbers every round.

Each successful rebalance produces a new generation, a counter that goes up by one. Each member also receives a member id when it first joins. Every important request the member sends, including heartbeats and offset commits, carries its member id and the generation it believes is current. The coordinator compares this with the truth and rejects requests that do not match.

Revoked versus lost

Kafka distinguishes two situations. A partition is revoked in an orderly way: the member still belongs to the group, so it can safely commit its offsets first. A partition is lost when the member has already been removed, for example after a session timeout. In that case the partition may already belong to someone else, and committing offsets would be wrong, since the new owner is reading from the last good commit.

The cooperative design added a separate callback for this second case, named onPartitionsLost. Its default behaviour is to call the revoke callback, which keeps older code working. But good code separates the two: commit in revoke, and only clean up local state, without committing, in lost.

The poll call is the engine of a consumer. Understanding what it does explains many rebalance surprises.

A Kafka consumer is not a background service that pushes records to you. It is a library driven by your own calls to poll. Inside that call, the client fetches records, checks whether a rebalance is needed, sends and receives group protocol messages, and calls your listener callbacks. Only one thread, the one calling poll, does this work. A second, hidden thread does nothing but send heartbeats.

What this means for rebalances

Cooperative details that matter

Under the cooperative protocol, poll can keep returning records for the partitions you retain while the rebalance is in progress. Partitions that were paused by your code stay paused if you keep them, because the consumer never gave them up. Under the eager protocol, in contrast, pause state was lost on every rebalance, since every partition was revoked and reassigned. Code that pauses partitions for back-pressure should therefore be reviewed when the protocol changes.

Your application can register a listener that is called around rebalances. Cooperative rebalancing changes what the callbacks mean, and code that ignores this can break.

The listener has three callbacks. One runs when partitions are taken away, one when partitions are added, and one when they are lost. In the eager protocol, these are simple: the first callback receives everything you owned, and the second receives everything you now own. In the cooperative protocol, they receive only the difference.

Common mistakes with listeners

A safe way to think about it

Keep a simple mental model: your set of owned partitions is a set. The revoke callback removes items from it. The assign callback adds items to it. Anything you keep per partition, such as a cache, a counter or a buffer, should be attached to that set and updated exactly when the set changes. If your code follows that model, it works correctly under both protocols, and you can switch between them without rewriting anything.

Handovers are exactly where duplicates and gaps sneak in. A careful commit strategy keeps them under control.

When a partition moves, the new owner starts reading from the last committed offset. Everything the old owner processed after that commit will be read again. So the closer the last commit is to the old owner’s real position, the fewer duplicates appear. This is why the revoke callback is the right place to commit: it runs just before the partition leaves, while the member still has the right to commit.

Automatic versus manual commits

Exactly-once processing

When a consumer reads, transforms and writes results back to Kafka, transactions can make the output and the offset commit one atomic step. For this to be safe when partitions move, the group’s generation information must be included with the offset commit, so that the broker can refuse a commit from a member that no longer owns the partition. With the cooperative protocol, this fencing is what prevents a slow old owner from committing work after the partition has been given away.

A few time limits decide how quickly a failure is noticed and how easily healthy consumers are mistaken for dead ones.

Two clocks, two failure types

The consumer runs a background thread that sends heartbeats, separate from the thread that processes records. The session timeout watches the background thread: if the whole process freezes or the network is cut, heartbeats stop. The poll interval watches the processing thread: if your code takes too long between polls, the consumer is considered stuck, even though heartbeats are still flowing. Either clock running out causes the member to be removed and a rebalance to begin.

How the settings interact with rebalancing

Cooperative rebalancing makes rebalances cheap. Static membership tries to make many of them disappear.

Normally, a restarting consumer looks like a stranger: it leaves, or disappears, and a new member with a new member id joins. That triggers two rebalances. With static membership, each consumer is given a stable name through the setting group.instance.id. When an instance with a known name restarts and rejoins within the session timeout, the coordinator recognises it, hands back its previous assignment, and does not start a rebalance at all.

How it combines with cooperative rebalancing

The two features cover different situations. During a planned restart that finishes inside the session timeout, static membership avoids any rebalance. During an unplanned, longer outage, or when capacity is truly added or removed, a rebalance is unavoidable, and the cooperative protocol makes it as gentle as possible. Used together, they turn most deployments into non-events and leave only the truly necessary movements.

Most consumer fleets today run in containers. How the platform stops and starts instances decides how gentle each rebalance is.

Container platforms start and stop instances in an orderly way, but their defaults do not know about Kafka groups. Three details make the biggest difference: how the instance is told to stop, how much time it is given, and how quickly replacements start.

A rollout compared

Turning the feature on is one setting. Getting the choice right takes a little more thought.

The consumer setting partition.assignment.strategy takes a list of assignor names in order of preference. The coordinator picks an assignor that every member supports. Among the supported ones, it uses the members’ order of preference to break ties. This makes it possible to run mixed lists safely during an upgrade, and it explains the default list in recent versions.

When not to use a sticky assignor

Sticky assignors do not promise that partition 3 of topic A and partition 3 of topic B end up on the same consumer. Applications that join two topics by key, relying on the range assignor’s alignment, will break silently if switched. Frameworks such as Kafka Streams use their own assignment logic that keeps co-partitioned tasks together, so this concern mostly applies to hand-written consumers.

You cannot flip a switch on a running group, because old and new members would disagree on the rules. The safe path takes two rolling restarts.

flowchart TD
  A["Group runs an eager assignor"] --> B["Update client library and rebalance listener code"]
  B --> C["First rolling restart: list old and cooperative assignors, old first"]
  C --> D["Group keeps using the eager protocol"]
  D --> E["Second rolling restart: remove the old assignor"]
  E --> F["Group switches to the cooperative protocol"]

Watch for these problems

Definitions are useful, but the behaviour really sinks in when you follow numbers through a bad day. These four stories are illustrations built on realistic settings.

Story 1: the Friday deployment

A service runs as 40 instances reading a topic with 120 partitions, so each instance owns about three partitions. The group uses the range assignor, which is eager. The deployment restarts instances one at a time. Each restart causes a rebalance when the old instance leaves and another when the new one joins, so about 80 rebalances in total. Suppose each one takes about eight seconds of complete silence for all 120 partitions.

That is roughly 640 seconds, more than ten minutes, during which nothing is processed, spread across the release in small bursts. Backlogs build every time and are worked off afterwards, but downstream systems see repeated delays. After switching to the cooperative sticky assignor, each event moves only about three partitions, and the other 117 keep running. With static membership added, and each restart shorter than the session timeout, most of the 80 rebalances disappear entirely.

Story 2: the slow member that held everyone up

One consumer in a group of 30 takes on an unusually large batch and spends four minutes processing it. The poll interval is five minutes, so it is not considered dead. At that moment another instance restarts, triggering an eager rebalance. Every member must revoke and rejoin, but the slow one cannot rejoin until it returns to poll. The other 29 members, having already revoked everything, sit waiting for up to the full poll interval.

Result: nearly four minutes of complete silence across the whole group, caused by one slow member. With the cooperative protocol, the other 29 keep processing the partitions they retain while the slow member finishes. Only the partitions that need to move wait for it, and they wait because their previous owner must release them, which is exactly as it should be.

Story 3: the listener that forgot what it kept

A team keeps an in-memory cache of recently seen message ids per partition, used to drop duplicates. Their revoke callback clears the whole cache, and their assign callback rebuilds it from scratch for the partitions it receives. Under the eager protocol this works, since every rebalance revokes and reassigns everything.

After switching to cooperative, a rebalance revokes one partition but the callback still clears the entire cache. The assign callback then receives only the one new partition and rebuilds only that. The retained partitions keep running with empty caches, and nothing rebuilds them. Duplicates that used to be dropped now pass through for hours. The fix is small: remove only the revoked partitions’ entries in the revoke callback.

Story 4: the join that quietly stopped matching

Two topics, orders and payments, each have 12 partitions, keyed by customer, so a given customer lands in the same partition number in both. A hand-written consumer joins them in memory. With the range assignor, partition 3 of both topics always lands on the same consumer, so the join works. After the team moves to a sticky assignor, orders partition 3 stays on Consumer B, while payments partition 3 ends up on Consumer D after a scale-out. Now the two halves of the join sit on different consumers, and matches are missed.

The lesson is that sticky assignors do not promise co-partitioning. Applications that need it should use a framework that handles it, such as Kafka Streams, or keep an assignor that provides the alignment.

Two of Kafka’s own frameworks use the same core idea, adapted to their own needs.

Kafka Streams

A Kafka Streams application divides its work into tasks, each tied to a partition (or a set of aligned partitions), and each task usually owns a local state store such as a table of running counts. Moving a task to another instance means rebuilding its state store from a log, which can take minutes or hours for a large store. This makes stickiness and gentle handover especially valuable.

Streams has its own assignor that keeps related partitions together and prefers to put tasks back where their state already lives. Recent versions also use a cooperative approach, and add a feature called warm-up replicas. Before a task moves to a new instance, the new instance quietly builds a copy of the state in the background. Once it has caught up, a follow-up rebalance, called a probing rebalance, moves the task over. The old owner keeps serving until then, so users see no pause. Settings such as the number of standby replicas, the acceptable recovery lag, the maximum number of warm-up replicas and the probing interval control this behaviour.

Kafka Connect

Kafka Connect runs connectors and their tasks across a cluster of workers. Early versions used stop-the-world rebalancing, so adding a worker paused every connector in the cluster. A later improvement, incremental cooperative rebalancing for Connect, lets running tasks continue during a rebalance and moves only the ones that must move. It also adds a grace period, controlled by a setting for the scheduled rebalance delay, during which the tasks of a worker that has just disappeared are not immediately handed out. If the worker comes back quickly, for example after a restart, it simply picks up its tasks again.

Cooperative rebalancing softened the classic protocol. A newer protocol redesigns the conversation so that most of the pain never appears.

Recent Kafka releases, with general availability in Kafka 4.0, introduce a new consumer group protocol (proposal number KIP-848). It is selected on the client with the setting group.protocol, and the older behaviour is still available under the name classic. The new protocol moves work from the clients to the brokers and removes the group-wide barrier.

How it works in outline

Rebalancing costs time and effort. Understanding where the cost comes from tells you which knobs are worth turning.

The cost of a pause

A pause is not only the seconds lost. While a partition is idle, messages keep arriving, so a backlog builds. After the pause, consumers must work off that backlog on top of the normal flow. The recovery time depends on how much spare capacity they have.

The arithmetic is simple. A ten-second pause at 5,000 messages per second creates a backlog of 50,000. If the consumers can process 7,500 per second, they have 2,500 per second to spare, so it takes 20 seconds to catch up. If they only had 500 per second to spare, it would take 100 seconds. Groups that run close to capacity feel every pause much longer than the pause itself, which is one more reason to keep headroom.

Where the rounds spend their time

Trade-offs of two rounds

The cooperative protocol sends more requests than the eager one, because a move needs a second round. It also makes moved partitions wait somewhat longer. In exchange, all other partitions run without interruption. For most groups, this is a clear win. For a very small group with no state, where a rebalance takes a fraction of a second, the difference may be too small to notice.

What can and cannot go wrong for your data when partitions change hands? The answer depends on the two rules the protocol enforces.

Two rules govern every rebalance. First, at most one owner at a time for each partition in a group. The cooperative design keeps this by always leaving a moved partition unowned for a round. Second, the new owner starts from the last committed offset. Together, these give you ordered processing per partition and at-least-once delivery, with duplicates limited to the work done since the last commit.

How the cooperative protocol changes the duplicate window

In the eager protocol, every partition is handed over, so every partition has a possible duplicate window. In the cooperative protocol, only moved partitions do. The window is the same size for a moved partition, but the number of affected partitions is smaller. That lowers the total duplicates without changing the guarantee in any one place.

Anyone who can join a group can disturb it. Group membership deserves the same care as topic access.

To join a group and commit offsets, a client needs permission to read the topic and permission to use the group itself. Because every join, leave and even a badly behaved member can trigger a rebalance, a careless access policy lets an accidental or malicious client interrupt everyone in the group.

Rebalance storms as an availability risk

A buggy client that joins and leaves in a loop, or a mis-scaled autoscaler that adds and removes instances every minute, keeps a group permanently rebalancing. Even with cooperative rebalancing, constant motion prevents the group from settling. Monitoring the rebalance rate, and setting alerts on it, is as much a reliability control as a security one.

You cannot tune what you cannot see. A handful of consumer-side and group-side signals tell you whether rebalancing is healthy.

Reading the logs

Coordinator logs on the broker record why a rebalance began: a new member, a member leaving, a member failing to heartbeat, or a subscription change. Client logs include messages when a consumer leaves because the time between polls exceeded the limit. Reading these two sources together answers the most important question: what triggered this rebalance?

Teams that run large consumer fleets tend to reach cooperative rebalancing for the same reasons, whatever their industry.

A few habits make cooperative rebalancing shine. A few shortcuts quietly undo its benefits.

Patterns that work

Anti-patterns to avoid

A checklist for design reviews, and a table of mistakes that keep returning.

Some ideas about rebalancing sound sensible and are wrong. These are the ones that cause the most confusion.

Gotchas worth remembering

Let us apply what we have learned to three different workloads. The method matters more than the exact numbers.

Ask four questions of any consumer group. How much state does each consumer hold? How often do instances change? How long can a partition safely sit idle? And how heavy is the processing per batch? The answers point to different choices.

Short answers to the questions that come up again and again.

Move only what needs to move, and never stop what does not need to stop.

A rebalance is Kafka’s way of sharing partitions again when a group changes. The old, eager way makes everyone revoke everything and wait, even when most partitions would end up exactly where they were. Sticky assignment chooses the plan that changes the least. Cooperative rebalancing carries out that plan in stages, taking a moved partition from its old owner first and giving it to the new one in a second round. Combined, they turn a group-wide outage into a small, local handover. Add static membership, careful listeners and sensible timeouts, and most deployments stop being events at all.