Rebalancing only makes sense once three ideas are clear: partitions, consumers, and groups.

Topics and partitions

Apache Kafka stores streams of events, such as orders, payments, clicks, or sensor readings. Events are grouped into named categories called topics. To make a topic big and fast, Kafka splits it into smaller logs called partitions. Each partition is an ordered, append-only list of events, and every event in it has a position number called an offset.

Consumers and consumer groups

Applications that read events are called consumers. Consumers that share the same group name form a consumer group. Kafka divides the partitions of the topic among the members of the group, so that each partition is read by exactly one member at a time. This is how a group shares the reading work and keeps events in order.

The assignment is not permanent

Which member reads which partition is called the assignment. It is a plan, and like any plan it must be updated when the situation changes. Members come and go, topics grow, and the plan must always cover every partition with exactly one owner. The act of updating the plan is the rebalance.

A rebalance is the process by which a consumer group redistributes topic partitions among its current members.

The simple definition

Whenever the set of members or the set of partitions changes, Kafka pauses the normal flow, gathers the group, and agrees on a new assignment. Once everyone knows their new partitions, reading continues, and each member picks up from the last saved bookmark of the partitions it now owns.

Why Kafka needs rebalancing

flowchart LR
  subgraph Before["Before"]
    A1["Consumer A: P0, P1, P2"]
    B1["Consumer B: P3, P4, P5"]
  end
  subgraph After["After Consumer C joins"]
    A2["Consumer A: P0, P1"]
    B2["Consumer B: P3, P4"]
    C2["Consumer C: P2, P5"]
  end
  Before --> After
        

Rebalances do not happen at random. A small list of events starts every one of them.

To start a rebalance, someone must first notice that the group has changed.

The group coordinator

For every consumer group, one Kafka server, called a broker, acts as the group coordinator. It keeps the list of active members, receives their regular signals, and decides when a rebalance must begin. The coordinator is like the manager of a team who keeps the attendance list.

Heartbeats

Each consumer sends a small signal, called a heartbeat, to the coordinator every few seconds. A heartbeat simply says “I am alive.” If the coordinator does not hear from a member for longer than the session timeout, it declares that member dead, removes it from the group, and starts a rebalance.

The processing time limit

Heartbeats alone are not enough, because a consumer could be alive yet stuck, for example waiting on a slow database. Kafka therefore also limits how long a consumer may go without asking for new events. If this limit is exceeded, the member is treated as failed and the group rebalances, even if heartbeats are still arriving.

Let us follow a rebalance from the first signal to the moment reading resumes.

sequenceDiagram
  participant M1 as Member 1 - Group Leader
  participant M2 as Member 2
  participant GC as Group Coordinator
  GC-->>M1: A rebalance is starting, please rejoin
  GC-->>M2: A rebalance is starting, please rejoin
  M1->>GC: Join group
  M2->>GC: Join group
  GC-->>M1: You are the leader, here are the members
  GC-->>M2: Joined, waiting for your assignment
  M1->>GC: Sync with the new plan
  M2->>GC: Sync and wait for plan
  GC-->>M1: Your partitions
  GC-->>M2: Your partitions
        

The group leader needs a rule to decide who gets which partitions. Kafka offers a few common rules.

Why “sticky” matters

Consumers often keep useful things in memory for the partitions they own, such as caches, running totals, or open connections. If a partition moves to another member, that member must rebuild all of it. A sticky strategy reduces these moves, so most members keep their partitions and their memory stays valuable.

Not all rebalances disturb the group equally. The style of the hand-over decides how much reading stops.

Eager rebalancing: stop the world

In the classic style, every member gives up all of its partitions at the start of the rebalance, and then everything is assigned again from scratch. While this happens, the entire group stops reading. Even members whose partitions will not change must pause. In a large group, or when rebalances repeat, these pauses add up and lag grows.

Cooperative rebalancing: only move what must move

The cooperative style is smarter. Members keep the partitions that are staying, and continue reading them without interruption. Only the partitions that need a new owner are released, and then handed over in a second, short round. Most of the group stays productive throughout. Because the change happens in two small steps, the total disruption is much smaller.

flowchart TD
  subgraph Eager["Eager style"]
    E1["All members release all partitions"] --> E2["Group stops reading"]
    E2 --> E3["New plan is drawn"]
    E3 --> E4["All members resume"]
  end
  subgraph Coop["Cooperative style"]
    C1["Plan is drawn; find partitions that must move"] --> C2["Only those partitions are released"]
    C2 --> C3["Released partitions go to new owners"]
    C3 --> C4["Other members never stopped reading"]
  end
        

Rebalancing keeps the group healthy, but it is not free. Here are the costs beginners should know.

Cost 1: Pauses in reading

While partitions are being reassigned, some or all members do not read. Events pile up in Kafka, and consumer lag, the gap between the newest event and the group’s bookmark, grows. Once reading resumes, the group must catch up.

Cost 2: Repeated work

A member may have processed some events but not yet saved its bookmark when the partition is taken away. The new owner starts from the last saved bookmark and reads those events again. This is why consumers must tolerate seeing an event twice.

Cost 3: Lost local state

Caches and running totals held in memory for a partition are useless to a new owner. It must rebuild them, which takes time and extra load on databases or on Kafka itself.

Cost 4: The rebalance storm

Sometimes a rebalance causes conditions that trigger another rebalance, and then another. For example, a rebalance makes members slow, which makes one exceed its time limit, which triggers a new rebalance. This chain is called a rebalance storm, and it can keep a group nearly useless for a long time.

flowchart LR
  A["Rebalance starts"] --> B["Members pause and rebuild state"]
  B --> C["Processing becomes slow"]
  C --> D["A member exceeds its time limit"]
  D --> E["Member is removed from the group"]
  E --> A
        

When rebalances happen too often, the cause is usually one of a few familiar culprits.

You cannot remove rebalancing, but a few simple habits make it gentle and rare.

A special feature exists to spare groups from the most avoidable rebalances: those caused by quick restarts.

The problem it solves

Normally, when a consumer restarts, Kafka sees the old member leave and a new member join. That means two rebalances for a change that lasts only a few seconds. In a large group updated one member at a time, this triggers a long chain of reshuffles.

How it works

With static membership, each consumer is given a fixed, unique name. If it disappears and returns with the same name within the session timeout, Kafka recognises it as the same member. It simply gets its previous partitions back, and the rest of the group is not disturbed. Only if the member fails to return in time does the normal rebalance begin.

The safety of a rebalance depends on how well members save their progress.

What is committed and what is not

As a member reads, it periodically saves its bookmark, called the committed offset, in Kafka. The bookmark says how far the group has finished. The member’s current reading position may be ahead of that saved bookmark. When a partition moves during a rebalance, the new owner starts from the committed bookmark, not from the old owner’s current position.

Two outcomes

flowchart LR
  A["Old owner reads offsets 100 to 149"] --> B["Committed bookmark: offset 130"]
  B --> C["Rebalance moves the partition"]
  C --> D["New owner starts at offset 130"]
  D --> E["Offsets 130 to 149 are read again"]
        

Reacting to the hand-over

Kafka lets a consumer be told just before it loses partitions and just after it receives new ones. Well-built applications use these moments. When partitions are about to be taken away, they finish their current work and commit the bookmark. When new partitions arrive, they prepare, for example by loading the state they need. This turns a rebalance from a messy interruption into an orderly hand-over.

You cannot fix what you cannot see. A few signals reveal the health of a group.

What to watch

Watch the number of rebalances over time. A calm group shows rebalances only when you deploy or scale. Watch how long each takes, because long rebalances mean long pauses. Watch consumer lag per partition, since lag that jumps and then recovers is a typical fingerprint of a rebalance. Watch the size of the group and the state of each member, such as whether the group is stable, preparing to rebalance, or waiting for members. In application logs, look for messages saying that a member was removed for exceeding its time limit, since these point straight at the cause.

Rebalancing is a bargain: automatic healing in exchange for occasional disruption.

Most rebalance problems come from the same handful of mistakes.

Best practices at a glance

Let us watch rebalancing during a busy day at an online shop.

flowchart TD
  S["Sale starts and traffic grows"] --> J["Two members join"]
  J --> R1["Cooperative rebalance moves a few partitions"]
  R1 --> O["One member becomes too slow"]
  O --> R2["Member removed and second rebalance"]
  R2 --> D["Survivors continue from committed bookmarks"]
  D --> F["Team fixes slow query and reduces batch size"]
  F --> C["Group is calm again"]
        

It is the process by which a consumer group redistributes the partitions of its topics among its current members after the group or the topic changes.

No. It is a normal, designed behaviour. It only becomes a problem when it happens too often, takes too long, or causes repeated work.

No. Producers may write to any partition. Only consumers in a group must coordinate to avoid reading the same partition at once.

One consumer in the group is chosen as the group leader and draws up the plan using the configured strategy, then sends it to the coordinator, which gives each member its share. In newer protocols, the broker performs much of this work.

No, events stay safely in Kafka. But events processed after the last committed bookmark may be read again, so consumers must tolerate repeats.

Common reasons are processing that exceeds the allowed gap between requests, members with unstable networks or machines, auto-scaling that changes the group size, or timeouts set too tight.

Eager makes every member give up all partitions and stops the whole group while a new plan is drawn. Cooperative moves only the partitions that must change owners, so the rest of the group keeps reading.

It gives each consumer a fixed name so that a quick restart is recognised as the same member returning. The member regains its partitions without any rebalance.

No. Only as many members as there are partitions can be active. The extra members sit idle, and each extra member still causes a rebalance when it joins.

Look at the group state reported by Kafka tools, at consumer log messages about revoked and assigned partitions, at a rise in consumer lag, and at the rebalance rate metrics of your monitoring system.

We began with a school where exam-paper bundles must be handed out again whenever a teacher joins or leaves. A Kafka rebalance is that reshuffle. Whenever a consumer joins, leaves, crashes, is judged too slow, or when the subscription or partitions change, the group coordinator starts a rebalance. Members stop, save their bookmarks, rejoin, let a group leader draw a new plan, receive their partitions, and resume from the last committed offsets. Rebalancing gives automatic fairness and healing, at the price of pauses, repeated work, and lost in-memory state. Cooperative strategies, static membership, small quick batches, graceful shutdowns, gentle deployments, and repeat-safe processing keep it calm. Watch the rebalance rate as carefully as you watch lag, and your groups will heal themselves without becoming a source of surprises.