Rebalancing is how a group of workers redistributes shared work whenever the group or the work changes.

The Simple Definition

A rebalance is the process by which a group of cooperating servers agrees on who is responsible for which piece of work. It happens every time a server joins, a server leaves, or the list of work changes. In Kafka Connect, the pieces of work are called connectors and tasks, and the servers are called workers. During a rebalance, the group figures out a new fair split and everyone moves to it.

Rebalances are normal and healthy. They are the reason a Connect cluster can survive the loss of a server, grow when you add servers, and pick up newly created connectors automatically. The problem is not that rebalances exist, but how disruptive they are. A poorly behaved rebalance can pause every data pipeline in the cluster, cause duplicate records, delay reports, and trigger alarms across the company.

Why This Matters in Real Life

Connect clusters often run business-critical pipelines: moving orders into a warehouse, syncing customer changes into search, or archiving events to cheap storage. Teams also deploy new versions, scale up for busy seasons, and lose servers to hardware failures. Each of those events causes rebalancing. If each rebalance stops all pipelines for even a minute, a rolling deployment of ten servers can add up to ten minutes of stalled data. The protocol used for rebalancing decides whether that happens.

A friendly tour of the words used in every later chapter.

A Picture of the Hierarchy

A Connect cluster contains workers. Workers host connectors. Each connector creates tasks. Tasks are the hardest workers, doing the actual reading and writing. When we say “the cluster is rebalancing,” we mean “the cluster is reshuffling connectors and tasks among workers.”

flowchart TD
  CL["Connect cluster"] --> W1["Worker A"]
  CL --> W2["Worker B"]
  CL --> W3["Worker C"]
  W1 --> C1["Connector orders"]
  C1 --> T1["Task 0"]
  C1 --> T2["Task 1"]
  W2 --> C2["Connector customers"]
  C2 --> T3["Task 0"]
  W3 --> T4["Task 2 of orders"]
        

There is no separate manager. The workers organize themselves using Kafka itself.

Group Membership Through Kafka

In distributed mode, every worker is configured with the same group name. Each worker contacts a Kafka broker acting as the group coordinator and announces itself as a member. The coordinator keeps a list of members and expects each one to send small “I’m alive” messages called heartbeats. When the list of members changes, the coordinator tells everyone that a rebalance is needed, and the members start the joining process again.

Electing a Leader

During each rebalance, the coordinator picks one member as the leader. This is not a permanent title; a different worker may be leader next time. The leader receives the list of all members and information about what each one currently holds, and it calculates the new assignment for everybody. The other workers wait for their part of the plan. The plan is delivered through the coordinator, and each worker starts or stops its pieces accordingly.

Where the Truth Is Stored

The leader needs to know which connectors exist and how many tasks each has. That knowledge lives in the internal topics. A configuration topic stores every connector’s settings and every task’s settings. An offsets topic stores how far source connectors have progressed. A status topic stores whether each connector and task is running, paused, or failed. Because these are ordinary Kafka topics, they are durable and replicated, and any worker can read them to catch up.

sequenceDiagram
  participant W as Worker
  participant GC as Group Coordinator
  participant L as Leader Worker
  participant CT as Config Topic
  W->>GC: Join group
  GC-->>L: You are leader, here are all members
  L->>CT: Read current connectors and tasks
  L->>L: Calculate assignment
  L->>GC: Send assignment plan
  GC-->>W: Your assignment
  W->>W: Start or stop connectors and tasks
        

Every rebalance has a cause. Learning the common causes is the first step to calming a noisy cluster.

The original protocol, which stops everything on every change.

How Eager Rebalancing Works

The word eager describes a protocol that acts immediately and completely. When any rebalance begins, every worker stops every connector and every task it owns. Only when all workers have released everything do they rejoin the group. The leader then computes a completely fresh assignment as if nothing had existed before, and workers start whatever they are given. From the first stop to the last start, no data moves anywhere in the cluster.

sequenceDiagram
  participant A as Worker A
  participant B as Worker B
  participant N as New Worker
  Note over A,B: Both running tasks normally
  N->>A: New worker joins, rebalance begins
  A->>A: Stop ALL tasks
  B->>B: Stop ALL tasks
  Note over A,N: No data moves anywhere
  A->>B: Everyone rejoins, leader assigns from scratch
  A->>A: Start new assignment
  B->>B: Start new assignment
  N->>N: Start new assignment
        

Why It Was Designed This Way

The eager protocol is simple and safe. Because everyone releases everything before anyone starts anything, there is never a moment when two workers both think they own the same task. Simplicity was a reasonable choice when Connect clusters were small and rebalances were rare. As clusters grew and pipelines became business-critical, the cost of that simplicity became too high.

Four everyday situations where stop-the-world rebalancing hurt real teams.

The Hidden Costs Beyond the Pause

Two words, two ideas: move only what changes, and let workers hand work over politely.

Idea One: Incremental

Incremental means “in small steps.” Instead of throwing everything away and starting over, the cluster keeps its current assignment and adjusts it. If a new worker joins, only a few pieces of work move to it. Everything else stays exactly where it is, running without interruption. The size of the disruption now matches the size of the change.

Idea Two: Cooperative

Cooperative means “workers work together.” Instead of every worker dropping everything, workers keep running during the rebalance and give up ownership of specific pieces only when the leader asks them to. A piece of work that must move is first released by its old owner, and only in a following round is it granted to its new owner. Because the release comes first, no piece of work is ever owned by two workers at the same time.

The Key Trick: Two Rounds for Anything That Moves

sequenceDiagram
  participant A as Worker A
  participant B as Worker B
  participant N as New Worker
  participant L as Leader
  Note over A,B: Both running tasks normally
  N->>L: New worker joins
  L->>A: Round one: revoke Task 3 only
  L->>B: Round one: keep everything
  A->>A: Stop Task 3, keep others running
  B->>B: Keep running everything
  A->>L: Rejoin holding remaining tasks
  L->>N: Round two: assign Task 3
  N->>N: Start Task 3
        

Following one common event through the cooperative protocol, moment by moment.

Suppose a cluster has two workers. Worker A holds connector Orders and two of its tasks. Worker B holds connector Customers and two of its tasks. Each worker is running smoothly. Now a third worker, called C, starts up.

A departure can be clean or chaotic. The protocol treats the two cases differently.

Case One: A Graceful Departure

When a worker is shut down normally, it stops its own connectors and tasks first and tells the group it is leaving. Because the work has been released deliberately, the cluster knows exactly what became unowned and can reassign it. The leader spreads the released pieces among the remaining workers in a single assignment round. This is the tidy path.

Case Two: A Sudden Disappearance

When a worker crashes or loses its network, it cannot announce anything. The cluster only finds out when heartbeats stop and the coordinator’s patience runs out. After that, the group rebalances. The pieces of work held by the missing worker are now called lost assignments: nobody stopped them politely, but they are clearly no longer running. Since the worker might still be alive but merely unreachable, the leader has to be careful about how quickly it hands those pieces to others.

flowchart TD
  D["A worker stops responding"] --> H{"Did it announce leaving?"}
  H -->|"Yes"| G["Graceful: work released, reassigned at once"]
  H -->|"No"| T["Heartbeats stop, timeout expires"]
  T --> LR["Group rebalances, work marked as lost"]
  LR --> W["Leader waits for the scheduled delay"]
  W --> R{"Did the worker return in time?"}
  R -->|"Yes"| B["Worker gets its old work back"]
  R -->|"No"| X["Work is reassigned to remaining workers"]
        

Why the Difference Matters

A built-in grace period that prevents needless shuffling when a worker returns quickly.

The Problem It Solves

Think about a routine restart. A worker leaves, and thirty seconds later a fresh copy of it starts again. If the leader instantly reassigned the departed worker’s tasks to the others, the cluster would move tasks away, and then, when the worker came back, move tasks back again. That is two rounds of unnecessary movement for something that was over in half a minute.

The Solution: A Grace Period

The protocol includes a configurable waiting time, the scheduled rebalance delay. When a worker disappears and its work becomes lost, the leader does not reassign that work immediately. It schedules another rebalance for later, after the delay, and lets the remaining workers keep running what they already have. If the missing worker comes back before the delay ends, it simply receives its previous work again, and nothing else in the cluster has moved. If the delay ends and the worker has not returned, the leader assigns the lost work to the remaining workers.

Choosing the Delay

The delay is a trade-off between two costs. A long delay is gentle on rolling restarts, because workers can leave and come back without disturbing the cluster. But if a worker is truly dead, its tasks sit idle for the whole delay before anyone takes over, which means data pipelines stay stopped for that long. A short delay recovers quickly from real failures but causes more unnecessary movement during routine restarts. Setting the delay to zero disables the grace period entirely and reassigns lost work immediately.

The leader’s assignment logic follows a few simple, sensible rules.

What the Leader Knows

At the start of each rebalance, every worker tells the leader three things: which workers exist, what each currently holds, and how up to date its view of the shared configuration is. The leader also reads the configuration topic to learn the full list of connectors and tasks that should exist. From this it can compare “what should exist” with “what is running where.”

The Rules It Follows

One setting decides which rebalancing language the workers speak, and how they upgrade together.

Three Choices

Each worker is configured with a protocol setting that names the rebalancing behavior it is willing to use. Because a cluster must agree on one behavior, the group picks the best option that every member supports. This makes safe upgrades possible: workers can be taught the new language before anyone is forced to use it.

flowchart TD
  S["Each worker declares its protocol"] --> Q{"Does every worker support cooperative?"}
  Q -->|"No, at least one is eager only"| E["Group uses eager"]
  Q -->|"Yes"| C["Group uses cooperative"]
  C --> K{"Does every worker support sessioned?"}
  K -->|"Yes"| SS["Cooperative plus signed internal requests"]
  K -->|"No"| CC["Cooperative without signed requests"]
        

What “Sessioned” Adds

In distributed mode, a worker that receives a request to change a connector may need to forward it to the worker that owns the connector, which uses the cluster’s internal web interface. Without protection, an outsider who could reach that interface might impersonate a worker. The sessioned protocol has the leader create a shared secret key that is rotated periodically and distributed to members through the rebalance itself. Internal requests carry a signature made with that key, so workers can verify that a request truly came from a fellow member.

A rolling upgrade recipe that never mixes languages in a dangerous way.

Suppose you run an older cluster using eager rebalancing and you want to move to cooperative rebalancing without downtime. The key fact is that the group always falls back to the least capable member. That means you cannot simply switch workers one by one to a cooperative-only setting; a half-upgraded group would contain members that cannot understand one another. The safe path uses the compatible setting as a bridge.

A source of endless confusion: Connect contains two separate rebalancing layers that can interact.

Layer One: Workers Sharing Connectors and Tasks

This is the layer this tutorial has described so far. The Connect workers form a group, and the unit being divided is the connector or task. The protocol we studied, eager or cooperative, applies to this layer.

Layer Two: Sink Tasks Sharing Topic Partitions

A sink connector reads from Kafka topics. Each of its tasks contains a regular Kafka consumer, and all the tasks of one sink connector belong to a single consumer group. That consumer group divides the topic partitions among the tasks, using the ordinary consumer rebalancing you may already know. This is a completely separate group with a separate coordinator conversation and a separate assignment strategy.

flowchart TD
  subgraph L1["Layer one: Connect worker group"]
    W1["Worker A"]
    W2["Worker B"]
  end
  subgraph L2["Layer two: sink connector consumer group"]
    T1["Sink task 0"]
    T2["Sink task 1"]
  end
  W1 --> T1
  W2 --> T2
  T1 --> P["Topic partitions"]
  T2 --> P
        

How the Layers Interact

When the worker-level rebalance stops and starts a sink task, that task leaves or joins its consumer group, which triggers a consumer group rebalance for that connector. With cooperative worker rebalancing, fewer sink tasks stop, so fewer consumer group rebalances occur. But the consumer layer has its own strategy setting. If sink tasks use an eager consumer assignment strategy, then each time one task joins or leaves, all tasks of that connector pause reading briefly, even when the worker layer was gentle. Modern consumers also offer a cooperative assignment strategy, which softens this layer in the same spirit.

What happens to the data when a task is stopped and started somewhere else.

Source Tasks: Progress Lives in Kafka

A source task reads from an outside system and writes to Kafka. It periodically reports how far it has read, for example a position in a database change log. Those reports are saved in the offsets topic. When a task is stopped and restarted on another worker, the new copy reads the saved position and resumes from there. Because the saved position is only updated periodically, the new task may re-read a few records that the old one already sent but had not yet recorded. This produces a small number of duplicates, which is part of the standard at-least-once behavior.

Sink Tasks: Progress Lives in the Consumer Group

A sink task reads from Kafka and writes to an outside system. Its progress is stored as a committed position in the connector’s consumer group. When a sink task is stopped, it should first finish writing what it has and commit its position. When another task takes over its partitions, it starts from the committed position. If the outgoing writes were done but the commit did not happen yet, the new task repeats those records. Again, a few duplicates are possible unless the outside system can safely ignore repeats.

Why Cooperative Rebalancing Helps Here Too

Every unnecessary stop is a chance for duplicates and delay. Because the cooperative protocol stops far fewer tasks, it produces far fewer duplicate records and less lag, even though the semantics of each individual move remain the same. The protocol reduces how often moves happen; it does not change what a single move means.

Cooperative rebalancing depends on tasks stopping cleanly and quickly. Sometimes they do not.

The Stop Request

When the leader revokes a task, the worker asks the task to stop. Well-written tasks notice the request, finish the current small piece of work, close their connections, and report that they have stopped. The worker waits a limited time for this to happen, controlled by a shutdown timeout. If the task does not stop within that time, the worker gives up waiting and continues, and the task may remain half-alive in the background.

What Is a Zombie Task?

A zombie task is one that the cluster believes is stopped but which is actually still running, perhaps stuck in a slow call to an outside system. If the cluster then starts a fresh copy of the same task on another worker, two copies exist at once, and both may read from the same source or write to the same destination. That can create duplicates or, in worse cases, conflicting writes.

sequenceDiagram
  participant L as Leader
  participant A as Worker A
  participant B as Worker B
  L->>A: Revoke Task 5
  A->>A: Ask Task 5 to stop
  Note over A: Task is stuck in a slow external call
  A->>L: Shutdown timeout reached, moving on
  L->>B: Assign Task 5
  B->>B: Start Task 5
  Note over A,B: Two copies may now be running, the old one is a zombie
        

How to Prevent Zombies

Seven clocks, each answering a different question about patience.

Most rebalancing trouble in production comes from misunderstood timing settings. Each timeout answers one specific question. Keeping them apart in your mind is the fastest way to reason about strange behavior.

How They Fit Together

There is no universal best value. The right numbers depend on how your workers live and die.

Start By Asking How Workers Restart

Before touching any timeout, describe how workers actually come and go in your environment. How long does a planned restart take from stop to healthy? How often are workers replaced? How quickly do you need a real failure to be repaired? The answers decide almost every setting.

A Reasoning Recipe Instead of Magic Numbers

Why an evenly split cluster can still have one exhausted worker, and what to do about it.

The Limitation

As Chapter 11 explained, the leader counts pieces of work; it does not weigh them. A task that copies a tiny reference table and a task that streams millions of records per second each count as one. If the heavy tasks happen to land on the same worker, that worker runs hot while its neighbors idle, yet the cluster is considered perfectly balanced.

What Contributes to Skew

Practical Remedies

A healthy cluster rebalances rarely and quickly. Metrics tell you when that stops being true.

Alerts That Actually Help

flowchart LR
  M["Worker rebalance metrics"] --> D["Dashboard"]
  S["Task status metrics"] --> D
  R["Resource use per worker"] --> D
  D --> A1["Rebalance rate alert"]
  D --> A2["Long rebalance alert"]
  D --> A3["Protocol fallback alert"]
  A1 --> O["On-call engineer"]
  A2 --> O
  A3 --> O
        

Simple habits that keep rebalancing boring, which is exactly what you want.

The Management Interface

Every Connect worker offers a small web interface for managing the cluster. Through it you can list connectors, see the status of each connector and each task including which worker currently hosts it, pause and resume connectors, restart connectors and individual tasks, and change configurations. Because any worker can answer, and requests are forwarded to the right owner when needed, you can talk to whichever worker is convenient. During a rebalance, some requests may briefly be refused or delayed while ownership is uncertain, and your tools should be ready to retry.

Helpful Habits

A six-worker cluster is upgraded one worker at a time, and we watch what the cooperative protocol does.

An analytics company runs a Connect cluster of six workers carrying thirty connectors and about ninety tasks, so each worker holds roughly fifteen tasks. The team has set a scheduled rebalance delay of five minutes, and a restart of one worker takes about ninety seconds. Tonight they deploy a new version.

Habits and reusable shapes that experienced Connect operators rely on.

Proven Practices

Reusable Design Patterns

The protocol can only be as gentle as the connectors it moves. Authors hold half of the responsibility.

Why Connector Design Matters

Whenever the cluster moves a task, it asks a piece of someone’s code to stop and asks another copy of that code to start. If the code stops slowly, starts slowly, or leaves messy leftovers, the protocol’s gentleness is wasted. Connector authors, whether they work inside your company or in a community project, influence rebalancing more than most operators realize. If you choose connectors or write your own, these qualities make the difference between a well-behaved neighbor and a troublemaker.

The Task Count Trap

Many connectors look at the outside system, for example the tables in a database or the files in a folder, and split the work into tasks accordingly. If the outside system changes often, the connector may decide the task list has changed and ask the cluster to reconfigure. Every such request causes a rebalance for everyone. A well-designed connector applies a sensible minimum time between checks, ignores trivial changes, and groups changes together instead of reacting to each one individually.

What to Test Before You Trust a Connector

Sizing the cluster so that losing a worker is an inconvenience, not a crisis.

The Survivors Must Absorb the Work

When a worker is lost for good, its tasks are handed to the remaining workers after the scheduled delay. Those workers must have room to take them. If every worker is already running at ninety percent of its capacity, no worker can absorb its neighbor’s load, and the cluster becomes overloaded exactly when it is under stress. A widely used rule of thumb is to plan for the cluster to survive the loss of one worker, or one whole group of workers, while staying comfortably within its limits. This is often called planning for N plus one.

A Simple Sizing Method

Failure Domains

A failure domain is a group of machines that can fail together, such as all machines in one rack, one power feed, or one availability zone. If all your workers sit in one domain, a single event takes down the whole cluster and no amount of clever rebalancing can help. Spreading workers across several domains means a single event removes only a fraction of them, and the rest continue running and take over the lost work.

flowchart TD
  subgraph Z1["Zone one"]
    A["Worker 1"]
    B["Worker 2"]
  end
  subgraph Z2["Zone two"]
    C["Worker 3"]
    D["Worker 4"]
  end
  subgraph Z3["Zone three"]
    E["Worker 5"]
    F["Worker 6"]
  end
  Z1 --> K["Kafka cluster and internal topics"]
  Z2 --> K
  Z3 --> K
        

The Cluster Beneath the Cluster

A Connect cluster depends on the Kafka brokers it talks to. The group coordinator lives on a broker, and the internal topics live on brokers. If a broker that hosts the coordinator becomes unavailable, the Connect workers see membership trouble even though nothing is wrong with the workers themselves. Likewise, if the internal topics are not replicated properly, a broker failure can make the shared configuration unavailable. Healthy Connect rebalancing needs a healthy Kafka cluster underneath, with internal topics replicated for safety.

Ways rebalancing setups go wrong, and how to avoid each one.

Quick Reference: Mistakes and Their Fixes

A practical checklist for the moments when the cluster feels restless.

A Simple Investigation Order

Short, honest answers to the questions people ask most often.

Incremental means only the work that needs to move is moved. Cooperative means workers keep running during a rebalance and release specific pieces of work when asked, so a piece is released by its old owner before it is granted to its new one.

No. The same events still cause rebalances. What changes is the cost: most work keeps running, and only the pieces that must move are interrupted.

Check the worker rebalance metrics, which report the protocol the group agreed on. Do not rely only on the settings of individual workers, because the group uses the least capable common option.

Compatible lets workers speak both eager and cooperative languages, mainly to support upgrades. Sessioned is cooperative rebalancing plus signed internal requests between workers using rotating keys.

Because the worker might come back soon, for example after a quick restart. The leader waits for the scheduled delay so that a returning worker can reclaim its own tasks without disturbing anyone else.

No. A failed task simply stays in a failed status until someone restarts it. Rebalancing concerns who owns work, not whether the work is healthy.

Yes, around each move, because tasks resume from saved positions that may be slightly behind. There are simply far fewer moves, so far fewer duplicates. Destinations should still tolerate repeats.

No. Worker rebalancing divides connectors and tasks among workers. Each sink connector also has a consumer group that divides topic partitions among its tasks, with its own settings. The two layers can trigger each other.

Not directly. The leader balances by counts of connectors and tasks. To handle uneven load, split heavy connectors into more tasks, separate heavy workloads into their own cluster, and leave headroom on every worker.

Yes, with care. Upgrade the software everywhere, move workers to the compatible setting one at a time, and the group switches to cooperative behavior when the last worker changes. Avoid mixing incompatible settings.

After the shutdown timeout, the worker moves on and the task may keep running as a zombie beside its replacement. Prevent this with responsive stop logic and timeouts on all outside calls.

No. Standalone mode runs a single process with no group of workers, so there is nothing to divide. Rebalancing belongs to distributed mode.

Only if you prefer immediate takeover over gentle restarts, for example on very unreliable machines where workers rarely come back. Otherwise a moderate delay avoids needless movement.

Yes. Cooperative rebalancing arrived in Kafka 2.3, and newer releases have refined defaults and added features. Check the documentation for your exact version before relying on specific setting names or default values.

A Kafka Connect cluster shares connectors and tasks among workers, and every change in membership or workload requires the group to divide that work again. The original eager protocol solved this by stopping everything and starting from scratch, which was simple but made the cost of every change equal to the size of the whole cluster. Incremental cooperative rebalancing keeps most work running, moves only what must move, and always releases a piece from its old owner before granting it to a new one. A scheduled delay prevents needless shuffling when a worker returns quickly, and protocol settings let clusters upgrade safely through a compatible stage. Good operation still depends on clean shutdowns, responsive tasks, sensible timeouts that agree with one another, repeat-safe destinations, attention to the separate consumer layer inside sink connectors, and metrics that show how often and how long the cluster rebalances.