Migration from ZooKeeper to KRaft in Kafka
How to move a running Kafka cluster from its old outside helper to its new built-in brain, one careful step at a time, without stopping the messages. Explained from zero, with everyday pictures, clear phases, safe rollback points, and real-world lessons.
Imagine your family lives in a house that has served you well for years. One day you learn that the old neighborhood is closing down, and every family must move to a new house next door. The catch is that you cannot pause your life. Dinner must still be cooked, the children must still go to school, and the mail must still arrive while the boxes are being carried across the street.
That is what migrating an Apache Kafka cluster from ZooKeeper to KRaft feels like. For many years, Kafka depended on a separate helper program called ZooKeeper to remember the cluster’s important facts. Newer Kafka versions keep those facts inside Kafka itself, using a built-in system called KRaft. The old helper has been retired in current major versions, so every team that still runs it must eventually move.
The good news is that Kafka includes a carefully designed migration path that can be done while the cluster keeps serving messages. In this guide you will learn what is being moved, why the move is safe, each phase in order, where you can turn back, what can go wrong, and how to check your work. You do not need any earlier knowledge of ZooKeeper, Raft, or Kafka internals to follow along.
Why Migration Matters: The Big Picture
Before any steps, let us understand what is really being moved and why it cannot simply be ignored.
A Kafka cluster is a team of servers, called brokers, that store and deliver streams of messages. To work as a team, the brokers need a shared notebook. In that notebook they write which topics exist, which broker holds which piece of data, who currently leads each piece, and which brokers are alive. This notebook is called the cluster’s metadata.
For a long time, that notebook was kept by a separate program, Apache ZooKeeper. Now the notebook lives inside Kafka. A migration is the careful act of copying the notebook from the old place to the new place and switching everyone to read and write the new one, while the cluster continues its daily work.
Think of a busy restaurant that keeps its table bookings in a notebook at a neighboring shop. The owner decides to keep the bookings inside the restaurant instead. She cannot close for a week to do it. So she copies the bookings to a new book, writes every new booking in both books for a while, moves the staff to the new book one by one, and only throws away the old notebook when everyone agrees the new one is right.
What does NOT move
Your messages do not move. The data sitting on the brokers’ disks stays where it is. Producers and consumers keep talking to the same brokers using the same addresses. Only the cluster’s notebook of facts and the job of managing it change places. This is why the migration can be done without asking your applications to change anything.
Why it must be planned, not rushed
The notebook is the cluster’s memory. If it is copied wrongly, Kafka may forget that a topic exists, or forget who is allowed to read it. So the migration is designed with safety points, backups, and a way to go back during most of the journey. Your job is to follow the steps in order and verify each stage before moving on.
Migration moves the cluster’s brain, not its messages. Treat the brain move with care, and the messages will barely notice.
Kafka Basics You Need First
A short tour of the words we will use throughout the guide.
Broker
A Kafka server that stores messages on disk and serves producers and consumers.
Topic
A named stream of messages, such as “orders” or “payments”.
Partition
A slice of a topic. Many partitions let many machines share the work.
Replica
A copy of a partition on another broker, so a failure does not lose data.
Partition leader
The replica that handles reads and writes for a partition. The others copy it.
Controller
The broker or node in charge of cluster decisions such as choosing partition leaders.
Producers are apps that write messages, and consumers are apps that read them. They talk to brokers about messages. They do not talk to ZooKeeper or to KRaft controllers, which is the main reason they are protected from most of the migration work.
Picture a network of libraries. Each library branch holds books (partitions), and each book has a main copy and spare copies (replicas). A head office keeps the master catalog (metadata) and decides which branch keeps the main copy when a branch closes (the controller). Visitors only deal with branches. They never visit the head office.
What is a KIP?
You will see the word KIP in Kafka discussions. It stands for “Kafka Improvement Proposal”, which is a written design for a change. The idea to remove ZooKeeper is known as KIP-500, and the migration procedure was described in a later proposal. You do not need to read them to migrate, but it helps to know the names when you search for documentation.
The Old World: How ZooKeeper Ran Kafka
To know what we are leaving, we look at how the old design worked and where it hurt.
ZooKeeper is a small coordination service. It stores tiny pieces of data organized like folders and files, and lets programs watch those pieces for changes. Kafka used it to keep its topic list, its settings, its access rules, and its list of live brokers. It also served as the referee that chose which broker would act as controller.
Brokers connect
Every broker opens a connection to ZooKeeper and creates a short-lived entry that says “I am alive.”
A controller is chosen
All brokers race to create one special entry. The first one wins and becomes the controller. The others watch the entry.
The controller loads everything
The winner reads the whole cluster’s metadata from ZooKeeper into its own memory.
The controller pushes updates
After each change, the controller sends messages to brokers so they learn about new leaders and topics.
Failure means a new race
If the controller dies, its entry disappears, and the brokers race again. The new winner must reload everything.
Where it hurt
Two systems to run
Two clusters to install, secure, upgrade, and watch, each with its own tools and its own ways of failing.
Slow controller recovery
A new controller had to reload all metadata, which took longer as the number of partitions grew.
Partition limits
Loading and pushing large amounts of metadata capped how many partitions one cluster could comfortably hold.
Two sources of truth
The controller’s memory and ZooKeeper could briefly disagree, causing rare and confusing problems.
The New World: What KRaft Changes
KRaft keeps the cluster’s notebook inside Kafka, in an ordered log that a small group of nodes agrees on.
KRaft means “Kafka Raft”. Raft is a well-known method that lets a small group of computers agree on one ordered list of changes, even if some of them crash. In KRaft, that list is stored as an internal Kafka topic on a small group of nodes called controllers. Each change to the cluster, such as a new topic or a new leader, becomes one small record at the end of the list.
A shared class diary. One student, the scribe, writes each event in order. Two other students copy every line into their own diaries. A line only counts once most of the diaries have it. If the scribe is absent, the three quickly vote for a new scribe who has the most complete diary.
| Topic | ZooKeeper mode | KRaft mode |
|---|---|---|
| Where metadata lives | In ZooKeeper | In a Kafka metadata log on the controllers |
| Who can be controller | Any broker | Nodes given the controller role |
| How controller is chosen | A race for a special entry | A vote among controller nodes |
| How brokers learn changes | The controller pushes them | Brokers pull from the log |
| New controller startup | Loads everything first | Already up to date in memory |
| Systems to operate | Kafka plus ZooKeeper | Kafka only |
Where the versions stand
KRaft appeared first as a preview in the 2.8 line, was declared ready for production in the 3.3 line, and the migration path from ZooKeeper became usable in the 3.4 line and mature in later 3.x releases. ZooKeeper support was removed in the 4.0 line. The last releases that still understand ZooKeeper are in the 3.x series, so a cluster that still uses ZooKeeper must reach KRaft before it can move to version 4 or beyond.
Exact minimum versions matter for migration. Always read the release notes and the official migration guide for the version you are running, and do not rely on memory or on old blog posts, including this one, for exact numbers.
Why Migrate, and When
Moving is work, so let us be clear about the reasons and the deadlines.
Reasons to migrate
- ZooKeeper is gone from current major versions, so staying blocks all future upgrades
- One system to secure, monitor, patch, and pay for instead of two
- Faster recovery when the controller changes
- Higher partition counts become practical
- Simpler mental model for new team members
Costs and risks
- Planning, rehearsal, and careful execution take real time
- New controller machines must be provisioned
- Teams must learn new tools and new metrics
- A mistake in a late phase can be hard to reverse
- Old scripts that talk to ZooKeeper must be updated
The deadline is set by your upgrade path
You do not need to migrate the day a new version appears. But your version of Kafka has a support window, and security fixes stop when it ends. Because the newest major version no longer includes ZooKeeper support, the migration must happen while you are still on a 3.x release. If you wait until the end of support, you will be forced to migrate in a hurry, which is exactly how mistakes happen.
It is like replacing an old roof. You could wait until it leaks during a storm and then do it in a panic with whatever workers are available. Or you can pick a sunny week, gather materials, and do it calmly. The roof costs the same either way, but the calm week costs far less stress.
Put the migration on your roadmap early. Start with a development cluster, then staging, and only then production. Each rehearsal teaches you something the documentation cannot.
Two Roads: In-Place Migration or a New Cluster
There are two broad ways to reach KRaft, and each fits different teams.
In-place (rolling) migration
Keep the same brokers and data. Add KRaft controllers next to the old setup, copy the metadata, restart brokers into KRaft mode one at a time, then retire ZooKeeper. This is the path this guide focuses on.
New cluster and copy (blue-green)
Build a brand new KRaft cluster next to the old one, copy topics and data across with a mirroring tool, and move applications over gradually. The old cluster is switched off at the end.
| Question | In-place migration | New cluster and copy |
|---|---|---|
| Extra hardware needed | A few small controller machines | A full second cluster during the move |
| Application changes | None | Applications must switch addresses and handle offsets |
| Data copying | None, data stays put | All data copied across the network |
| Rollback | Possible until the final step | Simple: keep using the old cluster until cutover |
| Complexity | Follow one precise runbook | Manage two clusters and their copying |
| Good when | You want minimal disruption | You also want to change versions, hardware, or layout |
In-place migration is like renovating your kitchen while still living in the house. Blue-green is like building a second house next door, moving in slowly, and selling the old one at the end. Renovating is cheaper but needs careful sequencing. A second house is calmer but costs a lot more.
Some managed Kafka services do not offer in-place migration and instead expect you to create a new cluster and copy data. If you use a managed service, check what your provider supports before planning.
The Migration Mental Model: Dual-Write
One simple idea makes the in-place migration safe: for a while, write everything in both places.
The whole in-place migration rests on a trick called dual-write. Once the new KRaft controllers have copied the old metadata, they do not simply switch off ZooKeeper. Instead, every new metadata change is written to the KRaft log and also copied back into ZooKeeper. Because both places stay up to date, brokers that still run in the old mode can carry on reading from ZooKeeper, and brokers that have been switched to the new mode can read from the KRaft log.
Two bank branches during a system change: every deposit is stamped into the old ledger and the new ledger at once. Customers can walk into either branch and see the same balance. Only when every branch has moved to the new ledger does the bank close the old one.
Why this gives you a safety net
As long as dual-write is on and ZooKeeper holds a full up-to-date copy, going back is possible. If something looks wrong, you can return the brokers to their old mode, remove the new controllers’ authority, and ZooKeeper already holds the truth. Once you take the last step, dual-write ends, ZooKeeper stops receiving updates, and the way back is closed.
A helpful picture: the state of the cluster
stateDiagram-v2
[*] --> ZooKeeperMode
ZooKeeperMode --> ControllersAdded : start KRaft controllers in migration mode
ControllersAdded --> MetadataCopied : brokers restarted with migration on
MetadataCopied --> DualWrite : controllers write to both places
DualWrite --> BrokersInKRaft : brokers switched to KRaft mode one by one
BrokersInKRaft --> Finalized : controllers restarted without migration settings
DualWrite --> ZooKeeperMode : rollback
BrokersInKRaft --> DualWrite : rollback by returning brokers to ZooKeeper mode
Finalized --> [*]
The exact rollback steps depend on the stage you are in and the version you run. The picture shows the idea: until you finalize, the old world is still kept up to date and can be returned to.
Architecture During Migration
For a while, the cluster lives in a mixed state. Here is who talks to whom.
flowchart TB
APP["Producers and Consumers"]
subgraph OLD["Old world"]
ZK[("ZooKeeper ensemble")]
end
subgraph NEW["New world - KRaft controller quorum"]
C1["Controller 1 - Active"]
C2["Controller 2"]
C3["Controller 3"]
end
subgraph BR["Brokers"]
BZ["Brokers still in ZooKeeper mode"]
BK["Brokers already in KRaft mode"]
end
APP -->|"messages"| BR
C1 -->|"dual-write copy of every change"| ZK
ZK -->|"initial metadata copy"| C1
C2 -->|"fetch"| C1
C3 -->|"fetch"| C1
C1 -->|"legacy update messages"| BZ
BK -->|"pull metadata log"| C1
BZ -->|"register and heartbeat"| C1
BK -->|"register and heartbeat"| C1
Read the figure from the middle. The active KRaft controller is now the boss. It copied the metadata from ZooKeeper at the start, and it keeps writing every change back to ZooKeeper. Brokers that have not yet been restarted in KRaft mode still behave in the old way: the controller sends them the same kind of update messages that the old controller used to send. Brokers that have been switched already pull the metadata log directly.
The key components
The ZooKeeper ensemble
Still running and still holding a full copy. It is the safety net until the last step.
The new controller quorum
Three (or five) dedicated controller nodes. They take over the controller job and run the dual-write.
Brokers in ZooKeeper mode
Not yet switched. They still register with ZooKeeper but also learn about the new controller and follow its instructions.
Brokers in KRaft mode
Already switched. They no longer talk to ZooKeeper at all.
Prerequisites and Readiness Checklist
A migration is won or lost before it starts. Check these items first.
Supported Kafka version
Run a release that officially supports migration, ideally a late 3.x release. Read its notes for known problems.
A healthy cluster
No offline partitions, no long under-replicated partitions, and brokers stable. Never migrate a cluster that is already struggling.
Log directory layout
Check whether you use multiple data disks per broker, since support for that in KRaft arrived in later releases than the first migration support.
Authorizer and listeners
Know which access-control plugin you use, and which listeners and security protocols exist, because the controllers need their own.
Client tools and scripts
List every script, dashboard, and automation that talks to ZooKeeper directly. They will stop working after the migration.
Backups and a rehearsal
Take a backup of the ZooKeeper data, record the current settings, and complete at least one full rehearsal in a non-production environment.
Do not start a migration on a Friday afternoon, during a holiday sale, or while other big changes are in progress. Choose a calm window, keep the team on call, and change only one big thing at a time.
Before a long road trip you check the tires, the fuel, the map, and the spare wheel. None of these checks are exciting, but each one prevents a bad day on the highway.
Version Planning: The Upgrade Path Around Migration
Migration and upgrades are related but should not be mixed carelessly.
Because ZooKeeper support ends with the 3.x series, the path for every ZooKeeper-based cluster looks like the same simple road: first bring the cluster up to a late 3.x release that fully supports migration, then migrate to KRaft, and only afterwards move to version 4 and beyond. Trying to jump straight from an old release to a new major version usually fails, because the new version does not understand ZooKeeper at all.
Upgrade to a suitable 3.x release
Do it with the normal rolling upgrade process, and let the cluster settle. Read the release notes for migration-related fixes.
Run the migration
Follow the five phases while staying on the same software version, so you change only one big thing at a time.
Stabilize
Run for a while in pure KRaft mode. Fix leftover tools and dashboards.
Upgrade to the next major version
Now you can move to newer releases using ordinary rolling upgrades, and raise the metadata version when ready.
You would not change your car’s engine and its tires and its steering all on the same afternoon while driving on the highway. You do one at a time, test, and then continue. Version changes and architecture changes deserve the same respect.
The exact minimum and recommended versions change as new releases appear. Always read the current migration guide and release notes of the exact version you plan to use, instead of relying on remembered version numbers.
Hardening the Cluster Before You Move
Rolling restarts are the heartbeat of the migration. Make sure your cluster can survive them.
Every phase of the migration leans on rolling restarts, so the cluster must be able to lose one broker at a time without harm. Many clusters look healthy but have hidden weaknesses that only show during a restart. Fix these in advance.
Replication factor of at least three
Topics with a single copy go offline whenever their broker restarts. Topics with three copies stay available even during a restart.
A sensible minimum in-sync setting
With three copies, requiring two in sync allows one broker to restart safely. Requiring all three blocks writes whenever any broker is down.
Unclean leader election off
Keep it off so a restart can never promote a stale copy. This protects confirmed messages.
Balanced partitions and spare disk
Even leadership spreads the restart load. Free disk space gives room for catch-up traffic.
Clients that retry properly
Check application settings for retries and timeouts, so a short leader change does not become an error page.
Before renovating a room, you move the fragile furniture out and cover the floor. The renovation itself is not harder, but the cost of an accident is much lower.
Topics with only one replica are the most common surprise. Find them with a listing before you start, and either increase their replication or agree with the topic owners that a short outage is acceptable.
Planning: Sizing, Identity, and Networks
Decide the shape of the new controllers before you build them.
How many controllers?
Use three dedicated controller nodes for most production clusters, and five for large or business-critical ones. Choose an odd number, because a group needs a majority to decide anything, and an even number adds cost without adding safety. Place them in different availability zones, and give each machine its own fast disk. Metadata is small, so machines do not need to be large, but disks must be steady and quick.
Identity matters
Cluster ID
Every Kafka cluster has a unique label. The new controllers must use the same label that the existing cluster already has, which you can read from the current cluster. A different label would make the controllers a stranger to the brokers.
Node IDs
Each controller needs a number that is unique among all nodes. Pick numbers that do not clash with any existing broker number.
The voter list
Every node needs to know the list of controller nodes with their addresses. Write it down carefully and keep it identical everywhere.
A separate controller listener
Controllers use their own network port and security setting, separate from the client-facing ones. Brokers must be able to reach it.
Moving into a new office building means using the same company name (cluster ID), giving every new employee a unique badge number (node ID), publishing a phone list (voter list), and opening a separate staff entrance (controller listener).
Network planning
Open the controller port between controllers, and from every broker to every controller. If you use firewalls or security groups, update them before the migration starts. Confirm that names resolve correctly from every machine. Many migration delays come from a single blocked port or a wrong hostname.
Phase 1: Provision the KRaft Controllers
The first real action is to build the new brain next to the old one, without letting it take control yet.
Prepare the machines
Provision the controller nodes with the same Kafka version as the brokers, with dedicated disks and correct network access.
Format the storage
Use the storage tool to prepare each node’s metadata folder, supplying the existing cluster ID.
Set the controller role and migration switch
Give each node only the controller role. Turn on the setting that allows migration, and give it the connection details of the existing ZooKeeper so it can read the old metadata.
Start the controllers
Start all of them. They hold their first election and choose an active controller among themselves.
Verify the quorum
Use the quorum description tool to confirm there is one leader and that every voter is following it.
At this point, the new controllers are alive but they do not run the cluster yet. They are waiting for the brokers to introduce themselves. The old ZooKeeper-based controller (one of the existing brokers) is still in charge, and messages flow as before. That is why this phase is safe and easy to undo: if anything looks wrong, you can simply stop the new controllers, and nothing has changed for the cluster.
Do not move on until the quorum is healthy and stable for a while. Watch the logs for repeated elections. A shaky new controller group will make the next phases harder.
Phase 2: Enable Migration on the Brokers
Now the existing brokers are introduced to the new controllers, one at a time.
In this phase, you change the settings of each broker so that it knows about the new controllers and allows the migration to begin. The settings tell the broker the list of controller nodes, which listener name the controllers use, and that migration is allowed. Then you restart the brokers in a rolling way: one broker, wait until it is healthy, then the next.
sequenceDiagram
participant B as Broker restarting
participant NC as New KRaft Controller Leader
participant ZK as ZooKeeper
B->>NC: Register with migration enabled
NC->>NC: Wait until all brokers have registered
NC->>ZK: Read topics, settings, access rules
ZK-->>NC: Full copy of metadata
NC->>NC: Write copied metadata into the KRaft log
NC->>ZK: Claim control of the cluster
Note over NC,ZK: Dual-write mode begins
Update the broker settings
Add the controller voter list, the controller listener name, its security mapping, and turn on migration. Keep the ZooKeeper connection, because the broker still needs it for now.
Restart one broker
Use a graceful shutdown so its partition leaders move away first. Start it again and wait until it has caught up.
Repeat for every broker
The migration only begins once the controllers see all brokers registered with the migration setting. Brokers that have not yet registered hold it back.
Automatic metadata copy
Once every broker has registered, the active controller reads the metadata from ZooKeeper and writes it into the KRaft log. This usually takes a short time, depending on cluster size.
Control moves to KRaft
The controller claims leadership from the old ZooKeeper-based controller, and dual-write starts.
A school is moving its records to a new office. Every teacher must first sign in with the new office so it knows who is who. Only when the whole staff has signed in does the new office start copying the old files, so that nobody is left out of the new system.
Restart brokers strictly one at a time and wait until the cluster has no under-replicated partitions before touching the next one. Restarting two at once can take partitions offline.
Phase 3: Dual-Write Mode, the Calm Middle
This is the safest place to pause. Everything works, and the way back is open.
Once the metadata is copied and the KRaft controller has taken charge, the cluster is in dual-write mode. The KRaft controller is now the active controller, and it also writes every change back into ZooKeeper. To your applications, nothing looks different. To your operators, there are new metrics, new logs, and a new set of tools to look at.
What the controller does in this mode
Decides partition leaders
When a broker fails, the KRaft controller picks new partition leaders, not the old controller.
Handles topic and setting changes
Creating a topic or changing a setting goes through the KRaft log, and is then copied to ZooKeeper.
Speaks the old language
For brokers still in ZooKeeper mode, it sends the older kinds of update messages so they stay informed.
Keeps ZooKeeper current
Because ZooKeeper always has the same facts, a rollback remains possible.
Think of a relay race where the new runner has taken the baton, but the old runner keeps jogging alongside with a hand on the runner’s shoulder, ready to take the baton back if needed. The team is safe, but not yet fully committed.
How long should you stay here?
Long enough to be confident. Many teams stay in dual-write for a day or more on a production cluster, watching for problems during normal daily traffic, busy hours, and any scheduled jobs. Try the usual operations here: create a test topic, change a setting, add a test access rule, and cause a controlled leader change. Confirm that each action works and appears in both the new log and the old ZooKeeper.
Dual-write is a fine place to stop for as long as you need. It is not meant to last forever, however. Leaving a cluster in the mixed state for months adds risk, because few people remember how it works.
Phase 4: Roll the Brokers into KRaft Mode
Now each broker stops talking to ZooKeeper and becomes a pure KRaft broker.
In this phase, you change each broker’s settings again. You remove the ZooKeeper connection, set the node’s role to broker only, and give it a node number that matches the old broker number. You also update the access-control plugin, because the older ZooKeeper-based one cannot run in KRaft mode. After that, you restart each broker, again one at a time.
Change the role
Set the process role to broker. The node no longer needs to think about ZooKeeper.
Keep the same identity
The node number must match the broker number it had before. Its data folders keep their contents, so its partitions stay with it.
Switch the authorizer
Replace the ZooKeeper-based access-control plugin with the KRaft-compatible one. Access rules were already copied into the metadata log.
Restart gracefully, one by one
Wait for the broker to rejoin, catch up, and for under-replicated partitions to return to zero.
Check the roster
After each restart, verify the broker appears in the cluster and takes back its share of partition leadership.
Employees moving into the new office one at a time. Each employee keeps the same badge number and the same desk contents, but hands in the old office key. When the last employee has moved, nobody needs the old building.
Mixed clusters are normal here
While you are rolling, some brokers are in ZooKeeper mode and some are in KRaft mode. That is expected. The controller talks to both kinds. You can pause between brokers, for example overnight, without any harm, as long as the cluster stays healthy.
Forgetting to switch the authorizer is a classic mistake. A broker in KRaft mode with the old plugin will refuse to start or will fail to enforce access rules. Test this exact change in staging first.
Phase 5: Take Over and Finalize
The last step closes the old world for good.
When all brokers run in KRaft mode and the cluster has been stable for a good while, you take the final step. You remove the migration and ZooKeeper settings from the controller nodes and restart them one at a time. When the last controller restarts without those settings, the cluster no longer reads from or writes to ZooKeeper. Dual-write ends. ZooKeeper is now a useless, out-of-date copy.
flowchart LR
P1["Phase 1 - Provision controllers"] --> P2["Phase 2 - Enable migration on brokers"]
P2 --> P3["Phase 3 - Dual-write mode"]
P3 --> P4["Phase 4 - Roll brokers into KRaft mode"]
P4 --> P5["Phase 5 - Finalize controllers"]
P5 --> DONE["Retire ZooKeeper"]
P1 -.->|"rollback easy"| R1["Stop new controllers"]
P2 -.->|"rollback possible"| R2["Return brokers to old settings"]
P3 -.->|"rollback possible"| R2
P4 -.->|"rollback with extra steps"| R3["Return brokers to ZooKeeper mode"]
P5 -.->|"no way back"| X["Point of no return"]
It is like burning the old map after you have arrived. Before that moment, you could still walk back. After it, you rely completely on the new place. That is why you only do it when you have looked around and are sure.
After the final step
Confirm health
Verify the quorum, the broker list, topic counts, and access rules once more.
Keep ZooKeeper for a short time
Leave the old machines idle but available for a few days, in case someone discovers an old script that still points to them.
Shut down ZooKeeper
After the waiting period, stop the ensemble, remove firewall rules, and delete monitoring for it.
Update documentation
Change runbooks, diagrams, and on-call guides so they describe the new reality.
Inside the Migration Engine: The Controller’s Checklist
The new controller does not just “copy files.” It follows a strict sequence of steps, like a pilot running a pre-flight checklist.
Inside the active KRaft controller lives a small helper that drives the migration. You can think of it as a careful assistant with a clipboard. It moves through a fixed list of stages, and it never skips ahead until the current stage is finished. Knowing these stages helps you read the logs and understand what the cluster is waiting for at any moment.
Wait for the controller quorum
The assistant first waits until the KRaft controllers have elected a leader and the group is healthy. Without a working quorum, nothing else can be recorded safely.
Wait for the brokers
It waits until every broker has registered and announced that migration is allowed. This is the stage where a forgotten broker holds everything back.
Become the controller
It claims the controller role in ZooKeeper, so that the old ZooKeeper-based controller steps aside. From now on there is only one boss.
Copy the metadata
It reads the topics, partitions, settings, and access rules from ZooKeeper and writes them as records into the KRaft log, in batches.
Tell the brokers
It sends the full picture of the cluster to the brokers that are still in ZooKeeper mode, so everyone starts from the same facts.
Enter dual-write
It starts copying each new change back to ZooKeeper. The migration is now “live.”
Think of a stage manager at a theater who changes the whole set between acts. First she checks that the crew is ready, then that every actor has arrived, then she takes the microphone, then she moves the props, then she tells the actors where things now stand, and only then does she raise the curtain. Every step depends on the one before it.
A small bookmark in ZooKeeper
During the migration, the controller keeps a tiny bookmark inside ZooKeeper that records how far the copy has progressed and which KRaft log position matches it. If the active controller crashes in the middle, the next KRaft leader reads the bookmark and continues from the right place instead of starting again or, worse, repeating work. This is a common pattern in careful long-running operations: leave a note behind so a replacement can pick up the thread.
When something seems stuck, ask which stage the assistant is in. The controller logs state which stage it has reached and what it is waiting for. That one question solves most puzzles.
How Dual-Write Keeps Both Worlds in Sync
A closer look at the trick that makes the safety net real.
Every metadata change in KRaft becomes a record in the log, and the log is committed only after a majority of controllers have it. In dual-write mode, the active controller does one extra thing: after a batch of records is committed, it translates those records into the matching updates for ZooKeeper and writes them there too. The order of changes is always the KRaft log order, so ZooKeeper receives the same story in the same sequence.
A change is requested
For example, an admin creates a topic.
The controller records it
It appends records to the KRaft log and waits for a majority of controllers to store them.
The change is translated
The same change is turned into ZooKeeper’s own format: the entries that the old controller would have written.
ZooKeeper is updated
The controller writes the entries and updates its bookmark so that the two places are known to match.
Brokers are informed
ZooKeeper-mode brokers get the legacy update messages, and KRaft-mode brokers pull the new records.
A court reporter types the official record, and a second clerk immediately copies every page into an older archive book for a while, so that if the new record ever had to be set aside, the old archive would still be complete and in the right order.
The direction of truth
An important detail is that the KRaft log is the source of truth during dual-write, and ZooKeeper is the follower. Nobody should edit ZooKeeper by hand during this period. If someone changes a topic setting directly in ZooKeeper using an old script, the KRaft controller will not see it as an official change, and the two places can drift apart. This is one more reason to find and retire old scripts before you begin.
What if ZooKeeper cannot be written?
If ZooKeeper is slow or unreachable, the controller cannot finish step four. It will retry, and metadata changes may wait or be delayed until ZooKeeper returns. Messages continue to flow through brokers, but tasks that need the controller, such as creating topics, may pause. It is a very good reason to keep ZooKeeper healthy, monitored, and properly sized right up until finalization.
Never edit metadata directly in ZooKeeper during dual-write, and never “clean up” its entries because they look old. Doing so can silently break the safety net or confuse the controller.
Talking to Old and New Brokers at the Same Time
During the middle of the migration, the controller must speak two languages.
ZooKeeper-mode brokers expect the old way of receiving news: the controller sends them a small set of messages that say “you are now the leader of this partition,” “here is the updated list of brokers and topics,” and “stop hosting this replica.” These are sometimes called the legacy control messages. KRaft-mode brokers do not use them at all. Instead they simply read the metadata log.
| Broker type | How it learns about changes | Where it registers |
|---|---|---|
| ZooKeeper-mode broker (during migration) | Receives legacy update messages from the KRaft controller | With the KRaft controller, and still keeps its ZooKeeper connection |
| KRaft-mode broker | Pulls records from the metadata log | With the KRaft controller only |
Imagine a company that is switching from paper memos to a shared online board. For a few weeks, the office manager still walks around handing paper memos to people who have not switched yet, while everyone who has switched simply checks the board. When the last person switches, the manager stops the paper round.
Why this is a short-lived arrangement
The legacy messages are a bridge, not a destination. They exist only so that the cluster can be half in the old mode and half in the new one without breaking. Each broker you move into KRaft mode reduces the need for the bridge. Once the last broker has moved, the controller no longer sends any legacy messages, and the bridge quietly falls unused.
Because of this bridge, you can pause with some brokers in each mode. Just avoid leaving the cluster half-migrated for a long time. The fewer moving parts, the safer the cluster.
A Metadata Change Through Every Phase
Follow one simple action, creating a topic, and see how its journey changes as the migration advances.
| Phase | Who handles “create topic”? | Where the facts are written | How brokers hear |
|---|---|---|---|
| Before migration | The old ZooKeeper-based controller | ZooKeeper only | Pushed update messages |
| Controllers built, not active | The old ZooKeeper-based controller | ZooKeeper only | Pushed update messages |
| Dual-write, brokers in ZooKeeper mode | The KRaft controller | KRaft log first, then ZooKeeper | Legacy update messages |
| Dual-write, mixed brokers | The KRaft controller | KRaft log first, then ZooKeeper | Legacy messages or log pulls, depending on the broker |
| All brokers in KRaft mode | The KRaft controller | KRaft log first, then ZooKeeper | Log pulls |
| After finalization | The KRaft controller | KRaft log only | Log pulls |
Think of a letter being posted while a post office changes its sorting system. The sender always drops it in the same box. Behind the scenes, the route changes from the old sorting room to a hybrid room and finally to the new machine. The sender never has to know.
This table shows a nice truth about the migration: from the outside, the same request works in every phase. Only the inner journey changes. That is why it is safe to test simple actions, like creating a test topic, at each step. If the action works in each phase and shows up correctly in your before-and-after fingerprint, the phase is healthy.
Suggested smoke tests after each phase
Create and delete a test topic
Confirms the controller is accepting changes and brokers are following.
Change a topic setting
Confirms settings travel through the log and reach every broker.
Produce and consume a few messages
Confirms the data path is healthy from an application point of view.
Add and remove a test access rule
Confirms security changes are flowing through the same path.
The Life of One Broker Restart
Every phase repeats the same small routine. Understanding it once makes the whole migration easier to trust.
Almost everything in the migration comes down to restarting one broker safely, many times. So let us follow a single broker from the moment you stop it until it is fully back, and see what the cluster does around it.
The broker is asked to stop politely
It tells the controller that it is leaving. The controller quickly picks new leaders for every partition this broker led, choosing from the copies that are fully caught up.
Clients notice and move
Producers and consumers receive a hint that leaders changed. They ask for a fresh map and continue with the new leaders, usually within a second or two.
The broker stops and its settings change
You edit the settings for the current phase while the process is off. Its data folders stay untouched on disk.
The broker starts and registers
It introduces itself to the controller, receives a current picture of the cluster, and checks its data folders.
It catches up
For each partition it hosts, the broker fetches whatever it missed while it was away. This is the slowest part on busy clusters.
It rejoins the in-sync lists
Once caught up, the partition leaders add it back to their lists of fully caught-up copies. Under-replicated partitions fall to zero.
Leadership is rebalanced
Over time the controller returns preferred leadership to the broker, which restores the even spread of work.
It is like a shift change at a hospital. The outgoing nurse hands over every patient to a colleague, leaves, returns after a break, reads the notes about what happened, and slowly takes patients back. No patient is ever left without a nurse.
Why the wait matters
If you restart the next broker before the previous one has caught up, some partitions may have two copies away at the same time. With three copies, that leaves only one, which is too risky. Waiting for zero under-replicated partitions is what turns a dangerous operation into a routine one.
Write the wait condition down as a hard rule: “we do not touch the next broker until under-replicated partitions are zero and stay zero for a few minutes.” Rules like this stop impatience from creating outages.
Rollback: How Far Back Can You Go?
A good plan includes the plan for when the plan does not work.
| Where you are | Can you go back? | What it involves |
|---|---|---|
| Controllers built, not yet in use | Yes, easily | Stop and remove the new controllers. The cluster never changed. |
| Brokers restarted with migration enabled, before the copy completes | Yes | Restore the brokers’ earlier settings and restart them one by one. |
| Dual-write mode | Yes | Return brokers to old settings, then hand the controller job back to a ZooKeeper-based controller according to the official steps. |
| Some brokers already in KRaft mode | Yes, with more steps | Return those brokers to ZooKeeper mode first, then follow the earlier rollback path. |
| After finalization | No | ZooKeeper is out of date. Recovery means restoring from backups, not rolling back. |
Rollback is a real safety net, but it is also a delicate procedure of its own. The official steps depend on your version and on how far you got. Read them before you start the migration, not while you are in the middle of a problem. Practice at least one rollback in your rehearsal environment so the team has seen it once.
A mountain guide always decides in advance at which point the group will turn around if the weather changes. Deciding while standing in a storm, tired and afraid, leads to bad decisions. Deciding in advance keeps everyone safe.
Write your “turn back” conditions in advance, for example: “if under-replicated partitions stay above zero for more than thirty minutes after a broker restart, we stop and investigate.” Written rules make calm decisions.
What Gets Migrated, and What Does Not
Knowing the boundary of the move prevents nasty surprises.
| Item | Moves automatically? | Notes |
|---|---|---|
| Topics and partitions | Yes | Including replica placement and leaders |
| Topic and broker settings | Yes | Dynamic settings are copied into the log |
| Access-control rules | Yes | The authorizer plugin must be switched when brokers go KRaft |
| Quotas and credentials stored in metadata | Generally yes | Verify each type in your version’s notes |
| Message data on disks | Not moved at all | Stays exactly where it is |
| Consumer group offsets | Not affected | Stored in a Kafka topic, not in ZooKeeper |
| Custom things you stored in ZooKeeper yourself | No | Anything outside Kafka’s own use must be handled separately |
| Scripts and tools that read ZooKeeper | No | They must be rewritten to use Kafka’s own tools |
The overlooked list: hidden dependents
Over many years, teams often build monitoring scripts, dashboards, and small automation that read Kafka information straight from ZooKeeper. Some third-party tools also do this. After the migration, these tools will either fail or show old, frozen data. Make an inventory before you start, and give each tool an owner and a plan to switch to Kafka’s own interfaces.
A dashboard that quietly keeps showing old data is worse than one that breaks loudly. After finalization, check that every graph and alert is fed from live sources.
Client Impact and Zero Downtime
What your applications will and will not notice.
Producers and consumers talk to brokers, and brokers keep running throughout the migration except for their brief rolling restarts. When a broker restarts gracefully, it first hands its partition leadership to other replicas. Well-written clients notice a short leader change, fetch the new map of the cluster, and continue after a brief pause.
Retries hide small blips
Client libraries retry failed sends and fetches. A restart that moves leaders for a few seconds is typically absorbed by retries.
Strong safety settings help
Producers that wait for all in-sync replicas to confirm are the best protected. Enough in-sync replicas must exist for a restart to be safe.
Addresses stay the same
Brokers keep their addresses, so applications need no configuration change.
Admin tools change
Anything that administers the cluster should use the brokers’ address, not ZooKeeper’s address.
Customers of a supermarket do not notice that the back office changed its filing system. They only see the shelves and the checkout. As long as the shelves stay stocked and cashiers stay at their posts, the change is invisible.
Topics with a single replica, or clusters where the minimum in-sync setting is too high for a rolling restart, can suffer real pauses. Fix those weaknesses before migration, because every rolling restart depends on them.
Performance, Timing, and Capacity Planning
How long will this take, and what load will it place on your systems?
The migration has two kinds of time: machine time and human time. Machine time is how long each automatic step takes, and it is usually short. Human time is how long you spend restarting brokers and waiting for them to settle, and it is usually the longest part.
Machine time
The metadata copy
Its length grows with the number of topics and partitions. Metadata is small compared with message data, so even large clusters usually finish in minutes, but measure it in your rehearsal.
Pressure on ZooKeeper
The controller reads a lot from ZooKeeper at the start and writes to it during dual-write. A weak ZooKeeper ensemble may be slow, so check its health first.
Controller memory
Each controller keeps the whole cluster picture in memory. Size the machines so this fits comfortably, with room to grow.
Metadata log disk
Small but latency-sensitive. Every commit waits for a disk write on a majority of controllers.
Human time: a simple estimate
Suppose you have twelve brokers, and each rolling restart takes about ten minutes including the wait for replicas to catch up. Phase 2 then takes about two hours of restarts. Phase 4 takes about the same. If each phase is done on a different day, and you rest in dual-write for a couple of days, the whole journey fits comfortably in a working week or two. Clusters with very large brokers, where each restart needs a long recovery, need proportionally more time.
Repainting a long fence: each board only takes a minute, but you must let each coat dry before the next one. The total time is dominated by the waiting, not the painting.
Time your rehearsal properly. Record the duration of each restart and of the metadata copy. Multiply by your production numbers, add a comfortable buffer, and schedule your window from that estimate, not from hope.
Failure Scenarios During Migration
Things can go wrong in any long operation. Here is how to think about the common ones.
A broker will not start after its settings change
Usually a mistyped controller address, a wrong listener name, or a missing security mapping. Fix the settings and try again. The other brokers keep the cluster running while you work.
The metadata copy never begins
The controllers wait until every broker has registered with the migration setting on. One forgotten broker, or one that cannot reach the controllers, holds it back. Check the controller logs for the list of missing brokers.
A controller node fails during dual-write
If a majority of controllers is still alive, the cluster carries on. A new leader is elected among the rest and dual-write continues. Bring the failed node back at your leisure.
Majority of controllers lost
Metadata changes stop until a majority returns. Existing partitions keep serving messages while their brokers stay healthy. This is why controllers should live in separate zones.
ZooKeeper becomes unavailable in dual-write
The controller cannot keep the safety copy up to date and may pause metadata changes. Restore ZooKeeper quickly. This shows why ZooKeeper must stay healthy until the last phase.
A rolling restart causes under-replicated partitions to linger
Stop the roll. Investigate slow disks, network limits, or overloaded brokers. Continue only after replicas are back in sync.
The most dangerous reaction to a scare is to change many things at once. Change one thing, wait, observe, and decide. This applies during the migration and afterwards.
Security During and After Migration
The controllers hold the keys to the cluster, so they need the strongest protection.
Lock the controller listener
Only brokers, controllers, and trusted admin hosts should reach the controller port. Apply firewall rules before you start the controllers.
Encrypt and authenticate
Use encryption on the controller listener and require each peer to prove its identity, just as you do for brokers.
Carry over access rules
Verify after migration that the same principals have the same permissions. Compare a list from before with a list from after.
Protect the ZooKeeper credentials during migration
The controllers need to read ZooKeeper during the move, so its connection details appear in their settings. Treat those settings as secrets.
Retire the old attack surface
After finalization, close ZooKeeper’s ports and revoke its credentials. One fewer system means one fewer target.
When moving offices, you do not leave the old building unlocked with the keys under the mat. You collect the keys, cancel the badges, and lock the doors on the last day.
Monitoring and Validation
You cannot trust a migration you cannot see. Here is what to watch in each phase.
| What to watch | Why it matters | Healthy sign |
|---|---|---|
| Quorum status | Shows the leader and each voter’s progress | One leader, all voters close together |
| Migration state indicator | Tells you which stage the controller believes it is in | Matches the phase you expect |
| Under-replicated partitions | Shows whether restarts are safe | Zero before the next broker restart |
| Offline partitions | Shows real unavailability | Always zero |
| Active controller count | There must be exactly one | One |
| Broker registration and fencing | Shows brokers that lost contact | All brokers registered and unfenced |
| Client error and latency rates | Shows what applications feel | No lasting rise |
| Topic, partition, and access-rule counts | Detects anything lost in the copy | Same numbers before and after |
A before-and-after fingerprint
Before you begin, export a simple record of the cluster: the list of topics with their partition counts and settings, the list of access rules, and the list of broker settings. After each phase, produce the same record and compare. A tiny difference in these lists is a strong early warning, long before anyone complains.
Automate the fingerprint comparison. A five-minute script that prints “identical” or shows the differences will save hours of doubt during a long maintenance window.
Tools You Will Meet
A short map of the built-in tools, described by what they do rather than how they are typed.
Storage formatting tool
Prepares a controller’s metadata folder and stamps it with the cluster ID. Used once per new controller.
Metadata quorum tool
Describes the controller group: who leads, what epoch, and how far behind each voter is.
Features tool
Shows and changes the cluster’s feature levels, including the metadata version that you raise after everything is stable.
Topics and configs tools
Used for the before-and-after fingerprint and for creating test topics during dual-write.
ZooKeeper shell
Used to inspect the old data during the migration, for example to confirm the safety copy is being updated.
Controller and broker logs
Tell the story of each phase in plain messages. Keep them safe and searchable.
The metadata version step
After finalization, the cluster still has a setting called the metadata version that controls which new record types it may use. It is raised as a separate, deliberate step once you are sure you will not return to older software. Doing this too early can close a door you might still want open. Doing it much too late leaves newer features switched off.
Do not raise the metadata version during the middle of the migration. Treat it as part of “after finalization” and do it in a quiet moment.
A Runbook Template with Go/No-Go Gates
A written checklist turns a stressful night into a series of small, calm decisions.
A runbook is a step-by-step guide that anyone on your team can follow. The best runbooks have gates: small checks where you must answer “go” or “no-go” before continuing. Here is a template you can adapt to your own environment.
| Gate | Question to answer | Go when |
|---|---|---|
| Gate 0: Before starting | Are backups, rehearsal, and rollback plan complete? | All three are done and reviewed by a second person |
| Gate 1: After building controllers | Is the controller group healthy and stable? | One leader, all voters close together, no repeated elections for a set time |
| Gate 2: Before each broker restart | Is the cluster ready for one more restart? | Zero under-replicated and zero offline partitions |
| Gate 3: After the metadata copy | Does the cluster match the fingerprint? | Topic, setting, and access-rule counts are identical |
| Gate 4: End of the calm middle | Did the smoke tests pass across a full business cycle? | No unexplained errors or alerts |
| Gate 5: After all brokers switch | Are all brokers in KRaft mode and stable? | Every broker registered, unfenced, and serving |
| Gate 6: Before finalizing | Do we truly accept the point of no return? | Named decision-maker says yes, in writing |
| Gate 7: After finalizing | Is everything healthy without ZooKeeper? | Quorum healthy, fingerprint matches, dashboards live |
Pilots use checklists even though they are experts, because forgetting one item on a long, tiring day is easy. Gates do the same for your team: they replace memory with a routine.
Roles for the migration window
Assign three roles, and ideally three people. The driver types the changes. The watcher follows the dashboards and calls out anything unusual. The decider answers go or no-go at each gate and decides whether to pause or turn back. Separating the roles prevents tunnel vision, where the person doing the work misses what the graphs are saying.
Special Setups: Big, Secure, Shared, and Multi-Disk Clusters
The basic recipe works everywhere, but some clusters need extra care.
Hundreds of brokers or many partitions
The metadata copy takes longer and rolling restarts take days. Break the work into groups, pause between them, and watch the controllers’ memory and disk closely.
Encryption and strict access rules
Prepare certificates and login settings for the controllers early. Every broker needs to trust every controller before the first restart.
Many teams on one cluster
Announce the plan widely, freeze risky changes, and keep a channel open for questions. A multi-tenant cluster makes communication as important as commands.
Several data disks per broker
Support for multiple data disks in KRaft arrived after the first migration support. Confirm that your chosen version handles your layout before starting.
Zone-aware placement
Keep your rack or zone labels consistent. They are part of the metadata that gets copied, and replicas depend on them for safety.
Connectors, stream apps, schema tools
Everything that talks to Kafka should be listed. Most are unaffected, but any that read ZooKeeper directly need attention.
Moving a small apartment is one trip with a van. Moving a large office building with security passes, shared floors, and heavy equipment is a project with schedules, notices, and floor-by-floor plans. The steps are similar, but the care is far greater.
Very large clusters: work in waves
For a cluster with many brokers, divide them into groups, for example one group per zone. Move one group, wait, verify, and only then move the next. If a problem appears, it is limited to one group, and you learn a lesson before the next wave begins. Never let the speed of the first group set the pace for the rest.
If your cluster uses a special layout, such as multiple disks per broker or unusual security plugins, test exactly that layout in rehearsal. Generic rehearsals do not reveal special-case problems.
Communication and Change Management
A migration is a team activity. People who depend on Kafka should never be surprised.
Even when applications are not expected to notice anything, the people who own them will want to know that something is happening. A short, clear plan for communication prevents panic and speeds up help when it is needed.
Early notice
Tell all application owners weeks ahead: what is changing, the dates, and what (if anything) they should do. Most will need to do nothing, and hearing this is comforting.
Change freeze
Agree on a period where risky changes, such as large deployments or topic reshuffles, are paused, so problems are easier to explain.
Status channel
Open one chat channel for the migration. Post short updates at each gate: “Gate 2 passed, restarting broker 4.”
Named contacts
Publish who is on call and how to reach them. Decide in advance who has the authority to stop the migration.
Closing note
When finished, send a short summary with what changed, what to watch, and where to ask questions.
When a city repaves a road, it posts signs, sends notices, and plans detours. The paving itself is only a small part of the job. The rest is making sure nobody drives into a surprise.
Keep messages short and factual. “Migration phase 2 in progress, no action needed, next update at 15:00” is far more useful than long explanations that nobody reads at midnight.
Managed Services, Containers, and the Cloud
Where you run Kafka changes how you migrate it.
Self-managed on servers
You control everything, so you follow the phases yourself. Give each controller its own instance with a persistent disk and a stable name.
Containers with an operator
Operators such as Strimzi offer guided migration steps. Controllers need stable identities and persistent volumes, because a controller must return with its own disk.
Provider-run clusters
The provider decides what is possible. Some offer an in-place upgrade path, while others expect a new cluster and a data copy. Always confirm in the provider’s documentation.
Packaged distributions
Commercial Kafka platforms often ship their own tested migration procedures and tooling on top of the community steps. Follow the vendor’s guide for your exact version.
Moving house yourself means renting the truck, packing the boxes, and carrying the couch. Hiring a moving company means they bring their own process and you follow their checklist. Either way you reach the new home, but your work looks very different.
On container platforms, tight CPU limits and shared storage are the most common causes of unexpected controller elections. Give the new controllers steady resources and their own volumes from the very first day.
The Other Road in Detail: Blue-Green with Mirroring
Sometimes building a fresh KRaft cluster is the wiser choice.
In the blue-green approach, you create a brand new cluster that is KRaft-only from day one. Then you copy topics, settings, and data from the old cluster using a mirroring tool, and move applications across in stages. Kafka’s own mirroring tool (MirrorMaker 2), cloud replicators, and vendor cluster-linking features all fit here.
Build the new cluster
Create it on a recent version with dedicated controllers, the right security, and the right capacity.
Mirror topics and settings
Start the mirroring tool so every topic and its messages flow from old to new.
Handle consumer positions
Consumers must continue from the right place in the new cluster. Use the tool’s offset translation features and test them carefully.
Move applications in waves
Switch low-risk applications first. Observe. Then move the important ones.
Retire the old cluster
After a quiet period, stop mirroring and shut down the old cluster.
Strengths
- Rollback is trivial until the last application moves
- You can change versions, hardware, or layout at the same time
- Works where in-place migration is not offered
Weaknesses
- Costs the price of two clusters for a while
- Requires application configuration changes
- Message ordering and duplicate handling need care during cutover
Trade-Offs You Are Making
Every migration plan trades one thing for another. Say the trade-offs out loud before you begin.
Fast finish or long pause
Finishing quickly reduces the time you spend in a mixed state. Pausing longer in dual-write gives more time to notice hidden problems. Most teams choose a short mixed period but a long observation time before the final step.
Extra machines
Dedicated controllers cost money and ZooKeeper must stay online longer. That expense buys a safer path and an easier rollback.
Rehearsal time
Rehearsal takes days, but it converts unknown risks into known steps. Skipping it moves the cost to the production window.
In-place or new cluster
In-place is simpler for applications but has one-way steps. A new cluster offers easy rollback but costs more and needs application changes.
Benefits after migrating
- One system to run, secure, and monitor
- Faster recovery when the controller changes
- Room to grow partition counts
- Clear path to newer Kafka versions
Costs paid to get there
- Planning, rehearsal, and on-call attention
- New machines for controllers
- Work to update tools and dashboards
- A one-way final step that needs confidence
Design Patterns Behind a Safe Migration
The migration path uses ideas that appear in many well-run engineering projects.
Expand, then contract
First expand the system so that it supports both the old and the new (dual-write, both broker modes), and only later contract by removing the old. This lets you verify the new path while the old path still guards you. It is the same reason people install a new bridge before closing the old one.
Canary first
Change one broker first, watch it carefully, and only then continue. If something is wrong, only one machine is affected. Farmers call this testing a small patch before planting the whole field.
Single source of truth
During dual-write, only the KRaft log is the truth. ZooKeeper is a follower. Having exactly one writer of truth avoids the confusion of two conflicting records.
Checkpoint and resume
The small bookmark kept in ZooKeeper lets a replacement leader continue from where the last one stopped. Long operations become safe when they can be resumed.
Observable state machine
Each migration stage has a name that appears in logs and metrics. Clear names let humans see where the system is instead of guessing.
Fingerprint and compare
Take a record of the system before and after each step and compare them. It is the simplest and cheapest way to detect silent damage.
Best Practices, Patterns, and Anti-Patterns
Habits that make migrations boring, and boring is exactly what you want.
Practice on a copy
Migrate a development cluster, then staging, then production. Each run reduces surprises.
Own a runbook
Write the exact steps, the checks after each step, and the conditions for turning back.
One change at a time
One broker, one setting, one phase. Wait and watch before continuing.
Use the calm middle
Stay in dual-write until you feel certain. There is no prize for speed.
Name roles
Have someone who executes, someone who watches metrics, and someone who decides. Do not let one tired person do all three.
Finish the job
Retire ZooKeeper, update documents, and fix old tools. A half-finished migration hides risk.
Anti-patterns
What people do
They skip the rehearsal because “the documentation looks simple.”
Why it hurts
Every environment has quirks: unusual listener setups, odd security plugins, old scripts. Those quirks first show up during the real migration, at the worst time.
Better approach
Run the complete procedure, including a rollback, in an environment that looks like production.
What people do
They finalize the migration the same hour that the last broker was switched.
Why it hurts
Some problems only appear under daily traffic patterns or scheduled jobs. Finalizing early removes the only easy way back.
Better approach
Wait through at least one full business cycle with all brokers in KRaft mode before finalizing.
What people do
They run the new controllers on the same machines as busy brokers to save money.
Why it hurts
A traffic spike or a pause on a shared machine delays controller heartbeats and causes needless elections during the most delicate operation.
Better approach
Use small, dedicated controller machines. The extra cost is tiny compared with an outage.
Common Mistakes and Troubleshooting
A quick guide from symptom to cause to first check.
| Symptom | Likely cause | First thing to check |
|---|---|---|
| Migration never starts | One broker has not registered with the migration setting on | Controller log for missing brokers and each broker’s settings |
| Controller refuses to start | Cluster ID does not match the existing cluster | The identity file in the metadata folder versus the real cluster label |
| Broker fails after switching to KRaft mode | Old authorizer plugin still configured | The authorizer setting on that broker |
| Brokers cannot reach the controllers | Firewall, listener name, or hostname problem | Network rules, name resolution, and the controller listener mapping |
| Repeated controller elections | Slow disk, tight CPU limit, or flaky network on a controller | Disk latency, container limits, and packet loss |
| Node number conflict | A controller uses a number already given to a broker | The list of all node numbers in the cluster |
| Dashboards show frozen numbers | They still read from ZooKeeper | The data source of each panel |
| Some topics or rules seem missing | Copy incomplete or checked too early | The before-and-after fingerprint comparison |
Common mistakes in short
- Starting with an unhealthy cluster and hoping the migration will fix it
- Using an even number of controllers
- Forgetting scripts and dashboards that talk to ZooKeeper
- Restarting several brokers at once
- Skipping the backup of ZooKeeper data
- Raising the metadata version before the migration is completely finished
Real-World and Industry Examples
The same ideas have been used at organizations of every size.
LinkedIn: where Kafka began
Kafka was created at LinkedIn to move activity events. Running one of the biggest Kafka deployments in the world made the metadata scaling limits of the old design obvious early, and that experience shaped the community’s decision to build KRaft.
Confluent and managed providers
Companies that run thousands of clusters for customers have strong reasons to remove a whole extra system. Confluent has spoken publicly about moving its cloud service to KRaft, and other providers offer KRaft-mode clusters on newer Kafka versions. The migration procedures used in the community are built on the same ideas these providers tested at scale.
Netflix and Uber-style event pipelines
Streaming and ride-hailing companies push huge event streams through Kafka. For them, a migration with no planned message downtime is not a luxury but a requirement, because the pipelines feed live features. Their approach is the same as in this guide: rehearse, roll slowly, and watch every metric.
Kubernetes operator users
Teams running Kafka on Kubernetes with operators such as Strimzi follow the same phases, but the operator automates rolling restarts and setting changes. The human decisions, such as when to pause and when to finalize, remain with the team.
The lesson from other Raft systems
Kubernetes keeps its state in etcd, and Consul and CockroachDB also use Raft. Teams that have migrated coordination layers in these systems report the same pattern: the technical steps are manageable, but preparation and patience decide the outcome.
Adoption dates and product features change over time. Verify the current state in each vendor’s documentation before making decisions based on the examples above.
Life After Migration: Running KRaft Day to Day
The migration ends, but the learning does not. Here is what changes in your daily work.
New signals
Replace ZooKeeper graphs with quorum graphs: leader, epoch, commit latency, and follower lag in the metadata log.
New alerts
Alert when there is no active controller, when leadership changes repeatedly, or when a voter falls behind.
New runbooks
Teach on-call engineers what an election looks like and what to do if a majority of controllers is lost.
New upgrade order
Upgrade controllers and brokers in a planned order, one at a time, and raise the metadata version only at the end.
New capacity questions
Watch controller disk latency and memory as partition counts grow. Brokers can grow without adding controllers.
Flexible controller membership
Newer releases allow adding and removing controllers while the cluster is running, which simplifies replacing a broken machine.
After moving into a new house you still need to learn where the light switches are, change your address on the mail, and update the emergency plan. The move is not over until daily life works smoothly again.
Train the team
Hold a short session where the team practices stopping the active controller in a test environment and watches the election. Show the quorum description tool and the key graphs. People who have seen an election once are calm when it happens for real, and calm people make good decisions.
Keep a short “lessons learned” note from your migration. The next person to migrate a different cluster, or to handle an incident, will thank you.
A Migration Story: QuickBite Moves Home
The whole journey, told as one story.
QuickBite is a food-delivery company. Its Kafka cluster has nine brokers, three ZooKeeper machines, and about a thousand topics. The platform team learns that its Kafka version will soon lose support, and that the next major version has no ZooKeeper. They decide to migrate in a calm month, before any deadline pressure.
Week 1: Inventory
They list every tool that reads ZooKeeper. They find two monitoring scripts and one old dashboard. Each gets an owner. They also record the cluster ID, the topics, the access rules, and the settings as a baseline fingerprint.
Week 2: Rehearsal
They migrate a copy of the cluster in staging, including a rollback. The team finds that one broker uses the old authorizer plugin. They fix their runbook.
Week 3, Tuesday: Controllers
They build three dedicated controllers in three zones, using the existing cluster ID. The quorum stays stable for a day.
Wednesday: Enable migration
They restart the nine brokers one at a time, waiting each time for zero under-replicated partitions. After the ninth broker registers, the controller copies the metadata, and the fingerprint matches.
Thursday and Friday: The calm middle
They stay in dual-write through the weekend dinner rush. They test creating a topic and a leader change. All is well.
Next Monday: Brokers into KRaft mode
They switch the brokers one by one, including the new authorizer, and the fingerprint matches again.
Next Thursday: Finalize
After a full business cycle, they restart the controllers without the migration settings. ZooKeeper is left idle for a week, then shut down. The team updates the runbooks and celebrates with lunch.
Notice how much of the story is preparation and waiting, and how little is dramatic action. That is the sign of a well-run migration.
What Changes for Your Tools and Automation
The cluster keeps its shape, but the way you look inside it changes.
| Before (ZooKeeper world) | After (KRaft world) | What to do |
|---|---|---|
| Find the controller by reading a ZooKeeper entry | Ask the quorum tool for the current leader | Update scripts and runbooks |
| List brokers by browsing ZooKeeper | Ask the cluster through Kafka’s own admin tools | Replace direct ZooKeeper reads |
| Manage topics and settings through tools that connect to ZooKeeper | Use tools that connect to the brokers | Switch every automation to broker addresses |
| Monitor ZooKeeper health separately | Monitor the controller quorum | Retire old dashboards, build new ones |
| Backups of ZooKeeper data | Backups and protection of controller disks and configuration | Update disaster-recovery plans |
| Access rules held in ZooKeeper | Access rules in the metadata log | Confirm the authorizer is switched |
| Third-party tools that read ZooKeeper | Tools must use Kafka’s interfaces | Upgrade or replace these tools |
When a town renames its streets, the houses do not move, but every map, delivery route, and address book needs updating. If you forget one, packages go to the wrong place.
A quick audit method
Search your code repositories, automation servers, and documentation for the address of your ZooKeeper machines and for the word “zookeeper”. Each hit is a candidate to change. Also search your monitoring system for graphs and alerts that use ZooKeeper data. Doing this search early turns the “surprise old script” problem into a normal to-do list.
Some teams turn on a network log or firewall counter on the ZooKeeper port during dual-write. Any connection that is not from the cluster itself shows you a forgotten client that still depends on it.
Glossary: Every Term in One Place
A quick reference for the words used in this guide.
| Term | Simple meaning |
|---|---|
| ZooKeeper | The older outside program that stored Kafka’s metadata and helped choose a controller |
| KRaft | Kafka’s built-in way of storing and agreeing on its own metadata |
| Raft | A well-known method for a group of computers to agree on one ordered list of changes |
| Metadata | Facts about the cluster: topics, partitions, leaders, settings, and access rules |
| Metadata log | The ordered list of every change to the metadata in KRaft |
| Broker | A Kafka server that stores messages and serves clients |
| Controller | A node that holds the metadata log and, when elected, leads cluster decisions |
| Quorum | The group of controllers whose majority decides everything |
| Cluster ID | A unique label shared by every node of one cluster |
| Node ID | A unique number given to each node in the cluster |
| Dual-write | The stage where changes are written to both KRaft and ZooKeeper |
| Rolling restart | Restarting servers one at a time so the cluster keeps working |
| Graceful shutdown | Stopping a broker politely so its partition leadership moves first |
| Under-replicated partition | A partition whose copies are not all caught up |
| Offline partition | A partition with no available leader |
| Authorizer | The plugin that decides who may do what |
| Metadata version | The level of record types the whole cluster may use |
| Finalization | The last step that ends dual-write and cuts ties with ZooKeeper |
| Rollback | Returning to the previous stage when something goes wrong |
| Blue-green migration | Building a new cluster, copying data, and moving applications over |
| MirrorMaker | Kafka’s tool for copying topics between clusters |
| KIP | A Kafka Improvement Proposal, a written design for a change |
Frequently Asked Questions
Quick answers to the questions that come up most often.
Summary and Key Takeaways
The whole journey in a few short lines.
Key Takeaways
- What moves: only the cluster’s metadata and the controller job. Messages, addresses, and consumer offsets stay where they are.
- Five phases: build the KRaft controllers, enable migration on brokers, run in dual-write, roll brokers into KRaft mode, then finalize.
- Dual-write is the safety net: ZooKeeper stays up to date until the final step, so rollback remains possible before finalization.
- Finalization is the point of no return: do it only after a full business cycle of calm, with backups and fingerprints checked.
- Prepare more than you act: rehearse with rollback, take an inventory of tools that read ZooKeeper, and fix authorizer and listener details early.
- Go slowly and one at a time: restart brokers gracefully, wait for zero under-replicated partitions, and never change many things at once.
- Choose the road that fits: in-place for minimal disruption, or a new cluster with mirroring when you need a bigger change or your provider requires it.