Active-Passive Failover Design in Kafka
How to keep a second Kafka cluster ready, so that when the first one is lost, your applications carry on. Explained from zero, with everyday pictures, clear diagrams, worked examples, and a careful look at the parts that usually go wrong: lost data, changed offsets, duplicate messages, and getting back home.
Imagine a bakery with one shop and one oven. One night, a pipe bursts and the shop floods. No bread, no customers, and no way to serve anyone until the repairs are done. Now imagine the same bakery with a second, smaller shop across town. Every evening, the owner sends copies of the day’s recipes and orders to the second shop. When the first shop floods, the staff walk to the second shop, open the doors, and keep serving. That second shop, which sits ready but is not used day to day, is the heart of an active-passive design.
Kafka is often the central nervous system of a company: orders, payments, sensor readings, and user activity all flow through it. If a whole Kafka cluster becomes unavailable because a data center loses power, a cloud region has an outage, or someone makes a terrible mistake, many programs stop at once. Kafka is built to survive the loss of a machine or even a rack, but a single cluster cannot survive the loss of everything it runs on.
The classic answer is to run a second cluster in another place, copy data to it continuously, and be ready to switch. That sounds simple. In practice, it hides many subtle questions. How much data might be lost? How long does the switch take? Are message positions the same in both clusters? What happens to programs that were in the middle of a job? How do you get back afterwards? This guide answers each of these from the very beginning, and assumes no earlier knowledge.
Why Failover Design Matters
Failures are not rare events. They are scheduled visitors that just have not arrived yet.
Every system eventually meets a failure larger than it was built to handle. Hardware breaks, networks are cut by a digger, cooling fails, a cloud provider has a regional problem, a wrong command deletes the wrong thing, or a software bug corrupts data. A team that has planned for it treats the day as an inconvenience. A team that has not planned for it treats the day as a crisis.
Keep the company running
When orders, payments, or alerts stop, money and trust are lost by the minute.
Meet recovery promises
Many industries must show that they can recover from a site loss within a stated time.
Replace panic with a checklist
A rehearsed procedure turns a chaotic night into a sequence of known steps.
Choose the right level
Not every topic needs the same protection. Design lets you spend where it counts.
A fire escape plan is worthless if you first read it while the building is burning. Failover design is the escape plan, drawn and practiced long before it is needed.
Failover is not a feature you switch on. It is a design made of copying, translating, deciding, switching, and returning. Each part must work, and the parts must work together.
Kafka Basics You Need First
Just enough vocabulary to follow the rest.
| Word | Plain meaning | Picture |
|---|---|---|
| Message (record) | One piece of data, such as “order 1234 placed” | A single letter |
| Topic | A named stream of messages | A labelled mailbox |
| Partition | An ordered slice of a topic, stored as files on a broker | One queue at the post office |
| Offset | The position number of a message in a partition | The line number on a page |
| Broker | A Kafka server | A post office building |
| Cluster | A group of brokers working together | A whole postal network in one region |
| Producer | A program that writes messages | The sender |
| Consumer | A program that reads messages | The reader |
| Consumer group | Readers that share the work of a topic and save one shared bookmark | A team dividing up a stack of mail |
| Committed offset | A saved bookmark showing how far a group has read | A bookmark in a book |
| Replica | A copy of a partition on another broker in the same cluster | A photocopy in another office of the same company |
Think of a cluster as one postal region. Inside the region, every parcel is kept in three post offices. If one office burns down, the region carries on. But if the whole region is hit by a flood, all three offices are gone together.
Two very different kinds of copies
Copies inside a cluster
Replicas on different brokers, managed by Kafka itself. They protect against loss of a machine or a rack. They are usually updated together with the write, so the copies are tightly in step.
Copies between clusters
A separate program reads from one cluster and writes into another. They protect against the loss of a whole site. They are usually updated a little later, so the copies are loosely in step.
The second kind is what this guide is about. It looks similar to the first, but it behaves very differently. The differences in timing, positions, and ownership cause almost every difficulty in failover design.
What Active-Passive Means
One cluster works. The other waits, and watches, and copies.
In an active-passive design, two Kafka clusters exist in two locations. Only one, the active or primary cluster, receives writes from producers and serves consumers. The other, the passive or standby cluster, receives a continuous copy of the data and does no other work for applications. When the primary is lost, the standby is promoted, and applications are pointed to it.
| Design | Who accepts writes | Simple picture |
|---|---|---|
| Active-passive | Only one cluster at a time | A main shop and a spare shop that opens only in an emergency |
| Active-active | Both clusters, each for different or the same data | Two shops both open, each serving its own customers and sharing news |
| Stretch cluster | One cluster spread over several sites | One shop with rooms in three buildings |
A hospital has a main generator and a standby generator. The standby is never used on normal days, but is tested, fueled, and ready. Nobody argues about who is in charge, because only one generator feeds the building at a time.
Why the simple rule is valuable
Allowing writes to only one place at a time avoids a huge class of problems: two clusters accepting conflicting versions of the same data, messages appearing in different orders, and confusion about which cluster tells the truth. The price is that the standby’s machines sit mostly idle and there is a moment of switching. Many teams happily pay that price for simplicity.
“Passive” means passive for applications, not idle. The standby is busy every second receiving copies, and is often also used to run tests and rehearsals.
What Can Go Wrong: Types of Failures
Different failures need different protection, and one kind cannot be solved by copying.
| Failure | Example | Protected by in-cluster copies? | Protected by a standby cluster? |
|---|---|---|---|
| Machine or disk failure | A broker’s disk dies | Yes | Not needed |
| Rack or zone failure | One data hall loses power | Yes, if copies are spread over zones | Not needed |
| Whole site or region failure | A regional cloud outage or a fire | No | Yes |
| Cluster-wide software problem | A bad upgrade breaks all brokers | No | Yes, since the standby can run the previous version or a different configuration |
| Network isolation | The site is up but nobody can reach it | No | Yes, but with care about which side is right |
| Human error | Someone deletes the wrong topic | No, the deletion is copied to all replicas | Partly: the deletion may also be copied, unless replication ignores deletes |
| Data corruption by a bug | A faulty program writes wrong data | No, all copies are wrong | No, the wrong data is copied too |
A backup shop protects you when the first shop floods. It does not protect you when the owner writes the wrong prices into the recipe book, because the wrong prices are faithfully copied to the second shop every evening.
A failover design is not a backup. It keeps you running through the loss of a site. It does not protect against wrong data, wrong deletions, or a bad program. Those need other tools, such as careful access control, retention settings, and separate archives.
RPO and RTO: The Two Numbers That Drive Everything
Before designing, decide how much loss and how much waiting the business can accept.
| Term | Full name | Meaning | Question it answers |
|---|---|---|---|
| RPO | Recovery point objective | The maximum amount of recent data you accept losing, measured as time | “How far back in time will we be after the switch?” |
| RTO | Recovery time objective | The maximum time from the failure until service works again | “How long will we be down?” |
You save your homework to a cloud drive every 10 minutes. If your laptop breaks, you lose at most 10 minutes of writing: that is the RPO. Then it takes 30 minutes to borrow a laptop and open the file: that is the RTO.
What sets each number in Kafka
Replication lag
Copying between clusters is delayed. Whatever has not yet arrived at the standby when the primary is lost is gone. So the RPO is roughly the replication lag at the worst moment, often a few seconds, but larger during heavy load or network trouble.
Detection plus decision plus switch
Time to notice the failure, time for humans or automation to decide, time to promote the standby and repoint applications, and time for applications to reconnect and warm up.
An RPO of exactly zero cannot be reached with ordinary copying between clusters, because the copy always arrives a little after the original. Zero requires writes to be confirmed by both sites before the producer is told “done,” and that has a cost in speed and complexity, which we discuss when comparing options.
Different topics, different needs
Not all data is equally precious. Payment events may need a tiny RPO and fast RTO. Click logs may tolerate minutes of loss. Grouping topics by need lets you protect the important ones fully and the others lightly, saving cost and network capacity.
Choosing the Pattern: Stretch, Active-Passive, or Active-Active
Three ways to survive a site loss, each with a different price.
flowchart TB
Q1{"Are your sites close together with fast, reliable links - a few milliseconds?"}
Q1 -->|"Yes"| Q2{"Do you need zero data loss and automatic recovery?"}
Q1 -->|"No - distant regions"| Q3{"Must both regions accept writes at the same time?"}
Q2 -->|"Yes"| S["Stretch cluster over three sites - synchronous copies, automatic leader change"]
Q2 -->|"No"| AP1["Active-passive is enough and simpler"]
Q3 -->|"Yes"| AA["Active-active - two clusters, conflict and loop rules needed"]
Q3 -->|"No"| AP2["Active-passive - one writer, one standby"]
| Pattern | Data loss on site failure | Recovery | Main cost | Best when |
|---|---|---|---|---|
| Stretch cluster | Can be zero, since copies are written synchronously to several sites | Automatic, using normal leader election | Needs fast links; every write waits for a distant site; needs three sites to keep a majority for the controllers | Sites are metro-distance apart |
| Active-passive | Small, equal to replication lag | Manual or scripted switch | Second cluster mostly idle; switch and return procedures | Regions are far apart, one writer is fine |
| Active-active | Small, per direction | Clients move between clusters | Must design for duplicates, conflicts, and loops | Users worldwide need local writes |
A stretch cluster is one company with offices across a city that talk by intercom in real time. Active-passive is a head office and a backup office in another country, exchanging a nightly courier bag. Active-active is two head offices, each running things, updating each other constantly.
Most organizations that span continents choose active-passive for the majority of their data, because it delivers strong protection with the fewest surprises. That is why we focus on it here.
The Architecture of Active-Passive Kafka
Two clusters, one replication pipeline, and a way to point clients at the right side.
flowchart LR
subgraph PRI["Primary site - active"]
P1["Producers"] --> PC[("Primary Kafka cluster")]
PC --> C1["Consumers"]
end
subgraph LINK["Replication pipeline"]
MM["Replication service - reads from primary and writes to standby - topics, settings, permissions, bookmarks"]
end
subgraph STB["Standby site - passive"]
SC[("Standby Kafka cluster")]
C2["Consumers - stopped"]
P2["Producers - stopped"]
end
PC -->|"1 read new data"| MM
MM -->|"2 write copy"| SC
DNS["Client routing - DNS, config service, or proxy"] -.-> PC
DNS -.-> SC
| Part | Job | Where it should run |
|---|---|---|
| Primary cluster | Serves all producers and consumers | Site A |
| Standby cluster | Receives copies, ready to be promoted | Site B, in a different failure domain |
| Replication service | Copies data, settings, permissions, and consumer bookmarks | Usually next to the standby, so it survives a primary loss |
| Client routing layer | Tells applications which cluster to use | Independent of both clusters |
| Monitoring and decision layer | Watches health and starts the switch | A third place, not inside either site |
A newspaper has a main office and a backup office. A courier service carries copies of every page to the backup. A road sign system tells reporters which office to go to. And a duty editor, sitting somewhere else, decides when to switch.
Place the things you need during a disaster outside the disaster. A monitoring tool that lives only in the failed site cannot tell you the site has failed.
MirrorMaker 2: The Standard Replication Tool
A tool that ships with Kafka and does the copying between clusters.
MirrorMaker 2 is the cross-cluster replication tool included in Apache Kafka. It is built on Kafka Connect, a framework for running data-moving jobs as reliable, restartable workers. Because it is a set of ordinary Kafka Connect jobs, it can be scaled by adding workers, and it stores its own progress inside Kafka.
| Component | What it does | Everyday picture |
|---|---|---|
| Source connector | Reads messages from topics on the source cluster and writes them to the target cluster; also copies topic settings and permissions | The courier who carries parcels between offices |
| Checkpoint connector | Records how consumer group bookmarks on the source map to positions on the target, and can write translated bookmarks for groups | A translator who converts page numbers between two editions of a book |
| Heartbeat connector | Sends small regular messages through the pipeline so you can measure whether it is alive and how delayed it is | A pulse monitor |
| Kafka Connect workers | Run all the above jobs, share the load, and restart failed tasks | The staff who run the courier service |
Think of a translation and shipping department: one team ships the books, one team maintains a table converting page numbers between editions, and a third team sends a “still here” note every minute.
Where should it run?
Best practice is to run the replication service close to the target cluster. That way, it “reads remotely and writes locally.” If the network breaks in the middle, reading fails, but writing to the local target is safe, and the service is still alive to resume. Also, after the primary site is lost, the replication service is not lost with it.
Other tools exist. Some vendors provide their own linking features that copy data byte for byte, keeping offsets identical. We compare the two approaches in a later chapter. The design questions in this guide apply to both.
What Gets Copied, and What Does Not
A full failover needs much more than the messages.
| Item | Copied? | Notes |
|---|---|---|
| Messages in selected topics | Yes | Chosen with allow and deny lists of topic names |
| Topic settings (retention, partition count, and others) | Yes, on a refresh interval | New partitions are noticed with a delay |
| Topic access rules (ACLs) | Optionally | Should be checked, since users on the standby must have the same rights |
| Consumer group bookmarks | Yes, translated | Copied only for groups that are not active on the target |
| Message positions (offsets) | No, they are new on the target | Positions differ, so they must be translated |
| Transactions as units | No | The atomic grouping is not preserved on the standby |
| Schemas and the registry | Not by this tool | Needs its own replication |
| Users, credentials, quotas | Not by this tool | Managed by your own automation |
| Broker settings and cluster identity | No | Each cluster has its own; keep them consistent by configuration management |
| Application state outside Kafka, such as databases | No | Needs its own failover plan |
Moving to a backup office means not only moving the paperwork, but also copying the access badges, the phone directory, the filing rules, and the keys. If only the paperwork travels, staff stand outside a locked door.
The most common surprise on the night of a failover is not missing messages. It is missing permissions, missing schemas, or a missing setting on the standby. Build a checklist of everything an application needs, not only the data.
Topic Naming on the Standby
By default, copied topics get a prefix, and that changes how applications must be built.
By default, MirrorMaker 2 names each copied topic with the name of the source cluster in front. A topic called “orders” on the cluster called “primary” becomes “primary.orders” on the standby. This avoids collisions and makes it obvious where a topic came from. It also prevents an infinite loop in two-way setups, since the tool ignores topics already marked with the target’s own name.
| Policy | Result on standby | Good for | Trade-off |
|---|---|---|---|
| Default prefix policy | “primary.orders” | Active-active and clear origin tracking | Applications must be told to read a different name after failover |
| Identity policy (keep the same name) | “orders” | Pure active-passive, where the standby only ever holds copies of one source | Not safe for two-way copying, because loops become possible |
A backup office can label shipped folders “from Head Office – Orders,” which is clear but means staff must ask for a different folder name. Or it can file them under the plain name “Orders,” which is convenient, as long as only one head office ever sends folders.
How this shapes your application design
With prefixed names
Consumers should subscribe by pattern, for example any topic ending in “orders,” so they read the local topic on the primary and the prefixed copy on the standby without changing code. During failover, the producers write to the plain name on the new active cluster, and consumers read both.
With identical names
Applications need no changes at all. They only change the address of the cluster. This is the simplest design for one-way protection, and is widely used.
Decide the naming approach at the start. Changing it later means changing applications and moving data. If active-passive is your goal and you never plan to copy in both directions, identical names are the friendlier choice.
Replication Lag: The Window of Possible Loss
Copying takes time, and whatever is in transit when disaster strikes is lost.
The replication service reads a message from the primary, sends it over the network, and writes it to the standby. Between the moment a producer’s write is confirmed on the primary and the moment the copy is confirmed on the standby, there is a gap. That gap, measured in time or in number of messages, is the replication lag. If the primary vanishes at the wrong moment, the messages in that gap are gone from the standby’s point of view.
Producer writes, primary confirms
The producer believes the message is safe.
Replication service reads it
A moment later, sometimes a fraction of a second, sometimes several seconds if it is busy.
Copy crosses the network
Long distances add tens or hundreds of milliseconds, and congestion adds more.
Standby writes it
Only now is the message safe against loss of the primary site.
| What raises lag | Why | What helps |
|---|---|---|
| Bursts of traffic | The pipeline cannot copy as fast as messages arrive | More replication tasks and network capacity |
| Long distance and small network windows | Each round trip takes longer | Larger batches and tuned network buffers |
| Too few tasks or partitions | Copying is parallel per partition | Match the task count to partition count and load |
| Slow standby brokers | Writes to the standby take longer | Size the standby like the primary |
| Compression mismatch and other conversions | Extra processor work in the pipeline | Consistent compression settings |
| Network trouble between sites | Data waits in the primary | Redundant links and monitoring |
A courier who collects papers every ten minutes cannot save the pages typed in the last nine minutes if the office burns down. Making the courier faster or more frequent shrinks the loss window, but never closes it fully.
The worst lag happens at the worst time: during traffic peaks, when links are stressed. Measure lag at peak, not only on quiet days, and use that number when you promise an RPO.
Why Offsets Differ Between Clusters
The same message can sit at different positions in the two clusters.
In the primary, a message might be number 5,000 in a partition. When the replication tool copies it, the standby’s partition assigns its own position number. The standby may have started later, may have had gaps because some messages were skipped, may have lost older data to retention, or may include control markers from transactions on the primary that do not appear in the copy. So the number on the standby is often different, for example 4,720.
| Reason offsets drift | What happens |
|---|---|
| Replication started after the topic already had data | The standby’s first message gets its own first position, while the primary’s position was already high |
| Retention removed old data at different times | The two clusters start their logs from different points |
| Transaction markers and compaction gaps on the primary | The primary’s numbering includes positions that hold no data messages |
| Data was re-copied, or the pipeline was reset | New positions are assigned again |
| Topic recreated on the standby | The numbering begins again at zero |
Two editions of the same novel are printed with different page sizes. “The duel scene” is on page 212 in one edition and page 187 in the other. If a reader’s bookmark says page 212, giving that reader the second edition open at page 212 puts them in the wrong place.
If a consumer group simply keeps its old bookmark number after switching clusters, it may jump ahead and skip messages, or jump back and replay a huge amount. Translation is not optional.
Consumers care, producers do not
Producers only append, so they never need to know the position numbers. Consumers depend on saved positions, so offset differences matter for them. That is why this whole part of the design is about readers.
Offset Translation: How the Bookmark Follows You
A small table, kept up to date, converts positions from one cluster to the other.
flowchart LR
subgraph PRI["Primary cluster"]
T1[("orders partition 0 - message at position 5000")]
G1[("Consumer group bookmark - position 4990")]
end
subgraph MM["Replication service"]
OS["Offset sync records - primary 5000 equals standby 4720"]
CP["Checkpoint - group bookmark 4990 translates to about 4710"]
end
subgraph STB["Standby cluster"]
T2[("orders partition 0 - same message at position 4720")]
G2[("Group bookmark written for the standby - 4710")]
end
T1 -->|"copy"| T2
T1 -.->|"remember pairs of positions"| OS
G1 -.->|"read bookmark"| CP
OS --> CP
CP -->|"write translated bookmark"| G2
Record matching pairs
While copying, the tool notes “primary position 5,000 became standby position 4,720” and stores such pairs in an internal topic. It does so for some messages, not every one, to save space.
Read the group’s bookmark
The checkpoint job looks at each watched consumer group’s saved position on the primary.
Convert with the nearest earlier pair
It finds the latest recorded pair at or before the group’s position and calculates the matching spot on the standby. Because it uses an earlier pair, the answer is safe: at or slightly before the true position.
Save the translated bookmark
The result is stored as a checkpoint and can also be written straight into the standby’s group bookmarks, if configured to do so.
A translator keeps a small list: “Chapter 5 starts on page 100 in edition one and page 87 in edition two.” For a bookmark on page 108, she starts from Chapter 5, counts eight pages in, and says “page 95 in edition two.” She rounds toward earlier pages so nothing is skipped.
Because the translation rounds toward earlier positions, a switched consumer may re-read a few messages it had already processed. This is why failover consumers must tolerate duplicates.
Synchronizing Consumer Group Bookmarks
Keeping the standby’s bookmarks fresh so that consumers can resume with almost no gap.
Newer versions of the replication tool can automatically write the translated bookmarks into the standby cluster’s own consumer group records, on a regular schedule. When consumers later start on the standby, they find a bookmark that already points to the right area, and continue as if nothing happened. This feature is limited to groups that are not currently active on the standby, because Kafka does not allow the bookmarks of a running group to be overwritten from outside.
| Setting idea | What it controls | Design impact |
|---|---|---|
| Automatic group bookmark sync | Whether translated bookmarks are written to the standby | Turn on for active-passive, so that failover is quick |
| Sync interval | How often bookmarks are refreshed | The interval adds to the amount of re-reading after failover; shorter means less duplicate work |
| Checkpoint interval | How often translations are calculated | Same idea; a shorter interval means fresher translations |
| Group allow list | Which groups are synchronized | Include every group that must fail over, and exclude the rest to reduce work |
| Offset sync frequency | How often matching position pairs are recorded | More pairs give finer accuracy and fewer duplicates |
A reader’s bookmark is photocopied every minute and slipped into the backup edition. If the reader must switch books at any moment, the bookmark in the backup is at most a minute out of date.
Do not start your consumers on the standby “just to test” while the standby’s bookmarks are being synchronized. An active group on the standby blocks the sync for that group, and can even start reading from the wrong place. Test in a way that uses a different group name, or test during a planned drill.
Fresh does not mean exact
Even with the tightest settings, there is a lag between what the group did on the primary and what the standby’s bookmark says. The right expectation is “resume from a little before where we stopped.” Plan for that overlap of duplicate messages, rather than hoping it will be zero.
The Failover Sequence Step by Step
A well-ordered switch, from “we lost the primary” to “we are running on the standby.”
sequenceDiagram
participant MON as Monitoring and decision layer
participant OPS as Operators or automation
participant PRI as Primary cluster
participant REP as Replication service
participant STB as Standby cluster
participant APP as Applications
MON->>OPS: Primary unreachable beyond agreed limit
OPS->>OPS: Confirm it is a real outage, get approval
OPS->>PRI: Fence - block or stop clients if reachable
OPS->>REP: Stop replication and note last copied position
OPS->>STB: Check that all needed topics, permissions, and settings exist
OPS->>STB: Make standby the active cluster - create missing topics, enable client access
OPS->>APP: Switch client routing to the standby
APP->>STB: Producers connect and write
APP->>STB: Consumers connect and resume from translated bookmarks
MON->>OPS: Confirm healthy, record time and lost-data estimate
Detect and confirm
Several independent checks agree that the primary is not working, not just one failing probe.
Decide
A named person or a well-tested automation makes the call, according to a written rule.
Fence the old primary
Make sure that no application can still write to it, so that two clusters do not both accept writes.
Stop replication
Halt the pipeline, and record how far it had got, which tells you what may have been lost.
Prepare the standby
Verify topics, settings, permissions, schemas, and quotas. If the standby was set to keep copies under a prefix, create the plain topics that producers will write to.
Repoint clients
Change routing so that applications reach the standby.
Start or resume applications
Producers first or consumers first depending on the design, with consumers resuming from translated bookmarks.
Verify and communicate
Check that data is flowing, lag is falling, and business checks pass. Tell everyone what the system state is.
An orderly evacuation to a backup office: first make sure nobody remains in the burning building, then count what documents made it, then open the backup office, then call everybody’s phones and send them there, and finally take attendance.
Pointing Clients at the Right Cluster
The routing layer is a small piece with a huge effect on the recovery time.
Every Kafka client starts with a list of “bootstrap” addresses, which it uses to find the rest of the cluster. To switch clusters, those addresses must change, or must lead somewhere new. There are several ways to do this, each with a different speed and risk.
| Method | How it works | Speed of switch | Risk |
|---|---|---|---|
| DNS name change | A name such as “kafka.company” points to the active cluster’s addresses, and is repointed during failover | Depends on the record’s lifetime and on how long clients cache the address | Clients that cached the old address, or that keep long-lived connections, may keep talking to the old side |
| Configuration service | Applications read cluster addresses from a central configuration store and reload on change | Fast if applications watch for changes | Applications that read it only at start need a restart |
| Load balancer or proxy layer | A stable front address forwards traffic to the active cluster | Very fast | Kafka clients need each broker’s address, so proxies must be designed carefully; adds a component to operate |
| Client-side dual list with rules | Applications know both clusters and choose the active one from a shared signal | Fast | Every client must implement the logic correctly |
| Redeploy with new settings | Change the configuration and restart | Slowest | Depends on deployment speed, but is simple and explicit |
Telling customers a shop has moved can be done by a changed sign on the door (fast if they visit), an announcement in the newspaper (slow for those who do not read it), or a phone operator who redirects calls (quick and reliable, but the operator must be present and correct).
After brokers answer the first request, Kafka clients switch to the addresses the brokers announce, not the original bootstrap address. So each broker must announce names that resolve correctly for that cluster. A cluster that announces private names unreachable from clients will fail even when the front address works.
Practice, do not assume
The time it takes for every application to actually connect to the standby is often the largest piece of the RTO. Measure it in rehearsals, including the slowest and oldest applications, the ones nobody wants to touch.
Producers During and After Failover
What a writing program experiences when the cluster changes under it.
Before the switch
Sends fail or time out. The producer retries and holds messages in its memory buffer. If the outage is long, the buffer fills and the application blocks or errors.
During the switch
Applications are restarted or reconnect to the new address. Any messages that were only in the memory buffer of a restarted program are lost unless the application stores them elsewhere.
After the switch
The producer writes to the standby, which now behaves as the primary. Its idempotence information, which prevents duplicates within a cluster, starts fresh, because the new cluster has never seen this producer.
| Problem | Why it happens | Design response |
|---|---|---|
| Messages lost from memory buffers | Producers hold unsent messages in memory only | Keep a durable record at the source for critical events, or accept the loss window |
| Long blocking or errors | The buffer fills while the cluster is unreachable | Set sensible time limits, and use a fallback such as local queuing for critical writers |
| Duplicate messages after retry across clusters | A write may have reached the old cluster and been copied, and then be sent again to the new one | Give messages a unique ID so downstream systems can drop duplicates |
| Writes to the wrong cluster | Some applications keep old addresses | Fencing and access rules on the old cluster |
| Topics missing on the standby | Topics created after the last configuration sync | Auto-create disabled, plus a pre-check step in the runbook |
A clerk who keeps the day’s orders on sticky notes on the desk loses them if the desk is evacuated quickly. A clerk who also logs each order in a notebook can rebuild the day after the move.
Design messages to carry a unique ID from the start. Almost every failover problem with duplicates becomes small when the receiver can recognize a message it has already handled.
Consumers After Failover: Living with Duplicates
The safe rule is: read a little too much, never too little.
After failover, a consumer group resumes from the translated bookmark, which is at or just before the correct point. That means messages processed shortly before the failure may be processed a second time. This is the normal outcome of “at-least-once” processing: nothing is skipped, but some things may repeat.
| Consumer type | What duplicates cause | How to handle it |
|---|---|---|
| Updates a database by key (upsert) | Nothing harmful, the same update is applied twice | Nothing extra needed |
| Adds to a counter or sends an email | Double counts or double emails | Record processed message IDs and skip repeats |
| Calls a payment or other outside service | Possible double charge | Use the service’s own idempotency keys |
| Builds derived data in Kafka | Repeated output records | Make the output idempotent, or dedupe downstream |
A proofreader who lost her place after moving desks starts a few pages before where she stopped. She might re-read a page or two, but she will never skip a chapter.
The dangerous bookmark settings
Reset to the latest
If no valid bookmark exists on the standby and the group is told to start from the newest message, every message between the failure and the restart is silently skipped.
Reset to the earliest
If no bookmark exists and the group starts from the very beginning, it replays the entire retained history, which can flood downstream systems.
Verify bookmarks before starting
Part of the runbook is to list the group’s bookmarks on the standby and confirm they exist and are recent.
Pick the reset policy on purpose
Choose per application, based on which is worse: skipping or replaying.
A group name that differs on the standby, a group missing from the allow list, or a group that was created after the last sync will start with no bookmark. Check every group before opening the gates.
Transactions and Exactly-Once Across Clusters
The strong guarantees of one cluster do not travel with the data.
Within one cluster, Kafka can offer transactions: a group of writes that becomes visible all together or not at all, and exactly-once processing between topics. These guarantees depend on coordination information stored inside that cluster. The replication tool copies message contents, but it does not copy that coordination state or keep the messages together as one transaction on the other side.
| Guarantee inside one cluster | What survives copying | What is lost |
|---|---|---|
| Messages of a transaction appear together | Only committed messages are copied, as ordinary messages | The all-or-nothing grouping on the standby; a failure in the middle of copying may leave part of a group copied |
| Producer identity and duplicate protection | Nothing | The standby has no record of the producer’s numbering |
| Exactly-once from input topic to output topic | Data of both topics, if both are copied | The link between the input bookmark and the output |
| Open transactions at the time of failure | Aborted or open ones are not exposed to copying | They simply vanish, which is correct: they never committed |
A bank clerk’s paperwork stapled together as “transfer, fee, and receipt” is photocopied page by page to the backup office. If the fire strikes in the middle, the backup may hold the transfer page but not the receipt page.
Assume at-least-once after failover
Treat everything on the standby as ordinary data that may repeat.
Make consumers idempotent
Design processing so that repeating a message has no extra effect.
Reconcile after the switch
Run a check that compares business totals between systems, to catch any partial group.
Restart stream processing with clean plans
Programs that keep internal state may need to rebuild it from the copied change-log topics.
If your business logic needs the strongest guarantees across a site loss, the choice is a stretch cluster within metro distance, or application-level protection such as unique IDs and reconciliation.
Ordering and Partition Layout
Order inside a partition survives copying, but only if the partitions match.
Kafka promises the order of messages within one partition. The replication tool writes each message to the same partition number on the standby as on the primary, so the order for every key is kept, provided the standby topic has at least as many partitions as the primary, and the same key-to-partition mapping still applies. Order across different partitions was never promised, and remains unpromised.
| Situation | Effect | What to do |
|---|---|---|
| Partition counts equal | Same key goes to same partition on both sides; order kept | Keep counts in sync |
| Primary gained partitions later | Standby learns of them on a refresh delay; copies of new partitions may wait or misplace | Watch for changes, and use short refresh intervals |
| Standby was created with a different count | Keys may map to different partitions after failover, and per-key order breaks across the switch | Create topics with identical counts, on purpose |
| Custom partitioner in producers | Must give the same result after failover | Keep the same logic in all clients |
If both offices have the same pigeonholes numbered the same way, “Green Street” letters go into hole seven in both. If the backup office has more holes, the same street is sent to a different hole, and the clerk who expects them in hole seven is confused.
Treat partition counts as part of the contract between the two clusters. Compare them automatically every day, and raise an alert when they differ.
The Other Things Your Applications Depend On
Kafka is rarely alone. Failover fails when the friends are forgotten.
| Dependency | Why it matters at failover | What to plan |
|---|---|---|
| Schema registry | Messages refer to schema numbers. Without the same schemas on the standby, readers cannot decode data | Replicate schemas, and keep the numbering identical |
| Kafka Connect and other data movers | Their own progress and settings live in internal topics | Decide whether to copy those topics, or restart connectors from a known point |
| Stream processing programs | They keep state in local stores backed by change-log topics | Copy the change-log topics too, or rebuild state after the switch |
| Databases and caches | Their own failover must line up with Kafka’s | Design one coordinated switch, with an order of steps |
| Identity and secrets systems | Applications need credentials valid on the standby | Pre-create and test credentials on both sides |
| Monitoring and alerting | You need eyes on the standby when it takes over | Monitor both clusters at all times |
| Network rules and certificates | Encrypted connections require valid certificates for the standby’s names | Certificates issued and renewed for both |
A backup kitchen is useless if the recipes were copied but the supplier accounts, the keys to the fridge, and the health certificate stayed behind in the burnt kitchen.
The weakest link in a failover is often not Kafka. Draw a complete map of everything your critical applications need, and test the whole chain, not just the cluster.
Split-Brain and Fencing
The worst outcome is two clusters both believing they are in charge.
Suppose the network between sites breaks. The primary is still running, and applications near it still write to it. Meanwhile, the operators cannot reach it, decide it is dead, and promote the standby. Now both clusters accept writes, with different data in each. This situation is called split-brain. When the network is repaired, the two histories cannot be merged neatly, and some data must be thrown away or reconciled by hand.
Use more than one signal
Decide a site is dead only if several independent observers agree, from more than one location.
Fence the old side
Cut it off so that it cannot accept writes even if it is alive: revoke the applications’ credentials, change firewall rules, shut down the brokers, or stop the routing to it.
Keep a single source of truth for “who is active”
A small, reliable service outside both sites records which cluster is the active one, and applications check it.
Make the rule explicit
A written rule says who may declare a failover, and under what evidence.
Two managers both believe they are in charge of the same shop, each giving orders to different staff. Everyone works hard, and by evening the stock records disagree. The cure is one clear rule about who is in charge, and a way to tell the other manager to stand down.
Automatic failover without reliable fencing is more dangerous than a slow manual one. A false alarm, such as a short network glitch, can trigger a switch that creates split-brain. Many teams choose a human decision, backed by good tooling, for exactly this reason.
The Life Cycle: States of an Active-Passive Pair
A clear list of states makes it obvious what is allowed at each moment.
stateDiagram-v2
[*] --> Normal
Normal --> Degraded : replication lag or link trouble
Degraded --> Normal : lag recovered
Normal --> FailingOver : primary lost - decision made
Degraded --> FailingOver : primary lost - decision made
FailingOver --> StandbyActive : standby promoted and clients switched
StandbyActive --> Rebuilding : old site returns - reverse replication started
Rebuilding --> Synchronized : old site caught up
Synchronized --> FailingBack : planned switch back
FailingBack --> Normal : original primary active again
Synchronized --> StandbyActive : postpone failback
| State | Who accepts writes | Replication direction | What is safe to do |
|---|---|---|---|
| Normal | Primary | Primary to standby | Everything routine, and rehearsals |
| Degraded | Primary | Slow or stopped | Fix the pipeline; avoid risky changes; RPO is currently worse |
| Failing over | Nobody for a short time | Stopped | Only runbook steps |
| Standby active | Former standby | None yet | Stabilize, then plan the return; you have no protection now |
| Rebuilding | Former standby | New active to old site | Wait for the old site to catch up or be rebuilt from scratch |
| Synchronized | Former standby | New active to old site, in step | Choose the time for failback, or stay |
| Failing back | Nobody for a short time | Stopped, then reversed | Runbook steps again, mirrored |
A relay race has exactly one runner holding the baton at any time, and rules for how the baton is handed over. Whenever the team is unsure who holds the baton, the race is in trouble.
Notice “Standby active.” Once you have failed over, you have no standby any more. Every hour in that state is an hour without protection, so building a new standby is urgent.
When to Fail Over: Detection and Decision
Failing over too early is nearly as harmful as failing over too late.
| Signal | What it tells you | How reliable alone |
|---|---|---|
| Broker health checks fail from the monitoring site | Brokers not answering from there | Low: could be a network problem on the monitor’s side |
| Checks fail from several independent locations | The cluster is unreachable from many places | Medium to high |
| Heartbeat messages stop arriving at the standby | The pipeline or the primary has stopped | Medium: could be only the pipeline |
| Producer error rates spike in applications | Real users are affected | High for impact, unclear for cause |
| Cloud provider or data center status notice | Known outage | High, though the estimated duration is often uncertain |
| Business metrics fall (orders per minute) | The company feels it | High for impact |
A shop owner does not close and move after a single missed phone call. She checks the street, asks neighbors, and calls the electricity company. Only when several signs agree does she move to the backup shop.
Automatic or human?
Safer against false alarms
A trained person reviews evidence, and approves. Slower, but avoids split-brain from glitches. Best when the outage rate is low and the business can accept several minutes.
Faster recovery
Scripts run the steps on a trigger. Needs reliable fencing and strict conditions. Best when the RTO is extremely short.
Automate the steps, human pushes the button
The whole runbook is one command, with checks built in, but a person approves it. Common and effective.
Write down thresholds
“If the primary is unreachable from three of four probes for five minutes” is a rule. “If it looks bad” is not.
Flapping is a real danger: a system that fails over, then fails back at the first sign of recovery, then fails over again. Add a hold-down period, and make each direction of the switch a deliberate decision.
The Failover Runbook
A short, exact list that a tired person can follow at three in the morning.
Declare and record the time
Open an incident, note the start time, and name the person in charge.
Freeze changes
Stop deployments and configuration changes on both sides.
Capture replication state
Record the last heartbeat time and last copied positions, for the loss estimate later.
Fence the old primary
Apply the agreed method and confirm that it worked.
Stop the replication service
Prevent it from writing anything further to the standby.
Run the readiness checks on the standby
Healthy brokers, all topics present with the right partition counts, permissions in place, schemas available, disk space, and quotas.
Verify consumer bookmarks
List the bookmarks of critical groups on the standby, and confirm each exists and is recent.
Switch the routing
Change the address or the routing layer, and confirm the change with a test client.
Start applications in the agreed order
Usually the consumers of the most critical topics first, then the producers, or the reverse if the design demands.
Verify business health
Check that messages flow, lag falls, and business numbers look normal.
Announce and start rebuilding protection
Tell everyone the system is on the standby and unprotected, and begin planning the new standby.
An airline pilot runs a printed checklist even after thousands of flights, because under stress memory fails. A runbook is the pilot’s checklist for a site loss.
A runbook is only as good as its last rehearsal. Every time you run it in a drill, fix the parts that were unclear, wrong, or slow, and update the document the same day.
Failback: The Hard Way Home
Going back is usually harder than going out.
After a failover, the original primary eventually returns, but it is out of date: while it was down, the standby accepted new writes. It may also hold a few messages that never reached the standby before the failure. Simply switching back would throw away everything written on the standby, or create two conflicting histories. So failback is not the reverse of failover; it is a second, careful project.
| Approach | How it works | Pros | Cons |
|---|---|---|---|
| Stay on the new site | Promote the former standby as permanent primary, then build a fresh standby elsewhere, maybe at the old site | Simplest and safest, no second switch | The roles are swapped; the site may be more expensive or farther from users |
| Rebuild the old site from scratch | Wipe the returned cluster, and replicate everything from the current active cluster to it, then plan a switch back | Clean; no risk of diverged data | Copying large data takes time |
| Reverse replication and resume | Start replication from the active side to the old site, keeping what it still holds, and wait to catch up | Faster for short outages | Old data that diverged can cause duplicates or confusion, requiring a review of leftover messages |
After the flooded shop is repaired, you cannot just move back with yesterday’s papers. The backup shop has served customers for weeks. You must first carry its records back, and only then move the staff. Or, you may decide the second shop is fine and make it your main shop.
Inspect the returned cluster
Is the data intact? What was in it that never reached the standby? Decide what to do with those messages, which may need manual handling.
Start replication in the reverse direction
From the active cluster to the returned one, and watch it catch up.
Wait for a steady, low lag
Confirm bookmarks are translated in the new direction as well.
Pick a quiet time
A planned, announced window, not during peak.
Run the failover runbook in the other direction
Fence, stop, verify, switch, start, and verify again.
Restore the original direction of replication
So that protection is back to normal.
Topic names matter here. With prefix naming, the reverse replication creates topics with new prefixes on the returned cluster, such as a prefix on top of a prefix, which are hard to reason about. Design and test the reverse naming in advance.
Testing: Game Days and Drills
A failover that has never been tried is a hope, not a design.
| Kind of test | What it proves | How often |
|---|---|---|
| Automated checks of parity | Topics, partition counts, settings, and permissions match on both sides | Daily |
| Replication health test | Heartbeats arrive, lag is within the RPO, bookmarks are being synchronized | Continuous |
| Tabletop exercise | People know the runbook and their roles, without touching systems | Every few months |
| Partial drill | One non-critical application fails over to the standby and back | Every quarter |
| Full game day | The complete switch, with real applications, timing, and verification | At least once or twice a year |
| Chaos and fault injection | Behavior with slow links, partial loss, and unexpected order of events | Regularly, in a test environment |
Schools run fire drills so that children know where to go without thinking. Nobody enjoys them, but nobody wants to learn the exits during a real fire.
What to measure in every drill
Time from decision to first successful write on the standby
The core of your real RTO.
Time until the slowest application is healthy
Often much longer than the first number, and closer to what the business feels.
Estimated data loss and duplicate volume
Compare the replication state with the primary’s final position, and count repeated messages.
Surprises
Every missing permission, forgotten dependency, and unclear step, written down and fixed.
Drill the failback too. Many teams practice going to the standby, and then discover during a real event that coming back is a mystery.
Byte-for-Byte Linking Versus Message Copying
A second style of replication removes many of the offset headaches.
The tool we described so far reads messages like a consumer and writes them like a producer. That is flexible and works between any two clusters, but it is why offsets differ and transactions are lost. Some platforms offer cluster linking, where the standby cluster’s brokers act like followers of the primary’s partitions and copy the raw log bytes. The copy is an exact replica, including offsets, and the standby topics are read-only “mirror topics” until they are promoted.
| Aspect | Message copying (MirrorMaker 2) | Cluster linking (byte-for-byte) |
|---|---|---|
| Offsets | Different; must be translated | Identical, so bookmarks work as they are |
| Consumer group bookmarks | Translated and synchronized | Can be copied directly |
| Transactions state | Not preserved | Preserved as part of the log |
| Where it runs | Separate workers built on Kafka Connect | Built into the brokers of the platform that offers it |
| Topic names | Prefix or identical, by policy | Usually identical |
| Availability | Open source, ships with Kafka | Offered by some commercial platforms and managed services |
| Flexibility (filter, transform, different partition counts) | Higher | Lower, since it copies the log as it is |
| Promotion during failover | Create topics, switch clients | A promote step turns the mirror topic writable |
Retyping every letter into a second ledger is flexible but produces different page numbers. Photocopying the ledger page for page gives identical pages, so any bookmark works, but you cannot change a single line on the way.
Everything else in this guide still applies with cluster linking: RPO from asynchronous lag, fencing, client routing, testing, and failback. Only the offset and transaction chapters become lighter.
How Ready Should the Standby Be: Hot, Warm, or Cold
Readiness costs money, and each level buys a shorter recovery.
| Level | What is running | Typical RTO | Cost |
|---|---|---|---|
| Hot standby | Full-size cluster receiving all data, applications installed and tested, with routing ready | Minutes | Highest: nearly a second copy of everything |
| Warm standby | Smaller cluster receiving all data, with the ability to grow quickly; applications ready but stopped | Tens of minutes to hours | Medium: less hardware, extra time to scale |
| Cold standby | Only backups or a few replicated topics; cluster created from templates on demand | Hours to days | Lowest, but longer downtime and larger data loss |
A hot standby is a fully furnished second shop with the lights on. A warm standby is a smaller shop with the furniture still in boxes. A cold standby is a signed lease and a moving plan.
Sizing the standby
Size for the replication load first
The standby must sustain the full write rate of every copied topic, with the same replication factor, or it falls behind in normal times.
Then size for serving all clients
After the switch, it must also serve producers and every consumer group. Read traffic can be several times the write traffic.
Keep storage equal
It must hold the same retention as the primary for the copied topics.
Decide which topics to protect
Leaving out unimportant topics reduces the size and the cost.
A standby that is too small keeps up in normal life and collapses when full traffic arrives after failover. Test it with a realistic load, not only with the trickle of replication.
The Network Between Sites
The link between clusters is the pipe that sets your RPO.
All copied data travels across a wide-area network. Its capacity, delay, and reliability set the lag. As a rule of thumb, the link must carry the peak write rate of all protected topics, plus a margin for catching up after a break. If the link can carry only the average, every small outage builds a backlog that never fully clears.
| Network factor | Effect | What to do |
|---|---|---|
| Bandwidth below peak write rate | Lag grows without limit during peaks | Increase capacity, or reduce protected data and use compression |
| High round-trip time | Each request takes longer, and a single connection carries less | Larger batches, tuned buffers, and several parallel tasks |
| Packet loss and jitter | Retries and stalls, uneven lag | Redundant paths and monitoring |
| Shared link with other traffic | Unpredictable lag | Reserve bandwidth, or use a dedicated link |
| Cost of data transfer | Recurring bills in public clouds | Compress, and replicate only what is needed |
| Catch-up after a break | The backlog competes with new data | Have headroom of, say, twice the average rate |
A pipe between two water tanks must carry the busiest hour’s flow. If it carries only the daily average, the second tank is always a little behind, and after a blockage it never catches up.
Compress data before crossing the wide-area link, and keep the producer’s compression choice in the copied batches. A well-compressed stream needs much less bandwidth and lowers cost.
Controllers and Cluster Identity in a Two-Cluster World
Each cluster is a complete, independent Kafka with its own brain.
In modern Kafka, each cluster has its own controller quorum, the small group of nodes that stores cluster decisions such as which broker leads which partition. In an active-passive design, the two clusters do not share a controller quorum. Each is separate, with its own unique cluster identity. This independence is exactly what allows the standby to keep running when the primary’s whole site is gone.
| Question | Answer in active-passive |
|---|---|
| Do the clusters share a controller quorum? | No. Each has its own, entirely inside its own site |
| Do they share a cluster identity? | No. Each has its own unique identity |
| What does the standby know about the primary’s brokers? | Nothing, apart from what the replication tool reads as a client |
| Can one cluster’s failure block the other’s leader elections? | No. This is a key benefit compared with stretch clusters, where a shared quorum can be lost with a site |
| What must be kept the same on purpose? | Topic settings, partition counts, security rules, and version compatibility |
Two independent shops, each with its own manager and its own keys. If the manager of one shop is unreachable, the other shop keeps its own management untouched.
Version and upgrade care
Keep the two clusters on compatible versions, and upgrade them in a planned order. A common practice is to upgrade the standby first, watch it for a while, and then upgrade the primary. This also acts as a test of the new version with real replicated data, while the risk is limited to the passive side.
Independence between the two clusters is a feature. Avoid clever shortcuts that make one cluster depend on the other, such as sharing coordination services or credentials stores that live in only one site.
A Worked Example: Calculating Your Real RPO and RTO
Turning vague promises into numbers you can defend.
A team says, “We can recover in five minutes with almost no loss.” Let us test the claim with the pieces we have learned. Suppose the write rate at peak is 50 megabytes per second, the wide-area link can carry 80 megabytes per second, and the pipeline normally lags by two seconds.
| Step | Assumption | Time or amount |
|---|---|---|
| Normal lag at peak | Measured during the busiest hour | 2 seconds |
| Backlog after a 5-minute link problem | The link is down while 50 megabytes per second keep arriving | 15,000 megabytes waiting |
| Catch-up speed after the link returns | 80 minus 50 leaves 30 megabytes per second of spare capacity | 15,000 divided by 30 = 500 seconds, about 8 minutes |
| Worst-case RPO if the primary is lost just before catch-up ends | The backlog is still partly uncopied | Up to about 5 minutes of data |
| Detection | Several probes must agree over a set time | 3 minutes |
| Decision and fencing | Approval, then blocking the old side | 5 minutes |
| Standby checks and routing change | Automated checker plus a configuration change | 4 minutes |
| Applications reconnect and warm up | The slowest service decides | 8 minutes |
| Total RTO | Sum of the four steps above | About 20 minutes |
So the honest statement is: “Normally we lose only a few seconds, but in the worst case we may lose about five minutes, and we will be running again in about twenty minutes.” That is very different from “five minutes with no loss,” and it is much more useful to the business, who can now decide whether it is acceptable or whether to spend money to improve it.
A delivery company that promises “one hour” should add up the time to pack, drive, park, and walk upstairs. The honest total, given to customers in advance, builds more trust than an optimistic promise that fails.
Where to spend money to improve the numbers
Add link capacity
More spare capacity means faster catch-up and a smaller worst-case backlog.
Automate the checks
A one-command checker turns a half-hour of manual checks into a minute.
Speed up client switching
A configuration service that applications watch is much faster than redeploying everything.
Fix the slowest application
One old service that needs a manual restart can define the whole recovery time.
Always calculate the worst case, not the average. A plan built on the average fails on the very day the worst case occurs.
When Active-Passive Is Not Enough
Knowing the limits of a pattern is part of using it well.
Active-passive is a fine fit for many companies, but there are situations where it starts to strain. Recognizing them early helps you plan a move to another pattern, rather than being pushed into it during an incident.
| Signal | What it means | Possible next step |
|---|---|---|
| Business demands an RPO of zero | Asynchronous copying cannot deliver it | A stretch cluster over three nearby sites, or application-level confirmation from both sides |
| Users in several continents need fast local writes | One active region makes far-away users wait | Active-active for regional data, with clear rules for conflicts |
| The standby is always idle and costly | Money is spent on capacity that never serves users | Use the standby for read-only work, or move to a smaller warm design |
| Failovers are frequent because of small outages | The primary site is unreliable, or the rules are too sensitive | Fix the site, or improve detection rules |
| Teams cannot finish drills on time | The process is too complex | Simplify: fewer topics, fewer applications, more automation |
| Data must never leave a country | A standby in another country breaks the rules | Standby within the same country, in a separate region |
A small shop does well with one spare key kept at a neighbor’s house. A growing chain needs a proper security system with several managers. The spare key was not wrong; the business simply outgrew it.
Many teams move in stages: from a single cluster to active-passive, and only later, if needed, to more advanced patterns. Each stage teaches lessons about duplicates, offsets, and operations that make the next stage safer.
Advantages, Disadvantages, and Trade-offs
Active-passive buys simplicity and safety, and pays in idle capacity and manual work.
What you gain
- Survival of a whole-site loss with a small, known data-loss window
- One writer at a time, so there are no conflicting histories
- Independent clusters, so no shared point of failure
- A safe place to try new versions first
- Works over any distance, because copying is asynchronous
What it costs
- A second cluster that spends most of its time waiting
- Some recent data can be lost, and duplicates are expected
- Offsets, transactions, and consumer bookmarks need special handling
- A switch and a return procedure to maintain and rehearse
- Wide-area bandwidth and transfer costs
| Design choice | Benefit | Cost |
|---|---|---|
| Hot standby, full size | Fast recovery | High running cost |
| Warm standby, small | Lower cost | Slower recovery and scaling risk |
| Automatic failover | Shortest RTO | Risk of false alarms and split-brain |
| Manual approval | Safer decisions | Added minutes to the RTO |
| Identical topic names | No application changes | Unsafe for two-way copying |
| Prefixed topic names | Clear origin and loop safety | Applications must handle both names |
| Replicating every topic | Nothing forgotten | More bandwidth and standby capacity |
| Selective replication | Lower cost | Some data is unrecoverable, by design |
Insurance always costs something every month, and always seems wasteful until the day it is needed. The skill is choosing the amount that matches what you can afford to lose.
Security in a Two-Cluster Design
Two clusters mean twice the doors, and a bridge between them.
| Area | Risk | Practice |
|---|---|---|
| Encryption on the wide-area link | Data crossing between sites could be read or altered | Use encrypted connections on both sides, and verify certificates |
| Replication credentials | The pipeline holds power over both clusters | Give it a dedicated identity with only the rights it needs: read on the source, and write and topic management on the target |
| Access rules parity | Applications may have rights on the primary but not on the standby | Manage rules from one source of truth and apply them to both; check parity daily |
| Copied access rules | Copying rules may grant more than intended on the other side | Review and test what is copied |
| Fencing power | The ability to block clients is also the ability to cause an outage | Restrict who can fence, and log every use |
| Secrets and certificates | Expired or missing on the standby at failover time | Rotate and renew for both sides together, and alert before expiry |
| Data at rest on the standby | Same sensitivity as the primary | Same encryption and access controls |
| Data location rules | Laws may restrict where certain data can be stored | Check that the standby’s region is allowed for each protected topic |
| Audit | Who promoted the standby, and when | Keep logs of every failover action |
A backup office needs its own locks, badges, and guards, and the courier between offices must be trusted with exactly the right keys, no more. A second office with weaker security becomes the easiest way in.
Do not relax the standby’s security “because it is only a backup.” It holds a full copy of your data, and it will hold your live traffic during the worst day of the year, when attention is elsewhere.
Monitoring, Logging, and Metrics
You cannot trust a standby you do not measure.
Exact metric names depend on the versions and tools you use, so treat this as a map of ideas. Watch three groups: the pipeline, the standby’s readiness, and the drill results.
| What to watch | Why it matters | Warning sign |
|---|---|---|
| Replication latency and record age | Directly sets the RPO | Above the agreed RPO, or steadily growing |
| Bytes copied per second per topic | Shows whether the pipeline keeps up with the write rate | Lower than the primary’s write rate for a long time |
| Heartbeat arrival at the standby | Proves the whole pipeline is alive | Missing for longer than a few intervals |
| Connector and task status | Shows failed or paused copy jobs | Any task not running |
| Checkpoint age per critical consumer group | Shows how fresh the translated bookmarks are | Older than the group-sync interval by a wide margin |
| Topic parity: names, partition counts, settings | Prevents surprises at failover | Any difference |
| Access-rule parity | Ensures applications can work on the standby | Rules on one side and not the other |
| Standby cluster health: under-replicated partitions, disk space, load | The standby must be a healthy cluster | Any unhealthy signs, even without traffic |
| Wide-area link use and errors | The pipe that carries everything | Near capacity or rising errors |
| Certificate and credential expiry dates | Common cause of failed failover | Fewer than a few weeks remaining |
| Drill results over time | Shows whether your real RTO and RPO are improving | Getting worse, or drills skipped |
A hospital tests its standby generator every week, running it under load, because a generator that has not run for a year probably will not start when the lights go out.
Alert on lag against your RPO
Warn at half the RPO, and page at the RPO.
Run a synthetic test message
Regularly write a marked message on the primary and confirm it arrives on the standby, measuring the time.
Host monitoring in a third place
So that it survives either site.
Keep the logs of the replication service
Errors about permissions, missing topics, or timeouts often appear days before a lag alarm.
Best Practices and Design Patterns
Habits that make a failover uneventful.
Tier your topics
Classify topics by importance and give each tier its own RPO and replication setup.
Idempotent everything
Unique message IDs and idempotent consumers make duplicates harmless.
Run the pipeline near the standby
Read remotely, write locally, so the pipeline survives the primary’s loss.
Infrastructure as configuration
Create topics, rules, and settings from one source in both clusters, so parity is by construction.
Single source of truth for “who is active”
One small, reliable service that both applications and operators consult.
Fence before you switch
Always stop the old side from accepting writes first.
One-command runbook
Automate the steps, keep a human decision, log everything.
Rehearse in production-like conditions
Drills with realistic load reveal capacity and timing problems that quiet tests hide.
| Practice | Reason |
|---|---|
| Disable automatic topic creation on both clusters | A mistyped name should fail loudly rather than create a lonely topic |
| Keep the same retention on both sides | The standby should hold what the primary holds |
| Document the reset policy of each consumer group | The wrong choice silently skips or replays data |
| Publish the measured RPO and RTO to the business | Everyone shares the same expectations |
| Review the map of dependencies every quarter | New services quietly add new needs |
| Keep a written record of every drill | Shows progress and satisfies auditors |
| Upgrade the standby first | A safer place to find problems |
Well-run hotels keep spare keys, spare rooms, and a written evacuation plan, and rehearse the plan with staff. Guests never see it, which is exactly the goal.
Anti-Patterns to Avoid
Four popular mistakes and the better ways to work.
What people do
They set up replication once, see data arriving, and never test a real failover.
Why it hurts
On the day of the disaster, they discover missing permissions, empty bookmarks, forgotten dependencies, and applications that cannot change address. The real RTO turns out to be hours instead of minutes.
Better approach
Schedule regular drills, measure them, fix what they reveal, and repeat.
What people do
They copy the old bookmark numbers unchanged when moving consumers to the standby.
Why it hurts
Positions differ between clusters, so consumers skip large amounts of data or replay huge histories.
Better approach
Use translated bookmarks, or byte-for-byte linking that keeps offsets identical, and verify before starting consumers.
What people do
They enable fully automatic failover triggered by a single health check, with no fencing.
Why it hurts
A short network glitch triggers a switch while the primary is still accepting writes. Two histories form, and someone must reconcile them by hand.
Better approach
Require agreement from several independent signals, fence the old side, and keep a human approval unless the RTO truly forbids it.
What people do
They treat the standby as a backup and assume it protects against bad data or accidental deletions.
Why it hurts
Wrong data and deletions are copied faithfully, so both clusters are wrong within seconds.
Better approach
Add protections aimed at those risks: strict access rules, longer retention, delayed copies for critical topics, and separate archives.
Troubleshooting Playbook
From symptom to likely cause to first check.
| Symptom | Likely cause | First thing to check |
|---|---|---|
| Replication lag keeps growing | Link too small, too few tasks, or slow standby | Link use, task count, standby broker health |
| A topic is missing on the standby | Not in the allow list, or created after the last refresh | Topic filter settings and the refresh interval |
| Consumers on the standby start from the beginning | No bookmark for the group there | Group in the sync allow list, and sync interval |
| Consumers on the standby skip data | Reset to newest because bookmark was missing | Reset policy and bookmark existence |
| Bookmark sync seems to do nothing | The group is active on the standby | Whether any consumer of that group runs on the standby |
| Access denied after failover | Rules missing on the standby | Access-rule parity report |
| Applications cannot decode data | Schemas not present on the standby | Schema replication status |
| Clients still connect to the old cluster | DNS caching or long-lived connections | Record lifetimes and client restart plan |
| Per-key order broken after failover | Different partition counts | Partition parity report |
| Two clusters both accept writes | Fencing failed | Credentials, firewall rules, routing layer |
| Endless copying loops in two-way setups | Identical names with two-way copying | Naming policy |
Most failover problems are found by comparing: primary versus standby topics, rules, settings, and bookmarks. A daily automatic comparison report prevents nearly all of them.
Real-World and Industry Examples
The same ideas appear across many industries.
Banks and payment providers
Regulators often require proof that a system can recover from the loss of a data center. Banks commonly run a primary and a standby region, rehearse the switch on a schedule, and combine Kafka replication with database failover in one coordinated plan.
Large online retailers
Retail platforms protect order and inventory streams across regions. During big sales events, they freeze changes, verify replication health, and keep the standby scaled up in case of a sudden switch.
Telecommunications
Network event streams feed billing and monitoring. Operators keep a warm standby in a different city, and use selective replication to protect billing topics more strongly than diagnostic logs.
Airlines and travel
Booking and flight-status data must survive site loss. These companies emphasize duplicate-safe processing, since a repeated booking event must not double-book a seat.
Software companies with global users
Some start with active-passive for simplicity and move parts of their data to active-active later, once they have learned to deal with duplicates and conflicts.
Managed streaming services
Cloud providers and vendors offer built-in cross-region linking features and guided failover. The design decisions in this guide, RPO, RTO, fencing, client routing, and failback, still belong to the customer.
A Day in the Life: FreshCart Grocery Delivery
Everything you learned, told as one small story.
FreshCart is a grocery delivery company. Its main Kafka cluster runs in region A, and a standby cluster in region B receives copies of the “orders,” “payments,” and “driver-locations” topics. Replication lag is normally two seconds. The team agreed on an RPO of one minute and an RTO of thirty minutes for orders and payments, and looser targets for driver locations.
A quiet Tuesday, 6:40 p.m.
Dinner-time traffic is at its peak. A cloud provider announces network problems in region A. Producer errors begin rising in several services.
Confirming
Probes from three locations cannot reach the region A cluster. Heartbeats at region B stop. The on-call engineer opens an incident and asks the duty director for approval to fail over, following the written rule.
Fencing and freezing
The engineer revokes the applications’ credentials on region A, so nothing can write there even if it comes back. Deployments are frozen. The last heartbeat shows a lag of three seconds when contact was lost.
Readiness checks
The one-command checker confirms the topics, partition counts, access rules, and schemas on region B. It shows bookmarks for all critical groups, the freshest only forty seconds old.
Switching
The configuration service is updated with region B’s addresses. Applications reload the setting within a minute. Payment consumers start first, then order consumers, then the producers.
Living with duplicates
The payment consumer re-reads about forty seconds of data. Because it uses the payment ID to skip repeats, no customer is charged twice. A few order emails are sent again, and support is warned.
Verification
Order volume returns to normal levels. The team compares the last copied positions with the numbers reported by region A after it returns, and estimates that three seconds of driver-location updates were lost. That is acceptable for that topic.
Recovery time
Total time from decision to healthy: 19 minutes, inside the RTO. The team notes that one old service needed a manual restart, and writes it into the runbook.
The next week
Region A returns. The team rebuilds it as the new standby by replicating from region B, and chooses to stay on region B as primary for now. A planned failback is scheduled for a quiet Sunday, after a full rehearsal.
Nothing in this story was heroic. Fencing, translated bookmarks, unique payment IDs, a checklist, and earlier drills did the work. That is what a good failover design looks like: calm, boring, and repeatable.
Glossary: Every Term in One Place
A quick reference for the words used in this guide.
| Term | Simple meaning |
|---|---|
| Cluster | A group of Kafka brokers working together as one system |
| Active or primary cluster | The cluster that currently accepts writes and serves applications |
| Passive or standby cluster | The cluster that receives copies and waits to take over |
| Failover | Switching from the failed primary to the standby |
| Failback | Returning to the original primary afterwards |
| RPO | Recovery point objective: how much recent data you accept losing |
| RTO | Recovery time objective: how long you accept being down |
| Replication lag | The delay between a write on the primary and its copy on the standby |
| MirrorMaker 2 | The Kafka tool that copies data between clusters, built on Kafka Connect |
| Kafka Connect | A framework for running reliable data-moving jobs |
| Offset | The position number of a message in a partition |
| Offset translation | Converting a position on one cluster into the matching position on another |
| Checkpoint | A saved translation of a consumer group’s bookmark |
| Heartbeat | A small regular message proving the pipeline is alive |
| Consumer group | Readers that share the work of a topic and one bookmark |
| Idempotent | Safe to repeat, since a second run has no extra effect |
| Fencing | Blocking the old side from accepting writes |
| Split-brain | Two clusters both accepting writes and holding different data |
| Stretch cluster | One cluster spread over several nearby sites |
| Active-active | Two clusters both accepting writes |
| Cluster linking | Byte-for-byte copying of partitions that keeps offsets identical |
| Hot, warm, and cold standby | How ready the standby is, from fully running to only planned |
| Runbook | A step-by-step written procedure for an operation |
| Game day | A planned rehearsal of a failure and its recovery |
Frequently Asked Questions
Quick answers to the questions beginners ask most often.
Summary and Key Takeaways
The whole story in a few short lines.
Key Takeaways
- Two independent clusters, one writer: active-passive keeps a standby cluster fed by asynchronous copying, and only one cluster accepts writes at a time, which avoids conflicting histories.
- Choose RPO and RTO first: replication lag sets how much data can be lost, and detection, decision, and switching set the downtime. Zero loss needs a different pattern.
- Copy more than messages: topics, settings, permissions, schemas, bookmarks, credentials, and certificates must all be ready on the standby.
- Offsets change between clusters: translated and synchronized consumer bookmarks let readers resume slightly before where they stopped, so consumers must tolerate duplicates.
- Guarantees do not travel: transactions and exactly-once processing stop at the cluster boundary, so design for at-least-once with unique message IDs.
- Fence, then switch: block the old side before promoting the standby, use several signals to detect failure, and prefer a human-approved, automated runbook to prevent split-brain.
- Rehearse both directions: failback is a second project. Regular drills, daily parity checks, and monitoring in a third location turn a disaster into a routine procedure.