A single location is a single point of failure, a single point of distance, and a single point of law.

Three kinds of trouble

Keeping everything in one place is simple, and simple is good. But three kinds of trouble appear as a business grows.

• Failure. A datacenter can go dark. Causes include a power cut, a cooling failure, a fire, a fiber cable cut by a digging machine, or a mistake during maintenance. When it goes dark, everything inside goes with it.
• Distance. Signals travel fast, but not instantly. A user in Mumbai talking to a server in Virginia waits much longer than one talking to a server in Mumbai. For some applications, that delay hurts.
• Rules. Some countries and industries require that certain data stays inside a border, or that a second copy exists somewhere else. A single cluster cannot satisfy both at once.

Why not just add more brokers?

A natural question is why a bigger cluster is not enough. Adding brokers inside one datacenter helps with load and with single-machine failures. It does nothing if the whole building is lost. Kafka does replicate each partition to several brokers, but if all of them sit in the same room, they all share the same fate.

A handful of words appear in every conversation about multi-datacenter Kafka. Learn them once.

Kafka already copies data. So why is a second kind of copying needed?

What Kafka does by itself

Every partition in Kafka has one leader and, usually, two followers. The followers constantly pull new messages from the leader. When a producer sends a message and asks for strong safety, Kafka waits until enough followers have the message before saying “done”. If the leader machine dies, a follower takes over. This all happens inside one cluster, over a fast local network, and it is automatic.

Why that is not enough

All those brokers belong to one cluster with one controller and one set of metadata. If the datacenter is lost, the whole cluster is lost. A second, separate cluster in another location has its own brokers, its own metadata, and its own life. Something must carry data from the first to the second. Kafka does not do this on its own between separate clusters, so we add a tool for it.

flowchart LR
    subgraph DC1["Datacenter one"]
        L1["Leader"] --> F1["Follower A"]
        L1 --> F2["Follower B"]
    end
    subgraph DC2["Datacenter two"]
        M1["Mirror tool"] --> L2["Leader"]
        L2 --> F3["Follower A"]
        L2 --> F4["Follower B"]
    end
    L1 -->|"read by"| M1
    

In the picture, each datacenter has its own leader and followers. The mirror tool reads from the first cluster like an ordinary consumer, and writes into the second cluster like an ordinary producer. Because it behaves like a normal client, the two clusters do not need to be the same size, the same version, or even in the same kind of environment.

Almost every real design is one of a few shapes, or a mix of them.

flowchart TB
    subgraph AP["Active-passive"]
        A1["Cluster A: serves users"] -->|"copies"| A2["Cluster B: standby"]
    end
    subgraph AA["Active-active"]
        B1["Cluster A"] |"copies both ways"| B2["Cluster B"]
    end
    subgraph HS["Hub and spoke"]
        S1["Site 1"] --> H["Central hub"]
        S2["Site 2"] --> H
        S3["Site 3"] --> H
    end
    

The first three shapes use separate clusters and a mirroring tool. The fourth is different: it is a single cluster stretched across sites. We will compare them carefully later. For now, notice that every use case in this guide is really a choice of shape, made to solve one specific problem.

Before choosing a design, ask two questions: how much can we lose, and how long can we wait?

RPO: how much data can we lose?

Mirroring between clusters normally happens after a message has already been written in the source cluster. This is called asynchronous copying. There is always a small delay before the copy arrives. If the source datacenter dies at that exact moment, the messages still in transit are lost. The Recovery Point Objective is the amount of such loss you accept. A bank might accept almost none. A page-view counter might accept several minutes.

RTO: how long can we be down?

The Recovery Time Objective is the time between the disaster and the moment users are served again. If switching to the backup needs a human to wake up, read a checklist, and change many settings, the RTO is long. If applications switch automatically, it is short.

The classic reason: if the main site is lost, keep the business alive from another one.

The problem

Your company runs orders, payments, or shipping events through Kafka. If the cluster disappears, those flows stop, and worse, the history stored in the topics may vanish too. You need a second location holding a recent copy of the data, ready to take over.

The solution

Build an active-passive setup. The primary cluster serves everyone. A mirroring tool continuously copies chosen topics to a standby cluster in a distant location. Nobody reads from the standby during normal times. When disaster comes, applications are pointed at the standby, which already holds nearly all the data.

flowchart LR
    P["Producers"] --> C1["Primary cluster"]
    C1 --> R1["Consumers"]
    C1 -->|"mirrored copy"| C2["Standby cluster"]
    C2 -.->|"only after failover"| R2["Consumers"]
    

What must be copied

• Topic data. The messages themselves.
• Topic settings. Number of partitions, retention rules, and so on, so the standby behaves like the primary.
• Consumer positions. Where each consumer group stopped reading. Without this, applications restart at the wrong place, either re-reading everything or skipping data.
• Access rules and schemas. Permissions and message format definitions, if you use them.

Sometimes the goal is not survival but speed.

The problem

A company has one central cluster in one region, but readers of the data are spread across the world. A dashboard in Singapore reading from a cluster in Frankfurt suffers from the long round trip. Every read of a batch waits for the distance, and network costs for crossing oceans add up.

The solution

Mirror the needed topics into a cluster close to the readers. Local consumers then read from the local copy, at local speed. Writes may still happen in the central location. Only the read traffic moves close to the audience.

flowchart LR
    W["Writers"] --> H["Central cluster"]
    H -->|"mirror"| E["Europe copy"]
    H -->|"mirror"| A["Asia copy"]
    E --> RE["European readers"]
    A --> RA["Asian readers"]
    

A hidden money saver

Cloud providers usually charge for data leaving a region. If ten applications in Asia each read the same topic from a cluster in Europe, the data crosses the ocean ten times. With a local copy, it crosses once, for the mirror, and the other nine reads stay local. For popular topics, this can cut the bill noticeably.

Many small sources, one big picture: the aggregation, or hub-and-spoke, pattern.

The problem

A retail chain has hundreds of shops. Each shop runs a small Kafka cluster near its cash registers, so it can keep selling even when the internet is down. But headquarters wants one combined view: total sales, stock levels, and fraud patterns across every shop.

The solution

Each shop cluster mirrors its topics upward into a central aggregate cluster. Analytics jobs at headquarters read from the aggregate cluster only, and never need to talk to hundreds of separate sites.

flowchart TB
    S1["Shop cluster 1"] --> AG["Aggregate cluster at headquarters"]
    S2["Shop cluster 2"] --> AG
    S3["Shop cluster 3"] --> AG
    S4["Shop cluster N"] --> AG
    AG --> AN["Analytics and reports"]
    

Why topic names matter here

If every shop calls its topic “sales”, the aggregate cluster would receive hundreds of streams with the same name. Mirroring tools often solve this by adding the source cluster’s name as a prefix, so “shop1.sales” and “shop2.sales” stay distinct. Analytics jobs can then read all of them together using a name pattern.

Both sites work, both sites serve users, and each site has the other’s data.

The problem

Some businesses cannot tolerate a cold standby. Users in two regions expect fast local service all the time, and a failure in one region should barely be noticed. An idle standby also feels wasteful when it could be doing real work.

The solution

Run two clusters, both active. Local applications write to their local cluster. Mirroring runs in both directions, so each cluster ends up with its own data plus a copy of the other’s. Each application can read a combined view.

flowchart LR
    subgraph US["Region US"]
        UP["US producers"] --> UC["US cluster"]
        UC --> UR["US consumers"]
    end
    subgraph EU["Region EU"]
        EP["EU producers"] --> EC["EU cluster"]
        EC --> ER["EU consumers"]
    end
    UC -->|"mirror"| EC
    EC -->|"mirror"| UC
    

The two big risks

• Loops. If cluster A copies a topic to B, and B copies it back to A, messages could bounce forever. Tools prevent this with naming rules, such as prefixing mirrored topics with the source name and never mirroring an already-prefixed topic back.
• Conflicts. If the same record, such as one customer’s address, is changed in both regions at once, the two versions collide. Kafka does not decide who wins. Your application design must handle it, for example by giving each region ownership of different records.

Sometimes the reason for replication is not technology at all. It is a law or a contract.

The problem

Many regions have rules about personal data. Some say that data about their citizens must be stored inside their borders. Others say that a company must keep a second copy of certain records, perhaps for financial audits, in a separate location. A single global cluster cannot follow both kinds of rule easily.

The solution

Run one cluster per legal region. Raw, sensitive data stays in its home cluster. Only what is allowed to leave is mirrored elsewhere. Often that means copying selected topics, or copying topics that have been cleaned first, with names, addresses, and other personal fields removed or scrambled.

flowchart LR
    subgraph R1["Region one, strict rules"]
        RAW["Raw personal data topic"] --> CLEAN["Cleaning step"]
    end
    CLEAN -->|"only safe data leaves"| GLOBAL["Global analytics cluster"]
    RAW -. "never copied" .-> X["Blocked"]
    

Practical rules

• Mirror by allow-list. Name the topics that may be copied, instead of copying everything except a few.
• Clean before copying, using a stream processing step that writes a separate, safe topic.
• Record who approved each mirrored topic. Auditors like to see the reasoning.
• Check the rules with legal experts. This guide explains the technique, not the law.

Move to the cloud without stopping the business, or live in both worlds on purpose.

Migration

A company has Kafka in its own datacenter and wants to move to a cloud provider. Switching in one night is risky, because dozens of applications depend on the cluster. A gentler path is to create a new cluster in the cloud, mirror all data from the old cluster into it, and then move applications over a few at a time. When the last application has moved, the old cluster is switched off.

Hybrid cloud

Some companies never finish moving, on purpose. Old systems that touch factory machines or stores stay on premises, while new analytics and machine learning live in the cloud. Mirroring keeps a steady stream of events flowing between the two worlds, so each side sees what the other produces.

flowchart LR
    subgraph ON["On premises"]
        OC["Local cluster"] --> LEG["Legacy systems"]
    end
    subgraph CL["Public cloud"]
        CC["Cloud cluster"] --> ML["Analytics and machine learning"]
    end
    OC -->|"events up"| CC
    CC -->|"results down"| OC
    

A second cluster is also a safe playground for risky changes.

Upgrading Kafka in place is usually safe, but large changes carry risk: a new major version, a new storage layout, a different number of partitions, or a move to a new hardware type. A trusted approach is called blue-green. The current cluster is “blue”. A new cluster, “green”, is built with the desired changes. Mirroring copies data into green. Applications move across gradually. If something goes wrong, they move back to blue, which is still healthy.

A separate cluster is a clean boundary. Mirroring lets data cross it on purpose.

Between business units

Large companies often let each team run its own cluster, so one team’s mistakes do not disturb another’s. But teams still need some shared events, such as “customer created” or “order paid”. Mirroring selected topics from one cluster to another shares exactly those events, and nothing more. The receiving team reads a local copy and cannot harm the sender’s cluster, even by accident.

Between companies

A retailer may want to share stock levels with its suppliers. A logistics company may want to share delivery updates with its clients. Giving outsiders direct access to your internal cluster is risky. Instead, you publish the shared topics to a dedicated exchange cluster and let partners read only from there.

flowchart LR
    INT["Internal cluster"] -->|"approved topics only"| EXC["Exchange cluster"]
    EXC --> P1["Partner one"]
    EXC --> P2["Partner two"]
    

Some places have poor internet, yet still need Kafka’s reliability.

Factories, ships, mines, wind farms, stores, and vehicles all create data in places where the network can be slow or can disappear. A small Kafka cluster on the spot keeps local systems running regardless of the link. Whenever the connection is working, mirroring pushes the data to a central cluster. When the link breaks, data waits in the local topics. When it returns, mirroring resumes from where it stopped and catches up.

What this needs

• Enough local storage. The edge cluster must keep data long enough to survive the longest expected outage.
• Patient copying. The mirror tool must retry quietly and not give up.
• Ordered catch-up. After a long gap, data arrives in a rush. The central cluster must handle the burst.

The open-source answer that ships with Apache Kafka.

What it is

MirrorMaker 2 is the mirroring tool included with Apache Kafka. It is built on top of Kafka Connect, the framework for moving data in and out of Kafka. Instead of being one program, it is a small set of connectors that run inside Connect workers. Each connector has a different job.

Why it is popular

• Included and free. It comes with Apache Kafka.
• Many-to-many. One setup can mirror between several clusters at once.
• Topic and setting sync. New topics and changed settings can follow automatically.
• Consumer position translation. Explained in the next chapters.
• Loop protection. Mirrored topics get a prefix, so active-active does not bounce messages forever.

Points to keep in mind

• Mirrored topics have different offsets from the originals, so positions must be translated.
• Default naming adds the source cluster’s name in front of the topic name, which applications must understand.
• It is asynchronous, so the target is always slightly behind.
• You run and monitor the Connect workers yourself.

MirrorMaker 2 is not the only path. Know the other families so you can compare.

Byte-for-byte linking

Some platforms offer a built-in feature that links two clusters directly. The Confluent platform calls it Cluster Linking. Instead of a separate consumer-and-producer program, the target cluster pulls data from the source at a low level and stores an identical copy, with the same offsets. Because positions match exactly, consumer groups can move over without complicated translation. The catch is that such features depend on the vendor’s platform, and they may have their own version and licensing rules.

Connector-based commercial replicators

Vendors also sell replication connectors with extra features, such as easier setup, richer monitoring, and support contracts. They work much like MirrorMaker 2, using the Connect framework, but add polish and help.

Cloud provider features

Managed Kafka services from cloud providers often include their own replication tools, which run as a service so you do not operate workers yourself. Their capabilities differ between providers and change over time, so check current documentation before choosing.

One cluster, several sites, and no copier at all. Sounds perfect, but it comes with strict conditions.

How it works

In a stretched cluster, the brokers of one Kafka cluster are placed in several datacenters, usually three. Kafka’s normal internal replication then keeps copies of each partition in different sites. The brokers know which site they belong to through a setting called rack awareness, and Kafka spreads replicas across sites.

Why people like it

• No data loss on site failure, because a message is confirmed only after replicas in several sites have it.
• Automatic failover. Leaders move on their own, and offsets are identical since it is one cluster.
• Nothing to translate. Consumers see one system.

Why it is hard

• The sites must be close and the network fast. Every safe write waits for replicas in other sites, so a slow link makes every write slow.
• Three sites are needed to keep a majority when one fails, because the cluster’s coordination system works on voting.
• A broken link between sites can freeze writes rather than lose data.
• It suits nearby sites, such as availability zones in one region, and not far-apart continents.

flowchart LR
    subgraph ONE["One stretched cluster"]
        S1["Site A: brokers"]
        S2["Site B: brokers"]
        S3["Site C: brokers"]
    end
    S1 |"fast link"| S2
    S2 |"fast link"| S3
    S1 |"fast link"| S3
    

A look under the bonnet, using plain words.

Step by step

sequenceDiagram
    participant S as Source cluster
    participant M as Mirror workers
    participant T as Target cluster
    M->>S: Read new messages
    S-->>M: Messages with source offsets
    M->>T: Write to mirrored topic
    T-->>M: Confirm with target offsets
    M->>T: Store offset mapping
    M->>S: Read consumer group positions
    M->>T: Store translated positions
    

Why the same partition number matters

Kafka guarantees order only within one partition. By writing source partition three into target partition three, the mirror keeps the order of messages that belong together, such as all events for one customer. Changing the partition count would break that promise, which is why tools keep it identical.

Copying messages is the easy half. Knowing where to resume reading is the tricky half.

The problem

A consumer group remembers how far it has read by storing an offset for every partition. After a failover, the group must continue on the standby cluster. But the standby’s offsets are usually different from the primary’s, because the mirror wrote its own copy. If the group starts with its old offset numbers, it could jump ahead and skip messages, or jump back and process old ones again.

Three ways teams handle it

flowchart TB
    F["Primary cluster fails"] --> Q{"How do consumers resume?"}
    Q -->|"Translated offsets"| T["Resume near the exact spot, few repeats"]
    Q -->|"Identical offsets"| I["Resume at the exact spot"]
    Q -->|"Reset by timestamp"| R["Resume a bit earlier, expect repeats"]
    

Why translated positions are slightly behind

Position mapping is recorded periodically, not for every message. So the translated position is usually a little earlier than the true one. The result is that after failover, a consumer may re-read a small number of messages. That is safe, as long as your processing can handle seeing a message twice.

Names are how the tools stay sane when many clusters copy each other.

The prefix habit

By default, MirrorMaker 2 renames a mirrored topic by adding the source cluster’s alias in front. A topic called “orders” in the cluster nicknamed “east” becomes “east.orders” in the target. This has two big benefits. First, the target can hold its own “orders” topic and the mirrored one side by side without mixing them. Second, the tool can recognize a prefixed name and refuse to mirror it back, which stops loops.

flowchart LR
    A1["east: orders"] -->|"mirrored"| B2["west: east.orders"]
    B1["west: orders"] -->|"mirrored"| A2["east: west.orders"]
    B2 -. "not sent back" .-> X["Loop stopped"]
    

What this means for applications

• A consumer that wants “all orders from everywhere” reads both “orders” and “east.orders”, or uses a name pattern that matches both.
• A consumer that wants only local orders reads the unprefixed topic.
• After a failover, applications must be able to read the prefixed topic name if the data they want now lives under a different name. Plan this in advance, or your failover will surprise you.

Keeping names identical

Some teams dislike prefixes and want the same names on both sides. Tools allow a custom naming rule that keeps names unchanged. This is convenient for active-passive, but dangerous for active-active, because without prefixes the loop protection disappears. Use identical names only when data flows one way.

What promises can a mirrored copy really keep?

Order

Order inside a partition is preserved, because the mirror copies partition to partition. But order between partitions was never promised by Kafka, and order between two clusters certainly is not. In active-active, events from two regions arrive interleaved in unpredictable ways.

Duplicates

If the mirror tool crashes just after writing a batch but before recording its progress, it will write the batch again after restart. Most mirroring setups therefore give at-least-once delivery: nothing is lost, but a few messages may appear twice. Newer versions of the tooling can offer stronger guarantees in some configurations, but verify carefully before relying on them.

Handling it in applications

• Idempotent consumers. Design processing so repeats are harmless.
• Unique message keys or ids. Let consumers recognize and skip messages they have already handled.
• Timestamps for tie-breaking. In active-active, timestamps or version numbers can decide which update wins, if clocks are reasonably in sync.

A guide from problem to solution.

flowchart TB
    START["What is your main need?"] --> Q1{"Survive site loss with zero data loss?"}
    Q1 -->|"Yes, sites close together"| SC["Stretched cluster"]
    Q1 -->|"Small loss is acceptable"| Q2{"Do both sites need to write?"}
    Q2 -->|"No"| AP["Active-passive mirroring"]
    Q2 -->|"Yes"| AA["Active-active mirroring"]
    START --> Q3{"Many sources, one view?"}
    Q3 -->|"Yes"| HS["Hub and spoke"]
    

Everything useful here has a price.

The main trade-offs

• Safety versus speed. Waiting for another site before confirming a write is safest but slowest. Copying afterwards is fast but may lose the last moments.
• Simplicity versus flexibility. Active-passive is easy to reason about. Active-active gives more but demands careful design.
• Cost versus readiness. A warm, full-size standby costs more but recovers faster than a small, cold one.
• Freshness versus bandwidth. Copying everything continuously keeps the target fresh, and uses the most network.

When data crosses a distance, protect it on the road and at both ends.

Protect the road

Traffic between datacenters travels over networks you may not control. Always use encryption in transit, so that anyone tapping the line sees only scrambled bytes. Where possible, use private links or dedicated circuits rather than the open internet.

Prove who is calling

The mirror tool logs in to both clusters. Give it its own identity, using certificates or strong credentials, and never share a person’s account. That identity should be allowed to read only the topics to be mirrored on the source, and to write only the mirrored topics on the target.

Match permissions on both sides

Access rules on the source do not automatically exist on the target. Someone who could not read a topic in the primary might be able to read its copy in the standby if you forget to copy permissions. Some tools can synchronize access rules, and some cannot, so plan this step explicitly.

Protect data at rest

The copy stores the same sensitive information. Encrypt disks, limit who can log in to the machines, and apply the same retention and deletion policies. When a user asks for their data to be deleted, remember that copies exist in more than one place.

A backup you do not watch is a backup you cannot trust.

The key numbers

What to log

• Every task restart, with the reason.
• Topics newly discovered or removed from the allow-list.
• Failovers and failbacks, with who did them and when.
• Authentication failures against either cluster.

What to alert on

• Replication lag above your RPO for longer than a set time.
• A stale heartbeat.
• Any failed or missing mirror task.
• A sudden drop of throughput to zero, which often means silence rather than health.

Copying is not free, so size it like any other important system.

Bandwidth

The link between sites must carry the average flow of mirrored data, plus room for bursts and for catching up after an outage. If the link can carry exactly the average, a one-hour outage will never be recovered, because there is no spare room to run faster than new data arrives. Aim for a healthy margin.

Catch-up time

Catch-up time depends on the size of the backlog and the spare capacity. A simple way to think: if the backlog is ten hours of data and your spare capacity equals the normal flow, catching up takes about ten hours. If spare capacity is triple the normal flow, roughly five hours. Estimate this in advance so that your RPO promises are honest.

Storage

The target stores a full copy of every mirrored topic for as long as its retention rules say. Two clusters roughly double storage. Reduce cost by mirroring only needed topics and by giving the standby suitable retention.

Network charges

Cloud providers usually charge for data moving between regions, and sometimes between zones. Compress mirrored data, mirror only what is needed, and let consumers read the local copy rather than the far one.

A message is useless if the other side cannot understand it.

Schemas

Many teams describe the shape of their messages in a schema registry. If the standby has no copy of those schemas, its consumers cannot decode the messages after failover. Registries can be replicated, or synchronized by tools, so that the same schema identifiers mean the same thing in both places. Test this specifically, because it is a popular cause of failed failovers.

Topic settings

Retention, compaction, and message size limits should match on both sides. If the standby keeps data for a shorter time, some history will vanish sooner than expected. If it accepts smaller messages, large messages will fail to copy.

Access rules and quotas

Permissions and rate limits should be recreated on the target. Otherwise, after failover, either nobody can read, or everybody can.

Connectors and stream jobs

Applications that run inside the cluster world, such as connectors to databases and stream processing jobs, also need a plan. Will they run in both sites, or start in the standby only on failover? Their own internal state topics must be available too.

The best plan on paper fails if nobody has tried it.

Why drills matter

Real disasters happen at bad times, with tired people and missing information. A plan that has been rehearsed becomes a habit. A plan that has never been tried is a guess. Practice reveals the small surprises: a forgotten permission, a hard-coded server name, an application that caches addresses forever.

Where the copier lives, and how to keep it healthy.

Where should it run?

A common recommendation is to run the mirroring workers close to the target cluster. The idea is that the long-distance leg is the read from the source, and the write into the target stays local and reliable. If the long link breaks, a reader can simply retry, while a producer trying to write across a bad link risks awkward states. Different teams weigh this differently, so test in your own network.

Keeping it running

• Run several workers so one machine failure does not stop copying.
• Automate restarts with your platform’s health checks.
• Roll out changes gently and watch lag while you do.
• Keep versions aligned between Connect workers and clusters, following the compatibility notes.
• Track configuration in version control so changes are reviewed.

flowchart LR
    SRC["Source cluster, far away"] -->|"long link, read"| W["Mirror workers near the target"]
    W -->|"local write"| TGT["Target cluster"]
    

Habits worth copying, and habits worth avoiding.

Patterns that work

Anti-patterns to avoid

A short checklist from many teams’ experience.

About failback

Many teams plan the jump to the standby and forget the return. After the original site recovers, it is missing everything that happened during the outage. To return safely, you must copy the new data back, check that it is caught up, and only then move users back. Often it is wiser to make the former standby the new primary and rebuild the old site as the new standby, and avoid a second disruption.

A made-up but realistic story, told slowly.

sequenceDiagram
    participant U as Users
    participant P as Primary site
    participant M as Mirror
    participant S as Standby site
    U->>P: Orders as usual
    P->>M: Events flow
    M->>S: Mirrored copy
    Note over P: Power failure
    U-xP: Orders fail
    Note over S: Failover declared
    U->>S: Orders continue
    

The story shows all the ideas together: an agreed RPO and RTO, a mirrored copy with monitored lag, translated consumer positions, idempotent processing, and a monitor outside the failing site. Each was a decision made long before the emergency.

Common beliefs that lead teams astray.

How different kinds of companies typically use these ideas. The examples are generic pictures, not descriptions of particular firms.

Follow one order event from the moment it is created until it can be read in a faraway cluster.

The journey

Consider a single message: “Order 1001 was paid”. A producer application in the primary site sends it to the primary cluster. The leader of the right partition stores it, and followers in the same site copy it. Only then is the producer told that the write succeeded. At this point the message is safe inside one datacenter, but it exists nowhere else.

Next, the mirror workers, which are always asking the source cluster for anything new, receive the message. They write it into the matching partition of the mirrored topic in the target cluster, where the target’s own leader and followers store it. From that moment the message lives in two sites.

Finally, a consumer in the target site, perhaps a reporting application or a standby order processor, reads it. If the primary site is healthy, this reader is just an ordinary consumer of a local copy. If the primary is gone, the same reader becomes the survivor that continues the business.

flowchart LR
    PR["Producer"] --> L1["Primary leader"]
    L1 --> FL["Primary followers"]
    L1 --> MW["Mirror workers"]
    MW --> L2["Target leader"]
    L2 --> FT["Target followers"]
    L2 --> CO["Target consumers"]
    

Where the delay hides

• Waiting for the mirror to notice. Workers poll in small cycles, so a message may wait briefly.
• Crossing the distance. Physical distance sets a lower limit that no tool can beat.
• Batching. Tools group messages into batches for efficiency, which adds a little wait in return for higher throughput.
• Target write confirmation. The target cluster also replicates internally before confirming.

Add these together, and the total delay is your replication lag. In a healthy setup it is often a matter of seconds. In an unhealthy setup, such as after a link outage, it can grow to minutes or hours.

The lifecycle in a table

Copies only help when applications know how to use them.

Three ways to read

Stream processing and mirrored topics

Applications that combine and count events, such as stream processing jobs, keep their own state. That state usually lives in internal topics of the cluster where the job runs. After a failover, a job started on the standby needs either a mirrored copy of its state topics, or the ability to rebuild state by re-reading its input. Rebuilding may take a long time for big jobs, which affects your recovery time. Decide in advance which approach each job will use.

Connectors

Connectors that move data between Kafka and databases also track their progress in internal topics. If a connector runs in both sites at once, it may load the same rows twice. If it runs only in the primary, it must be started in the standby after a failover, with its progress data available. Write the answer down for each connector, as part of the failover checklist.

flowchart TB
    subgraph PRI["Primary site"]
        J1["Stream job"] --> S1["State topics"]
    end
    subgraph STB["Standby site"]
        J2["Stream job, stopped"] -.-> S2["Mirrored state topics"]
    end
    S1 -->|"mirror"| S2
    

Numbers make the trade-offs easy to see.

The setting

A company writes about twenty megabytes of order data per second into its primary cluster during the busiest hour, and about five megabytes per second at a quiet time. The standby is in a distant region. The company wants to lose no more than one minute of data and to recover within thirty minutes.

Bandwidth question

During the busiest hour the link must carry at least twenty megabytes per second. But the link also needs room to catch up. If the mirror is stuck for ten minutes at the peak, a backlog of about twelve thousand megabytes forms, since twenty megabytes per second for six hundred seconds is twelve thousand. If the link can carry forty megabytes per second, spare capacity is twenty per second, so the backlog clears in six hundred seconds, or another ten minutes. With only twenty-five per second, spare capacity is five, and the backlog would take forty minutes to clear, far longer than the outage. This is why headroom matters.

RPO question

While the mirror is stuck, the exposure grows. After a minute of backlog, the company has already reached its promised maximum loss. So the alert must fire well before one minute, perhaps at fifteen or twenty seconds of sustained lag, to leave time for humans and automation to react.

RTO question

Thirty minutes must cover detection, decision, switching applications, and warming up. If detection takes two minutes, the decision five, switching ten, and warming up ten, the total is twenty-seven minutes, which leaves almost no slack. The team might invest in automating the switch to bring it down to two minutes, and gain a comfortable margin.

flowchart LR
    D["Detect"] --> C["Decide"]
    C --> S["Switch"]
    S --> W["Warm up and verify"]
    W --> OK["Users served again"]
    

Questions to answer, in order, before building anything.

Signs you may be over-building

• Nobody can explain which business promise the second cluster protects.
• Active-active was chosen but no region actually needs local writes.
• Every topic is mirrored, and nobody knows why.
• The team spends more time running the mirror than the applications it protects.

When you see these signs, step back and simplify. A modest active-passive pair that is well monitored and regularly drilled protects a business better than a grand design nobody fully understands.

Questions to ask before choosing a tool or vendor

• Are offsets preserved, or must consumer positions be translated, and how well is that translation tested?
• Who operates the copier, your team or the provider, and what happens when it fails at night?
• How are schemas, permissions, and topic settings synchronized, if at all?
• What monitoring is included, and can it alert you on lag and heartbeat?
• What are the limits, such as maximum throughput, number of topics, and supported versions?
• What does it cost, including the network charges between the sites?
• Can we leave later? Tools that lock you in should earn their place with real benefits.

Write the answers in a small comparison table and share it with the people who will be on call. The person who is paged at three in the morning should have a say in the choice.

Everything in one place.

Cross-datacenter replication answers three basic worries: what if a site is lost, what if readers are far away, and what if rules say where data must live. From these worries come the use cases: disaster recovery, local reads, aggregation, active-active operation, compliance, migration, safe upgrades, partner sharing, and edge sites. Each maps to a shape: active-passive, active-active, hub and spoke, or a stretched cluster. The tools that do the copying, from MirrorMaker 2 to platform linking features, differ mainly in whether they preserve offsets and who runs them. Under all of them lie the same hard problems: copy delay, consumer positions, loops, conflicts, duplicates, schemas, security, and practice.