MirrorMaker 2 in Kafka: Keeping Two Clusters Telling the Same Story
MirrorMaker 2 copies topics, settings, and consumer progress from one Kafka cluster to another, so you can survive disasters, serve users near their homes, and move between platforms. This guide explains how it works, where it shines, and where it surprises people.
Imagine a national newspaper with its main printing house in one city. Every night it prints thousands of copies and ships them out. One day a flood closes the printing house. If nobody had a backup, the newspaper would vanish for weeks. So the company builds a second printing house in a different city, and each evening a courier carries an exact copy of tomorrow’s pages there. If the first house fails, the second one can print immediately. There is a catch, though: the courier takes a little time, so the very last pages written before the flood may not have arrived. And the editors need a way to know exactly which page the readers were on, so nobody rereads yesterday’s news or skips tomorrow’s.
That courier is the idea behind MirrorMaker 2, often shortened to MM2. It is the tool that comes with Apache Kafka for replicating data between separate Kafka clusters. In this tutorial we will start from the very beginning, explain every new word in simple language, and then build up to the design decisions, failure plans, and tuning choices that matter in production.
1Kafka Basics You Need First
A short refresher on the pieces that MirrorMaker 2 copies and connects.
Apache Kafka is a platform for storing and moving streams of events. An event is a small record of something that happened, such as an order placed, a payment made, or a temperature measured. Kafka stores these events on disk in order, so that many applications can read them now or later.
Cluster
A group of Kafka servers working together. A company may run several separate clusters.
Topic
A named category of events, such as “orders”. Think of it as a labelled folder of related messages.
Partition
A topic is divided into partitions. Each partition is an ordered log stored on disk.
Offset
The position number of an event inside a partition. It is how a reader remembers where it stopped.
Consumer Group
A team of consumers that share a topic and remember their progress together using committed offsets.
Replication
Copying a partition’s data to other brokers inside the same cluster for safety. This is different from copying data between clusters.
Two kinds of copying
Kafka already copies data inside a cluster. Every partition can have several replicas on different brokers, so that a machine failure does not lose data. But all of those copies live inside one cluster, usually in one data centre or one cloud region. If the whole data centre goes dark, every copy goes dark with it. Copying data between different clusters, ideally in different locations, is a separate need, and it is exactly what MirrorMaker 2 does.
Keeping several photocopies of a document in different drawers of the same office protects you if one drawer breaks. It does not help if the whole office burns down. For that, you need a copy stored in a different building, which is what cross-cluster replication provides.
Why offsets matter here
Remember that an offset is a position inside one specific partition of one specific cluster. Two clusters that hold copies of the same topic do not necessarily use the same offset numbers for the same event. That fact creates one of the most interesting problems in this whole subject, and we will spend a full chapter on how MirrorMaker 2 solves it.
2Why Copy Data Between Clusters?
There are several very different reasons, and each shapes how you set things up.
Disaster recovery
A second cluster in another location holds a copy of the important topics. If the primary location fails, applications switch to the backup and carry on with little lost.
Geographic distribution
Users in Europe, Asia, and America each write to a nearby cluster for low latency, while data is shared across regions so that global views are possible.
Aggregation
Many small edge clusters, such as those in shops or factories, send their data to one large central cluster for company-wide analytics.
Migration
A company moves from on-premises servers to the cloud, or from one cloud provider to another, and needs to shift data and applications gradually without a big-bang cutover.
Isolation of workloads
Heavy analytical readers can be given their own copy of the data on a separate cluster, so they cannot slow down the production cluster that serves critical applications.
Data sharing and compliance
A selected subset of topics is copied to a cluster used by another department or partner, while sensitive topics stay behind. Some regulations also require data to be kept in particular regions.
flowchart LR
A[Primary cluster] -->|replicate| B[Backup cluster for disaster recovery]
C[Edge cluster 1] -->|aggregate| D[Central analytics cluster]
E[Edge cluster 2] -->|aggregate| D
F[On premises cluster] -->|migrate| G[Cloud cluster]
The reason you replicate decides almost every later choice, from topic naming to how you plan for failure. Write down the goal, such as disaster recovery with a certain recovery time, before you configure anything.
3So, What Is MirrorMaker 2?
The definition in one sentence, followed by what it does in practice.
MirrorMaker 2 is a tool included with Apache Kafka that replicates data between Kafka clusters. It was introduced in Kafka version 2.4 and is built on top of Kafka Connect, the framework for moving data in and out of Kafka. Instead of being a standalone program with its own machinery, it is a set of ready-made connectors that run inside Connect.
Copy Topic Data
Reads events from topics in a source cluster and writes them into matching topics in a target cluster.
Sync Topic Settings
Keeps the target topics’ partitions and many configuration settings aligned with the source topics.
Translate Consumer Progress
Records how consumer group offsets in the source correspond to offsets in the target, so consumers can resume after a failover.
Monitor Connectivity
Sends small heartbeat events through the whole path so that you can measure whether replication is alive and how delayed it is.
The key vocabulary
| Word | Meaning |
|---|---|
| Source cluster | The cluster that data is copied from |
| Target cluster | The cluster that data is copied to |
| Cluster alias | A short label, such as “primary” or “backup”, that names each cluster inside the MirrorMaker 2 configuration |
| Remote topic | The copy of a source topic that lives in the target cluster |
| Replication flow | One direction of copying, such as primary to backup |
| Checkpoint | A record of how a consumer group’s progress in one cluster maps into the other |
Think of a translation and shipping office between two libraries. It photocopies new books from one library into the other, keeps the catalogue descriptions aligned, and writes down which page each reader had reached, translated into the second library’s page numbering.
4From MirrorMaker 1 to MirrorMaker 2
To appreciate the new design, it helps to see what the old one lacked.
The original tool, now called MirrorMaker 1, was essentially a consumer and a producer glued together. It read from one cluster and wrote to another. That worked for basic copying, but real deployments quickly exposed its weaknesses. The older tool is deprecated, and recent Kafka releases have removed it, so new projects should use MirrorMaker 2.
| Question | MirrorMaker 1 | MirrorMaker 2 |
|---|---|---|
| Built on | A simple consumer and producer pair | The Kafka Connect framework |
| New topics in the source | Needed configuration and often a restart | Discovered automatically at regular intervals |
| Topic settings and partition counts | Not synchronised, causing mismatches | Synchronised from source to target |
| Consumer group progress | Not translated, so failover meant guessing where to resume | Translated through checkpoints, and optionally written to the target automatically |
| Avoiding replication loops | Difficult, manual care needed | Built in through topic naming |
| Monitoring | Very limited | Heartbeats and replication metrics |
| Multiple clusters | Awkward, one instance per direction | One deployment can manage many flows |
| Scaling and recovery | Manual | Provided by Connect’s tasks and workers |
Some articles online still describe MirrorMaker 1 or an early version of MirrorMaker 2. Features such as automatic consumer offset syncing and improved deployment modes arrived in later Kafka releases, so always check the documentation for the version you run.
MirrorMaker 1 was like a courier who carries boxes between offices without asking what is inside or writing anything down. MirrorMaker 2 is a logistics company that tracks each parcel, mirrors the shelf labels, and keeps a record of who has read which page.
5Built on Kafka Connect: The Three Connectors
MirrorMaker 2 is really a team of three connectors, each with its own job.
Kafka Connect is a framework that runs small plugins called connectors, splits their work into tasks, and handles scaling, retries, and progress tracking. MirrorMaker 2 provides three connectors that run on this framework, and together they deliver everything described so far.
Mirror Source Connector
Does the main work. It reads topic data from the source cluster and produces it into the target. It also creates the remote topics and keeps their settings and partitions in step.
Mirror Checkpoint Connector
Tracks consumer groups. It translates the offsets committed by groups on the source into equivalent offsets on the target and stores them as checkpoints, and can also apply them to the target’s groups.
Mirror Heartbeat Connector
Emits a small heartbeat event at regular intervals into a dedicated topic. The heartbeats travel through replication, so their arrival proves the path works and reveals delay.
flowchart LR
subgraph SRC[Source cluster]
T1[Application topics]
G1[Consumer group offsets]
H1[Heartbeat topic]
end
subgraph MM[MirrorMaker 2 on Kafka Connect]
C1[Source connector]
C2[Checkpoint connector]
C3[Heartbeat connector]
end
subgraph TGT[Target cluster]
T2[Remote topics with alias prefix]
G2[Checkpoints and synced offsets]
H2[Replicated heartbeats]
end
T1 --> C1 --> T2
G1 --> C2 --> G2
H1 --> C3 --> H2
Internal helper topics
MirrorMaker 2 keeps a few small internal topics to do its bookkeeping. One records offset synchronisation points, which are pairs of matching offsets between source and target that are used to translate consumer progress. Another holds the checkpoints. A third holds the heartbeats. These are ordinary Kafka topics, so they can be inspected, secured, and monitored like any others, and they need proper permissions and replication settings.
Because these are separate connectors, you can enable only some of them. A team that only wants to copy data can skip checkpoints, and a team that mainly wants to prepare consumer failover will enable them. Each part can be tuned separately.
Picture three staff at the shipping office: one loads the parcels, one keeps the ledger of reader progress, and one sends a postcard every minute saying “still here”. Each has a clear role, and if one falls ill the others can carry on.
6How Replication Works, Step by Step
Let us follow one topic on its journey from the source cluster to the target.
The connector finds the topics to copy
It lists the topics in the source cluster and keeps those whose names match your include rules, dropping those that match exclusion rules. It repeats this on a schedule, so new topics are picked up without restarts.
Remote topics are created
For each selected topic, MirrorMaker 2 creates a matching remote topic in the target cluster, using the same number of partitions, and a name that carries the source cluster’s alias.
Tasks are assigned partitions
The work is split into tasks, and each task is given some of the source partitions to read. Kafka Connect spreads tasks across its workers.
Records are read from the source
Each task acts as a consumer of its assigned source partitions and fetches batches of records.
Records are written to the same partition number in the target
The task produces each record to the remote topic, keeping the partition number, the key, the value, the headers, and the timestamp. Copies are made as raw bytes, so no format conversion happens.
Progress is saved
Once the target acknowledges the records, Connect saves the task’s progress, so that after a restart it continues from where it stopped.
Offset mapping points are recorded
Periodically, the connector notes a pair of offsets, one from the source and the matching one from the target, into a helper topic. These pairs become the foundation for translating consumer progress later.
sequenceDiagram
participant S as Source cluster
participant T as MirrorMaker 2 task
participant D as Target cluster
T->>S: Fetch records from a source partition
S-->>T: Batch of records
T->>D: Produce records to the remote topic, same partition number
D-->>T: Acknowledge
T->>T: Save progress and note an offset mapping
T->>S: Fetch the next batch
What is preserved and what is not
| Aspect | Preserved in the Copy? | Comment |
|---|---|---|
| Record key and value | Yes | Copied as raw bytes |
| Headers | Yes | Carried across with the record |
| Timestamp | Yes | The original timestamp is kept |
| Partition number | Yes | A record lands in the same partition number, so per-partition ordering is kept |
| Offset number | No | The target assigns its own offsets, which is why translation is needed |
| Transaction boundaries | No | Records are copied, but atomic groups are not recreated as transactions |
| Retention settings | Partly | Many topic settings are synced, but retention of older data in the target is separate from the source |
Because the data is copied byte for byte, MirrorMaker 2 works with any format, be it text, JSON, Avro, or Protobuf. It does not need to understand the contents. But this also means that anything encoded with references, such as schema identifiers, is copied with those references intact, which matters a great deal when schemas are involved.
7Topic Naming and Preventing Loops
The name of the copy is not a detail. It is the mechanism that keeps two-way replication from going round in circles.
The default: prefix with the source alias
By default, MirrorMaker 2 names each remote topic by putting the source cluster’s alias in front of the original name, separated by a dot. A topic called “orders” in the cluster aliased “primary” becomes “primary.orders” in the target cluster. The original topic name is never reused in the target, so the two are always clearly distinguished.
flowchart LR
A[orders in primary cluster] -->|MirrorMaker 2| B[primary.orders in backup cluster]
C[orders in backup cluster] -->|MirrorMaker 2| D[backup.orders in primary cluster]
Why the prefix prevents loops
Suppose you replicate in both directions, primary to backup and backup to primary. Without prefixes, an event written to “orders” in the primary would be copied to “orders” in the backup, and then copied back to “orders” in the primary, and again, forever. With prefixes, the copy in the backup is called “primary.orders”. MirrorMaker 2 knows that a topic already carrying the alias of the target cluster’s counterpart is itself a copy, and does not send it back. The naming rule quietly breaks the loop.
Two offices photocopy each other’s memos. If every photocopy is stamped with the name of the office that first wrote it, then each office can see the stamp and refuse to copy back a memo that originally came from itself.
The alternative: identity naming
Some situations call for keeping the exact same topic name in the target, especially migrations, where applications should not need to change their topic names. Newer Kafka versions offer an identity replication policy that does this. It is convenient, but it removes the built-in loop protection, so it is safe only for one-directional flows, and you must ensure that nothing replicates the topic back.
| Policy | Remote Topic Name | Loop Protection | Best For |
|---|---|---|---|
| Default, alias prefix | Source alias plus a dot plus the original name | Yes | Disaster recovery, active-active, aggregation |
| Identity, same name | Exactly the original name | No, one direction only | Migrations where names must not change |
| Custom policy | Defined by your own logic | Depends on your design | Special naming conventions |
After a failover with the default policy, consumers must read from the prefixed topic name in the backup cluster, not from the original name. Applications must be prepared for this, for example by subscribing with a pattern that matches both local and prefixed names, or by using the identity policy in one-way designs.
8Consumer Offsets and Checkpoints
The trickiest problem in cross-cluster replication is helping a consumer resume in the right place.
The problem
A consumer group in the primary cluster has read up to offset 5,000 of a partition. That number means “the five-thousandth event in the primary’s log”. In the backup cluster, the same event might sit at offset 4,973, because the backup started copying later, or because retention removed older records, or because of duplicates or gaps. If the group fails over to the backup and simply asks for offset 5,000, it will skip some events or reread others. Offsets are local to each cluster, so they must be translated.
sequenceDiagram
participant SRC as Source cluster
participant MM as MirrorMaker 2
participant TGT as Target cluster
MM->>MM: Note that source offset 5000 matches target offset 4973
MM->>SRC: Read the committed offsets of a consumer group
SRC-->>MM: Group is at source offset 5000
MM->>MM: Translate using the recorded mapping
MM->>TGT: Write a checkpoint saying target offset 4973
Note over TGT: After failover the group resumes near 4973
How translation works
Offset syncs are recorded during copying
As data flows, the source connector records pairs saying that a certain source offset corresponds to a certain target offset.
The checkpoint connector reads group progress
It regularly looks at where each selected consumer group has committed in the source cluster.
It converts the position using the nearest known pair
Using the recorded pairs, it computes the equivalent target offset. The result is safe but approximate, tending to be at or slightly before the exact position, so a few events may be reread rather than skipped.
The translated position is stored
It is written as a checkpoint into a helper topic in the target, and, if enabled, applied directly to the same-named consumer group in the target cluster.
Automatic offset syncing
Earlier versions of MirrorMaker 2 only wrote checkpoints, and left it to you to read them and apply them during a failover, using a client utility. Newer versions can automatically write translated offsets into the target cluster’s consumer groups on a schedule, controlled by a setting for syncing group offsets, which is off by default. This makes failover much simpler, but there is an important condition: a group in the target cluster can only be updated while it has no active members, so it must not already be running there.
Translated offsets deliberately err on the side of rereading rather than skipping. After a failover, expect a small number of events to be processed twice. Consumers should therefore be idempotent, meaning that processing the same event twice must be harmless.
Checkpoints are produced only for the groups that match the group include rules in your configuration. If an application’s group is not included, it will have no translated position and will need a manual decision at failover time.
A reader marks page 250 in one edition of a book. The second edition has different pagination, so page 250 there would land in the wrong spot. A librarian keeps a conversion chart of matching passages and tells the reader, “In the new edition, continue from page 243, just to be safe.”
9Keeping Settings in Sync
Copying the data is only half the story. The topics around the data must match too.
Partition Counts
If new partitions are added to a source topic, MirrorMaker 2 adds partitions to the remote topic, so partition numbers keep lining up.
Topic Configurations
Settings such as retention time and cleanup policy are copied from the source topic to the remote topic, and kept in step as they change.
Access Control Rules
Topic access rules can be copied as well, so the remote topics have similar permissions. Behaviour depends on version and settings, so review it carefully.
Consumer Group Positions
Translated group offsets, as described in the previous chapter, can be kept in step automatically.
Settings that are deliberately not copied
Some topic settings describe the source cluster’s own layout or safety rules, and copying them would be wrong or harmful. By default, MirrorMaker 2 leaves out a list of such properties, including those about replication throttling, timestamp handling, unclean leader election, and the minimum in-sync replicas. The target cluster is a different environment, possibly with fewer or more brokers, and should apply its own values for these.
Because the minimum in-sync replicas setting is not copied by default, remote topics use the target cluster’s defaults, which may be weaker than the source’s. Review these for every replicated topic, especially in disaster recovery clusters that must be trustworthy on the day they are needed.
Replication factor of the remote topics
The number of copies inside the target cluster is chosen by a separate MirrorMaker 2 setting for the remote topics’ replication factor, not copied from the source. Set it to a sensible value for the target cluster, typically three for production, to make sure remote topics are as durable as the originals.
When you open a branch office, you copy the company’s product catalogue to it, but you do not copy the head office’s fire safety rules word for word. The branch follows the rules of its own building.
10Replication Topologies
The same tool supports several different shapes of design. Choosing the right one is a major decision.
Active-passive
One cluster is the active primary, where all applications write. The second cluster is a passive standby that receives copies. Nobody uses the standby until a disaster. This is the simplest and most common disaster recovery design.
flowchart LR
A[Applications] --> P[Primary cluster active]
P -->|MirrorMaker 2| S[Standby cluster passive]
S -.->|used only after failover| A
Active-active
Two or more clusters are live at the same time, each serving local applications, and each replicating to the others. Every cluster holds its own local topics plus prefixed copies of the others’ topics. Applications that need the global picture read both the local topic and the remote copies.
flowchart LR
E[East region applications] --> CE[East cluster]
W[West region applications] --> CW[West cluster]
CE -->|MirrorMaker 2| CW
CW -->|MirrorMaker 2| CE
Hub and spoke, or aggregation
Many smaller clusters, called spokes, replicate into one central hub cluster. The hub holds the combined view of everything for analytics, reporting, and machine learning. Data usually flows one way, from the spokes to the hub.
Fan-out
One cluster replicates selected topics to several destination clusters, for example distributing reference data to every region.
| Topology | Main Benefit | Main Challenge |
|---|---|---|
| Active-passive | Simple, clear ownership of data | The standby is idle capacity, and failover must be practised |
| Active-active | Low latency for users everywhere, no idle cluster | Same data may be written in two places, so conflicts and ordering across regions need design |
| Hub and spoke | Company-wide view in one place | Hub capacity, and topic naming across many sources |
| Fan-out | Consistent shared data everywhere | Many flows to operate and monitor |
Replication takes time, and events written in two regions at once can interleave differently in each cluster. There is no global order across clusters. If the same business entity, such as one customer account, is updated in both regions, you need a clear rule for resolving conflicts, or you should route each entity to a single home region.
Active-passive is a main office with a locked backup office. Active-active is two busy offices that fax each other every update. The second design is more powerful, but two people editing the same contract at the same time need a clear rule about whose version wins.
11Disaster Recovery: Failover and Failback
The whole point of a backup is the day you use it. Let us walk through what happens.
Two numbers that define your promise
Recovery Point Objective
How much recent data you can afford to lose. Because MirrorMaker 2 copies asynchronously, events written just before the disaster may not have arrived, so the loss is the replication lag at that moment.
Recovery Time Objective
How quickly service must resume. It depends on how fast applications can be pointed at the backup and how well the process has been practised.
The failover journey
Declare the disaster
Decide that the primary cluster cannot be relied upon. Agree who has the authority to make this call, since switching too early or too late are both costly.
Check what the backup has
Look at the latest heartbeat and replication metrics to estimate how much data may be missing, and record this for later reconciliation.
Prepare consumer positions
Make sure each consumer group has a translated position in the backup, either already synced automatically or read from checkpoints and applied.
Redirect producers and consumers
Change the applications’ connection addresses to the backup cluster, and, with the default naming policy, make consumers read the prefixed topic names.
Verify and monitor
Confirm that applications are healthy, that lag is falling, and that new data is flowing. Expect a small number of duplicate events from consumers.
flowchart TD
A[Primary cluster fails] --> B[Check replication lag and heartbeats on the backup]
B --> C[Ensure consumer group positions exist on the backup]
C --> D[Point applications at the backup cluster]
D --> E[Consumers resume from translated offsets]
E --> F[Service continues with a small window of possible loss]
Failback: returning home is harder than leaving
Once the primary recovers, you cannot simply switch back. During the outage, the backup received new events that the primary does not have, and the primary may still hold events that never reached the backup. Failing back safely usually means replicating in the reverse direction to bring the primary up to date, reconciling any events that were lost or duplicated, and then choosing a calm moment to switch again. Some teams decide to treat the former backup as the new primary permanently, and rebuild the old primary as the new standby, which avoids a second risky switch.
A failover plan that has never been rehearsed is a hope, not a plan. Run regular drills in a test environment, and measure the real time it takes, the amount of data lost, and how applications behave. Discover the surprises when the stakes are low.
Moving the newspaper to the backup printing house is a fire drill you can practise. Moving back is like relocating a whole newsroom after the flood: pages produced in the meantime must be merged with the originals, and it is often wiser to stay in the new building.
12Deployment Options
MirrorMaker 2 can run in several ways, depending on how much control and infrastructure you already have.
Dedicated MirrorMaker 2 Cluster
Run the provided driver program on its own machines. It starts the connectors for you from one configuration file. This is the simplest way and is a popular default.
On an Existing Kafka Connect Cluster
Register the three MirrorMaker 2 connectors on a Connect cluster you already operate, and manage them like any other connector through its interface.
Standalone Connect Process
Run the connectors in a single Connect process. Fine for tests, but it lacks fault tolerance and is not advised for production.
Managed or Operator-Based Services
Some Kubernetes operators and cloud services provide MirrorMaker 2 as a managed component that you configure declaratively.
| Option | Strengths | Weaknesses |
|---|---|---|
| Dedicated cluster | Isolated from other workloads, simple to start, one configuration file | Another system to run and monitor, and management options depend on your version |
| Existing Connect cluster | Reuses tooling, monitoring, and the familiar management interface | Shares resources with other connectors, so heavy replication can disturb them |
| Standalone process | Trivial to try out | No failover, single point of failure |
| Managed or operator | Less hands-on work, integrates with the platform | Less control, and features follow the provider’s pace |
Where should MirrorMaker 2 run?
The workers must connect to both clusters, and the long-distance network link is the most fragile part. A widely used guideline is to run MirrorMaker 2 close to the target cluster, so that it reads across the long link and writes locally. The reasoning is that a consumer interrupted by a network glitch simply retries its fetch, whereas a producer interrupted mid-write faces harder recovery questions. Test in your own environment, because network characteristics vary.
Whichever mode you choose, use dedicated credentials for MirrorMaker 2 on each cluster, size its machines for the throughput you expect plus headroom, and keep its configuration in version control like any other production system.
13Delivery Guarantees, Duplicates, and Ordering
What can you truly promise about the copy? It is important to be honest here.
At-Least-Once by Default
Every record is delivered one or more times. After a crash between writing and saving progress, some records may be written twice.
Per-Partition Order
Because a record keeps its partition number, order within each partition is preserved in the copy, apart from any duplicates.
Asynchronous Copying
The target is always slightly behind. This delay is the replication lag, and it defines how much can be lost in a sudden disaster.
Exactly-once options
Recent versions of Kafka Connect introduced framework support for exactly-once delivery from source connectors, and recent releases extended this to the MirrorMaker 2 source connector. It has to be enabled deliberately on the workers and on the connector, and it needs matching versions and settings, so check the documentation for the version you run before relying on it. Even then, exactly-once applies to the copying step, not to what your own consumers do afterwards.
Transactions do not travel
If producers use Kafka transactions in the source cluster, the atomic grouping of records is not recreated in the target. The records are copied, but consumers of the copy do not see the same transactional boundaries. Design consumers of remote topics accordingly, and do not assume that a group of records that arrived together will stay together.
Ordering across clusters
Within a partition, order is kept. Across partitions, there was never a guarantee. And across clusters in active-active designs, events written in different places have no global order at all. If your application depends on a strict global sequence, replication between live clusters will not provide it.
Between at-least-once copying and approximate offset translation after failover, duplicates are a normal part of life. Make downstream processing idempotent, for example by using unique event identifiers or upserts keyed by business identifiers, so that a repeated event changes nothing.
A courier who occasionally delivers the same parcel twice after a mix-up is better than one who loses parcels. As long as the recipient can recognise a duplicate parcel by its label and ignore it, nothing goes wrong.
14Performance and Tuning
Replication across long distances has its own physics. A few knobs make a large difference.
What limits throughput
The network link between clusters
Bandwidth and round-trip delay set a hard ceiling. A high-delay link needs larger batches and larger buffers to stay busy.
Number of tasks and partitions
Parallelism comes from tasks, and each source partition is read by only one task at a time. A topic with few partitions cannot be copied faster by adding tasks.
Producer batching and compression
Larger batches, a small waiting time before sending, and compression reduce the cost of moving data across the long link.
Consumer fetch sizes
Bigger fetches per request keep the pipe full on long-distance links, at the cost of memory.
Capacity of both clusters
The source must serve the extra reads, and the target must absorb the extra writes and their replication. Replication traffic is additional load on both.
| Tuning Lever | Effect | Trade-off |
|---|---|---|
| More tasks | More parallel copying, up to the number of partitions | More connections and memory, more rebalancing impact |
| Compression on the producer | Less network traffic across the link | Some extra CPU on the workers |
| Larger batches and short linger time | Higher throughput | Slightly higher delay per record |
| Larger consumer fetch sizes | Better use of a high-delay link | More memory used |
| Fewer replicated topics | Less load and lag | Only what you choose is protected |
| Filtering by topic patterns | Avoids copying unneeded or temporary topics | Requires care so that important topics are not excluded by accident |
Replicating everything is tempting but wasteful. Test topics, scratch topics, and rebuildable derived topics may not deserve disaster recovery treatment. A clear include list keeps cost down and lag low for the topics that truly matter.
Rebalances and replication pauses
Because MirrorMaker 2 runs on Connect, adding or removing workers, or changes to the list of topics and partitions, can trigger a rebalance during which some tasks briefly pause. Frequent rebalances, for example from unstable workers or from topics constantly appearing and disappearing, add lag and should be investigated rather than tolerated.
15Monitoring Lag and Health
A backup you do not monitor is a backup you cannot trust.
| Signal | What It Tells You | Why It Matters |
|---|---|---|
| Replication latency | How long records take to travel from the source to the target | Your real, current recovery point exposure |
| Record age at replication | How old records are when they are copied | Rising values indicate a growing backlog |
| Consumer lag of the replication tasks | How far behind the source’s newest data the copying is | A classic sign of insufficient capacity or a stalled task |
| Heartbeat arrival | Whether replication is alive end to end | Silence means the path is broken, even if processes look healthy |
| Checkpoint latency and freshness | How up to date the translated consumer positions are | Stale checkpoints mean a rougher failover |
| Task and connector status | Whether tasks are running, paused, or failed | A failed task quietly stops copying its partitions |
| Worker resource use | CPU, memory, and network of the workers | Saturation causes lag |
| Topic and partition parity | Whether target topics have the same partitions and matching configuration | Mismatch breaks partition alignment and consumer expectations |
The heartbeat trick
The heartbeat connector writes a small timestamped event into a heartbeat topic in the source cluster. This event is replicated like any other data. By looking at the heartbeats in the target, and comparing their timestamps to the current time, you get a measure of the end-to-end delay of the entire replication path, tested with real data, not merely a check that processes are running.
Alert on stale heartbeats
If no new heartbeat arrives in the target for several intervals, treat replication as broken and investigate immediately.
Alert on rising replication latency
Set thresholds based on your recovery point objective, so you know when your promise is at risk.
Alert on failed tasks
A failed task may not stop the whole connector, so status must be checked per task.
Regularly compare source and target
Periodically confirm that the record counts and latest offsets of key topics are consistent within an expected margin of lag.
A common failure is a MirrorMaker 2 process that stays running while copying nothing, for example because a topic pattern does not match, permissions are missing, or a task has failed. Always monitor the data actually arriving in the target, not just whether the process is alive.
16Security Considerations
MirrorMaker 2 holds keys to two clusters, and it copies data across networks. Both facts need care.
Encrypt the traffic
Use TLS on connections to both clusters, especially across the long link, which often crosses networks you do not fully control.
Use dedicated identities
Give MirrorMaker 2 its own credentials on each cluster, so its activity is identifiable in logs and its permissions can be limited.
Grant only the needed permissions
On the source it needs to read and describe the replicated topics and groups. On the target it needs to write, create, and describe the remote topics, adjust their settings, and use its helper topics.
Protect the credentials
Store secrets in a proper secret store, not in plain text in configuration files or repositories.
Mind data residency and privacy rules
Replication moves data into another location. Make sure the target’s country, region, and access rules are allowed to hold what you copy.
Secure the backup as strongly as the primary
A disaster recovery cluster holds a full copy of your data, so weak security there is a back door to everything.
If topic access rules are copied to the target, remember that the principals in them must also exist and be meaningful in the target cluster. Review the resulting permissions instead of assuming they are right, and confirm that the target’s own security model is consistent.
A backup warehouse full of exact copies of your goods must have locks and guards as good as the main warehouse. Otherwise thieves simply go to the weaker building.
17Surprises: Schemas, Naming, and Other Gotchas
Most MirrorMaker 2 problems are not about the tool itself. They are about the things around it.
Schemas and the registry
Many systems store data in formats such as Avro or Protobuf, where each record carries a small identifier pointing to a schema stored in a separate schema registry service. MirrorMaker 2 copies the record bytes exactly, including that identifier. If the target cluster’s applications use a different registry, or a registry where the same identifier means a different schema, they cannot decode the data. You must make sure that the schemas are available under the same identifiers in the registry that the target applications use, using the registry’s own replication or linking features, or a design that shares one registry.
Data can replicate perfectly, and yet every consumer in the backup fails to read it because the schema identifiers are unknown. Test reading replicated data from the target with real consumers, not just counting records.
Other frequent surprises
Topic patterns that match nothing, or too much
A typo in the include pattern means nothing is copied, while a very broad pattern copies scratch topics and floods the link. Review patterns and list what is actually being replicated.
Consumer group offsets that never appear
Automatic offset syncing only updates target groups that have no active members. If the application is already running against the backup cluster, its group is skipped.
Applications hard-coded to the original topic name
After failover with the default naming policy, they find nothing to read. Plan topic naming and subscription patterns during design.
Partition count differences
If someone changes partitions on a remote topic manually, the partition numbers no longer align, and replication can fail or misplace data.
Internal and system topics
By default, internal topics such as consumer offsets and names ending in an internal marker are excluded. Trying to copy them can cause conflicts and is rarely useful.
Clock differences and retention
Retention in the target is measured from record timestamps. Very old data replicated into a topic with short retention may be deleted almost immediately.
18MirrorMaker 2 Compared with Other Approaches
MirrorMaker 2 is not the only way to connect clusters. Knowing the alternatives helps you choose well.
| Approach | How It Works | Strengths | Weaknesses |
|---|---|---|---|
| MirrorMaker 2 | Connectors that consume from one cluster and produce to another | Built into Kafka, free, flexible topologies, offset translation, wide adoption | Offsets differ, asynchronous, duplicates possible, you operate it |
| Stretch cluster | One Kafka cluster spread over several locations using rack awareness | No separate copy process, synchronous replication possible, offsets identical | Needs very low latency between sites, and a single cluster is a single failure domain for cluster-level problems |
| Byte-for-byte cluster linking from commercial platforms | The target cluster fetches directly from the source’s partitions | Preserves offsets exactly, simple failover | Vendor-specific and usually commercial |
| Custom Connect or application code | Your own program reads from one cluster and writes to another | Total control, transformations possible | You rebuild everything MirrorMaker 2 gives you |
| Dual writes from applications | Applications write to both clusters themselves | Seems simple | Risk of inconsistency when one write fails, and complex application logic |
How to choose
Are your sites within a few milliseconds of each other?
If yes, and you need zero data loss, a stretch cluster may be worth considering. If not, asynchronous replication like MirrorMaker 2 is the realistic choice.
Do you need identical offsets?
If exact offsets matter, consider tools that preserve them. If translated offsets are acceptable, MirrorMaker 2 does the job.
Are you comfortable operating another system?
MirrorMaker 2 is free software, but someone must run, monitor, and upgrade it. Managed offerings can reduce this burden.
Do you need to transform data in flight?
MirrorMaker 2 copies bytes as they are. For filtering or reshaping, use a stream processing application between clusters, or a custom connector.
Writing to two clusters from the application looks easy, but if one write succeeds and the other fails, the clusters silently diverge. Replication that reads from one authoritative source is far more dependable.
19Design Patterns and Anti-Patterns
A few habits separate calm MirrorMaker 2 deployments from anxious ones.
Helpful patterns
Replicate by explicit list
Choose the topics that need protection on purpose, rather than copying everything by default.
Idempotent consumers
Design downstream processing so that repeated events are harmless, making failovers and retries safe.
Regular failover drills
Practise the full switch, measure it, and improve the runbook after each rehearsal.
Configuration as code
Keep the MirrorMaker 2 configuration, cluster aliases, and topic lists in version control with review.
Anti-patterns to avoid
Problem
The team sets up replication once, assumes it works, and never tests a failover.
Why It’s Harmful
On the day of a real disaster they discover that applications cannot read the prefixed topics, consumer positions are missing, schemas are unavailable, or credentials were never created on the backup. Recovery takes hours instead of minutes.
Correct Approach
Rehearse failover and failback regularly in a realistic environment, and turn every surprise into an update to the runbook.
Problem
Using the identity naming policy in a two-way active-active setup.
Why It’s Harmful
Without prefixes, the same topic is copied back and forth endlessly, creating loops, exploding data volumes, and duplicates everywhere.
Correct Approach
Use the default prefixing policy for any bidirectional design, and reserve identity naming for strictly one-way migrations.
Problem
Monitoring only whether the MirrorMaker 2 processes are running.
Why It’s Harmful
Processes can be healthy while replication is stalled, lagging by hours, or copying nothing because of a pattern or permission problem.
Correct Approach
Monitor heartbeats, replication latency, task status, and the freshness of data in the target, and alert on the recovery point you have promised.
Problem
Writing the same business entity in both regions of an active-active design with no conflict rule.
Why It’s Harmful
Updates cross in flight and are applied in different orders in each region, leaving the copies inconsistent with no clear winner.
Correct Approach
Assign each entity a home region for writes, or design explicit conflict resolution, and treat remote copies as read-mostly views.
Problem
Leaving the backup cluster with weaker durability settings and weaker security than the primary.
Why It’s Harmful
Topics in the backup may have a low minimum in-sync replica setting or a single replica, and the cluster may be an easier target for attackers, undermining the entire purpose of the backup.
Correct Approach
Review durability and security settings on the target as carefully as on the source, remembering that some settings are not copied by default.
Problem
Replicating data encoded with schema identifiers without planning schema availability in the target.
Why It’s Harmful
The data arrives intact but cannot be decoded, and the team discovers this only during a failover.
Correct Approach
Replicate or share the schema registry with matching identifiers, and test reading replicated data with real consumers.
20Real-World Use Cases
Here is how organisations typically put MirrorMaker 2 to work.
Bank: disaster recovery across regions
Payment and account topics are replicated to a standby cluster in another region. Consumer group offsets are synced, failover drills happen quarterly, and monitoring alerts if replication lag exceeds the agreed recovery point.
Global retailer: regional clusters with a shared view
Stores in each continent write to a local cluster for speed. Selected topics are replicated to a central hub, where the company’s analytics and forecasting systems read a combined picture of sales.
Cloud migration
A company replicates its on-premises topics into a new cloud cluster with the identity naming policy. Consumers are moved one application at a time, then producers, and finally the old cluster is retired.
Manufacturer: edge to central
Every factory runs a small Kafka cluster that keeps working even if the internet link drops. When the link returns, MirrorMaker 2 catches up and sends the backlog to the central cluster.
Analytics offload
Data scientists read from a replicated cluster, so that heavy experimental queries never slow the production cluster that runs the business.
Sharing data with a partner
A carefully chosen set of topics is replicated to a cluster the partner can access, instead of giving the partner direct access to the production cluster.
Compliance-driven regional copies
Data for customers in a regulated region is kept in a cluster inside that region, with only permitted, anonymised topics replicated onward.
21A Troubleshooting Journey
Let us follow a realistic incident, from a failed drill to a lasting improvement.
A company runs an active-passive setup with MirrorMaker 2 between a primary and a backup cluster. During a scheduled failover drill, the team points applications at the backup cluster, and immediately several problems appear: a few applications process nothing, and one application reprocesses several hours of data.
Check the topics the applications are reading
The idle applications are subscribed to the original topic names. In the backup cluster the data lives under the primary’s alias prefix, so their subscriptions find nothing.
Fix the subscriptions
The team changes those applications to subscribe with a pattern that matches both the local and the prefixed topic names, so the same code works in either cluster.
Investigate the reprocessing application
Its consumer group had no translated position in the backup. The group name was not covered by the group include rules, so no checkpoints were ever created, and the application started from the earliest available data.
Widen the group rules and enable offset syncing
They include all production groups in the configuration, and enable automatic syncing of group offsets so that positions are written to the backup ahead of time.
Measure the real numbers
The drill showed a replication delay of a few seconds, a recovery time of forty minutes, and a small number of duplicate events. These become the documented, honest figures for the recovery objectives.
Turn lessons into permanent controls
They add an alert on stale heartbeats and rising replication latency, make idempotent processing a rule for new consumers, add schema availability to the drill checklist, and schedule the next drill.
The lesson is that the copying itself worked perfectly. Every problem was in the surroundings: naming, group selection, and preparation. That is why failover drills are so valuable, because they find these gaps when nothing is at stake.
22Frequently Asked Questions
It is the tool included with Apache Kafka that replicates topics, topic settings, and consumer group progress from one Kafka cluster to another, built on top of Kafka Connect.
Internal replication copies partitions between brokers of the same cluster for safety within one location. MirrorMaker 2 copies data between separate clusters, usually in different locations, and the two clusters have independent offsets.
By default the source cluster’s alias is added in front of the topic name. This keeps local topics and remote copies distinct and prevents endless replication loops in two-way setups.
No. Each cluster assigns its own offsets. MirrorMaker 2 records offset mappings and produces checkpoints so that consumer group positions can be translated approximately from one cluster to the other.
Possibly a little. Replication is asynchronous, so events written just before a failure may not have reached the backup. The amount at risk equals the replication lag at that moment, which is why you monitor it.
Yes. Copying is at-least-once by default, and translated offsets after failover lean toward rereading rather than skipping. Make consumers idempotent. Newer versions offer exactly-once options for copying, subject to version and configuration.
Order is preserved within each partition, because records keep their partition number. There is no global order across partitions or across independent clusters.
Not by themselves. Checkpoints hold translated positions, and newer versions can automatically write them into target groups that have no active members. Applications still need to be redirected and to read the right topic names.
A common recommendation is close to the target cluster, so that it consumes across the long link and produces locally. It can run as a dedicated cluster, on an existing Kafka Connect cluster, or through managed services.
Yes, with an identity replication policy available in newer versions. It is useful for one-way migrations, but it removes loop protection, so it must not be used for two-way replication.
No. It copies record bytes as they are, including any schema identifiers. You must make the schemas available under the same identifiers in the registry used by the target applications.
No. Records are copied, but transaction boundaries from the source are not recreated in the target.
Watch heartbeat arrival in the target, replication latency, the status of every task, the lag of the replication tasks, and the freshness of checkpoints. Also verify that target data can actually be read by real consumers.
It can be powerful for low latency and full use of both clusters, but it requires careful design. There is no global order, and entities written in both places need home regions or conflict rules.
Rehearse failover regularly and monitor real end-to-end replication health. Most problems appear in the surroundings, such as naming, groups, schemas, and credentials, not in the copying itself.
23Summary and Key Takeaways
MirrorMaker 2 is Kafka’s built-in solution for replicating data between clusters. It runs three connectors on the Kafka Connect framework: one copies topic data and settings, one translates consumer group progress into checkpoints, and one emits heartbeats to prove the path is alive. By default it names remote topics with the source cluster’s alias, which keeps copies distinct and prevents loops, and it preserves keys, values, headers, timestamps, and partition numbers, but not offsets.
Success comes from being clear about your goal, choosing the right topology, selecting the topics and groups that really matter, and designing consumers to tolerate repeats. It also comes from monitoring real end-to-end lag rather than process health, securing the backup as strongly as the primary, planning for schemas and topic names, and practising both failover and the harder failback. The tool copies faithfully. The reliability of your disaster recovery is decided by everything you build around it.
Key Takeaways
- It replicates between clusters, not within one — MirrorMaker 2 copies topics, settings, and consumer progress across separate Kafka clusters, using Kafka Connect connectors.
- Three connectors, three jobs — the source connector copies data, the checkpoint connector translates consumer positions, and the heartbeat connector measures whether the path is alive and how delayed it is.
- Prefixed names prevent loops — remote topics carry the source alias by default, so consumers must read the prefixed names after failover, and identity naming is for one-way migrations only.
- Offsets are translated, not copied — each cluster has its own offsets, checkpoints give approximate positions that favour rereading, and automatic group offset syncing helps only for groups with no active members on the target.
- Copying is asynchronous and at-least-once by default — expect replication lag that defines your data loss window, and design idempotent consumers for duplicates; transactions and global ordering do not carry across.
- Choose topology deliberately — active-passive is simplest, active-active needs home regions or conflict rules, and hub-and-spoke or fan-out suit aggregation and distribution.
- Rehearse and monitor the real thing — practise failover and failback, watch heartbeats and replication latency instead of process status, and plan for schemas, credentials, and target safety settings before the disaster arrives.