A single cluster can be very large and very reliable. Yet many organisations end up running several clusters, and then the trouble begins.

Kafka in one paragraph

Apache Kafka stores streams of messages, called records, in named channels called topics. Each topic is split into partitions, and every record in a partition gets a position number called an offset. Applications that write records are producers, and applications that read them are consumers. A group of Kafka servers working together is called a cluster.

Reasons for having more than one cluster

flowchart LR
    subgraph OnPrem[Own data centre]
        C1[(Cluster A)]
    end
    subgraph CloudR1[Cloud region 1]
        C2[(Cluster B)]
    end
    subgraph CloudR2[Cloud region 2]
        C3[(Cluster C)]
    end
    C1 |Data must flow| C2
    C2 |Data must flow| C3
        

The core problem

Once you have two clusters, the same question keeps appearing: how do we get the data from one to the other, reliably, without losing track of where each application is in its reading? Answering that question well is the whole reason Cluster Linking exists.

Before Cluster Linking, the usual solution was a separate program that reads from one cluster and writes to the other.

How the older approach works

Tools such as MirrorMaker, and its improved second version, work like a pair of applications glued together. One part acts as a consumer and reads records from the source cluster. The other part acts as a producer and writes those records into the destination cluster. In many setups this runs inside a separate service, such as Kafka Connect, which itself needs to be installed, scaled, secured and monitored.

flowchart LR
    S[(Source cluster)] -->|Consumer reads| M[Separate copy tool]
    M -->|Producer writes| D[(Destination cluster)]
        

The problems this creates

Cluster Linking connects two Kafka clusters directly and keeps chosen topics identical between them, without any separate copying program.

Cluster Linking is a feature that lets one Kafka cluster pull topics from another cluster and keep exact copies of them, including the position numbers of every record. The copying is done inside the Kafka brokers themselves, using the same mechanism that brokers already use to copy data between each other for safety. There is no consumer-and-producer pair in the middle, and no separate service to run.

The three big ideas

A few words appear again and again. Once these are clear, everything else falls into place.

Why mirror topics are read-only

If both clusters allowed writes to the same topic, they would quickly disagree about which record sits at which position, and the exact match would be lost. To keep copies perfect, a mirror topic accepts data only from the link. Applications may read from it as much as they like, but they cannot write to it until it has been detached from its source through promotion or failover.

The trick is that the destination cluster behaves like a very well-behaved follower of the source cluster.

A quick reminder about replication inside a cluster

Inside one Kafka cluster, each partition has a leader broker and follower brokers. Followers fetch data from the leader and store an exact copy at the same offsets. That is how Kafka survives a broker failure. Cluster Linking extends that same idea across cluster boundaries: the destination’s brokers act like remote followers that fetch from the source’s leaders.

flowchart LR
    subgraph SRC[Source cluster]
        SL[Leader of partition 0]
    end
    subgraph DST[Destination cluster]
        ML[Mirror partition 0]
    end
    ML -->|Fetch bytes over the link| SL
    SL -->|Records with original offsets| ML
        

Why this is more efficient

Since the data is copied as raw batches of bytes, the destination does not need to unpack each record, rebuild it and write it again. It also does not need a consumer group, a producer or a separate service. This is why the approach is lighter than a traditional copy tool, and why offsets, timestamps and even the arrangement into partitions can be kept exactly the same.

Keeping the same position numbers sounds like a small detail. In practice, it removes the hardest part of moving between clusters.

What consumers remember

A consumer saves its progress as an offset for each partition, meaning “I have read up to here.” When it restarts, it continues from that saved place. The saved offset only makes sense inside the cluster where it was recorded. If the same data lives at different offsets in another cluster, the saved number points to the wrong place.

Consumer offset syncing

The link can also copy the saved positions of consumer groups from the source cluster to the destination. You can choose which groups to include with filters. When you later switch a consumer application to the destination, it can continue from roughly where it stopped, limited only by the sync interval and by the lag of the mirror.

A mirror topic passes through a small number of stages, from creation to becoming a normal topic.

flowchart LR
    A[Created as mirror topic] --> B[Active: read-only and catching up]
    B --> C{Decision}
    C -->|Source is healthy, planned switch| D[Promote: wait until fully caught up]
    C -->|Source is down or lost| E[Failover: stop immediately]
    D --> F[Normal writable topic]
    E --> F
    B --> G[Paused: temporarily stopped]
    G --> B
        

Promote versus failover

A link can carry more than raw records. Knowing exactly what travels avoids surprises.

Schemas deserve special attention

Records that use a schema registry contain a small number that refers to a schema stored in that registry. If the destination applications use a different registry, or a registry without the same schemas and the same numbers, they may not be able to read the records. Teams therefore plan how schemas travel with the data, for example by sharing one registry or by using registry features that keep schemas in step.

No changes on the way

Because Cluster Linking copies bytes exactly, you cannot filter records, rename fields, change the number of partitions of a mirror or transform the data as it passes. If you need such changes, you add a separate processing step afterward, for example a stream processing application on the destination that reads the mirror and writes a new topic.

You do not need to remember every setting. It is enough to understand the groups of choices you make.

Automatic mirroring with filters

Instead of creating mirror topics one at a time, you can tell the link to create them automatically for every source topic whose name matches a pattern. New topics that appear later in the source, and that match the pattern, are mirrored without any manual step. This is convenient for large environments, but it needs careful patterns, because a pattern that is too broad may copy topics you never meant to move, along with their cost.

Prefixes for clarity

A prefix on mirror topic names, such as the name of the source region, makes it obvious where a topic came from and helps avoid clashes when the destination already has topics with the same names or when several sources are linked in. The trade-off is that applications must use the prefixed name, which may need changes at switch-over time.

A link carries all your data between clusters, so it must be protected like any other important connection.

Encrypt and authenticate the link

The destination connects to the source just like any client, so it needs a way to prove its identity and a way to protect the connection from eavesdropping. Use encrypted connections, and give the link its own dedicated identity, with a password or key or a certificate, instead of reusing a person’s account. That identity should be able to do only what the link needs: read the chosen topics, and, if syncing is turned on, read group information and access rules.

Network paths

The destination brokers must be able to reach the source brokers over the network, on the ports used for Kafka. Across data centres or clouds, this may mean private connections, peering between networks, or careful firewall rules. If the source sits behind a strict firewall that only allows outgoing connections, a link that is started from the source side may be the answer. Private networking options reduce exposure to the public internet and are preferred for sensitive data.

Data protection and regulations

Mirroring copies data into another region or another country. That may be forbidden or restricted by privacy laws or company policy. Before linking clusters across borders, check which data may leave its home region, and choose topics accordingly. Remember that everything in a mirrored topic is copied, including any sensitive fields inside the records.

Many companies live in two worlds at once: their own data centres and one or more public clouds. Cluster Linking is a bridge between them.

flowchart LR
    subgraph ONP[On-premises cluster]
        T1[Orders topic]
    end
    subgraph CLD[Cloud cluster]
        M1[Mirror of orders topic]
    end
    T1 -->|Cluster link| M1
    M1 --> A1[Cloud analytics]
    M1 --> A2[Machine learning services]
        

Why this is better than many direct connections

Without linking, every cloud application that needs on-premises data would connect back to the on-premises cluster across the network. That multiplies traffic, complicates security and makes every application depend on the distant cluster. With a link, the data crosses the distance once, and all local applications read from the nearby copy.

Moving a busy Kafka system is one of the scariest jobs in operations. Linking turns it into a calm, staged process.

Why migrations are hard

Dozens or hundreds of applications write to and read from the old cluster. Each has saved positions. You cannot stop the business for a weekend, and you cannot move everything at the same second. You need a way for both clusters to hold the same data during the move, and a way for applications to switch one at a time.

flowchart TD
    A[Create link from old cluster to new cluster] --> B[Mirror topics and sync consumer offsets]
    B --> C[Wait until mirroring lag is near zero]
    C --> D[Move consumers to the new cluster, group by group]
    D --> E[Stop producers on the old cluster]
    E --> F[Promote mirror topics once fully caught up]
    F --> G[Point producers at the new cluster]
    G --> H[Decommission the old cluster]
        

A second cluster in another region is an insurance policy. Cluster Linking keeps that policy up to date.

The active and standby pattern

In the most common disaster recovery design, one cluster does all the work, and a second cluster in another region quietly mirrors its important topics through a link. If the main region fails, applications are switched to the standby cluster. Because offsets are preserved and consumer positions can be synced, applications can carry on from nearly where they stopped.

sequenceDiagram
    participant App as Applications
    participant Pri as Primary cluster
    participant Sec as Standby cluster
    App->>Pri: Normal operation, writes and reads
    Pri-->>Sec: Link keeps mirror topics up to date
    Note over Pri: Region fails
    App->>Sec: Operations team fails over mirror topics
    Sec-->>App: Topics writable, consumers resume from synced offsets
        

Two numbers every recovery plan needs

A typical failover checklist

Failing back afterwards

When the original region returns, its data is behind the standby, which has accepted new writes. Returning to it needs a plan: typically you rebuild the old primary as a mirror of the new one, wait until it catches up, and then perform a planned switch back. Some environments support reversing the direction of a link to make this easier. Whatever the method, practise it, because failing back badly can cause data loss just as failing over badly can.

A mirror is not only a backup. It is also a local copy that applications can read cheaply and quickly.

Both connect clusters. They make different trade-offs, and the right choice depends on your needs and your platform.

With more than two clusters, the arrangement of links matters as much as the links themselves.

flowchart TD
    subgraph HS[Hub and spoke]
        H[(Hub cluster)] --> S1[(Spoke 1)]
        H --> S2[(Spoke 2)]
        H --> S3[(Spoke 3)]
    end
    subgraph FI[Fan-in]
        A1[(Regional 1)] --> AG[(Aggregate cluster)]
        A2[(Regional 2)] --> AG
        A3[(Regional 3)] --> AG
    end
    subgraph AS[Active and standby]
        P[(Primary)] --> SB[(Standby)]
    end
        

Avoiding loops and confusion

Because mirror topics are read-only, you cannot accidentally create the endless copy loops that plagued some older two-way setups. But you still must plan naming. In an active-in-both-places design, topics should carry a prefix such as the region name, so that readers know which cluster wrote the data and never mix two histories in one topic.

Chains of links

A destination can itself be the source for another link, forming a chain from one cluster to a second and then to a third. Each hop adds delay, and the total lag is the sum of the delays. Keep chains short, and watch lag at every step.

Copying data across distance costs time and money. Understanding both keeps surprises away.

Throughput

The speed of a link depends on the network between the clusters, the load on the source brokers and the number of partitions being mirrored. Since each partition is copied independently, more partitions give more parallelism. A slow or congested long-distance connection is the most common limit, so measure real bandwidth between the regions before deciding how much to mirror.

Delay and lag

Copying is asynchronous. A record written to the source appears in the mirror after a delay, usually short, but growing with distance, load and network trouble. This delay is the mirroring lag. It matters most for disaster recovery, because it decides how much recent data might be missing at failover, and for applications that read the mirror, because they see events slightly later than readers of the source.

Cost

A link that silently stops is worse than no link, because everyone believes the copy is safe. Watch it closely.

Turning numbers into alerts

Most link problems fall into a few familiar groups: reaching the source, permissions, and mismatched expectations.

flowchart TD
    A[Data not appearing in the mirror] --> B{Is the link active?}
    B -- No --> B1[Check network path, credentials and encryption]
    B -- Yes --> C{Is the topic covered by a mirror or filter?}
    C -- No --> C1[Create the mirror topic or fix the filter]
    C -- Yes --> D{Is the mirror active, not paused or failed?}
    D -- No --> D1[Resume or repair the mirror topic]
    D -- Yes --> E{Is lag growing?}
    E -- Yes --> E1[Check bandwidth and broker load]
    E -- No --> F[Check that the source is actually producing]
        

Every design has edges. Knowing them early prevents unpleasant surprises during a real event.

Habits that keep links healthy and traps that turn them into hidden risks.

Good habits

Anti-patterns to avoid

Let us see how linking helps in situations that teams often face.

It is a feature that lets one Kafka cluster fetch topics directly from another and keep exact, read-only copies, called mirror topics, with the same partitions and offsets.

No. It is a feature of Confluent Platform and Confluent Cloud. Open-source Kafka users usually rely on MirrorMaker for copying between clusters.

MirrorMaker consumes and then produces records through a separate Connect service and usually changes offsets. Cluster Linking copies raw bytes inside the brokers, keeps offsets identical and needs no extra service, but it cannot transform data.

A read-only topic in the destination cluster that stays in step with a topic in the source. It becomes a normal writable topic only after promotion or failover.

Records are copied byte for byte into the same partitions at the same positions. Consumers can therefore switch clusters and continue from a saved offset without complicated translation.

Promote is a planned conversion that waits for the mirror to catch up fully, so nothing is lost, and needs a healthy source. Failover converts immediately when the source is unavailable, and may lose records that were not yet copied.

No, not while it is a mirror. Applications can only read. After promotion or failover, it becomes a normal topic that accepts writes.

Not in an emergency. Copying is asynchronous, so recent records may be missing after a failover. A planned promotion with a healthy source does not lose data.

No. The link copies topic data. Schemas in a schema registry need their own arrangement so that applications in the destination can decode the records.

No. Records are copied unchanged. If you need changes, add a stream processing step on the destination that reads the mirror and writes a new topic.

Link status, the state of each mirror topic, mirroring lag, throughput, consumer offset sync health and the load on both clusters. Alert on anything that shows the copy has stopped or is falling behind.

Not to the same mirrored topic. Because mirrors are read-only, each region should own different topics, typically with region prefixes, and readers combine them.

Cluster Linking connects two Kafka clusters so that the destination cluster fetches chosen topics directly from the source and stores exact, read-only copies, called mirror topics, with the same partitions, offsets and timestamps. Because the copying happens inside the brokers and needs no separate service, it is simpler to run than traditional copy tools, and because offsets are preserved, moving consumers between clusters becomes far easier. It fits migrations, disaster recovery, hybrid and multi-cloud sharing and local reads. Mirror topics turn into normal topics through promotion, which waits for a full catch-up, or failover, which acts immediately and may lose recent data. The feature belongs to Confluent Platform and Confluent Cloud, cannot transform data and leaves schemas, applications and connectors for you to plan. With careful topic selection, narrow security identities, good monitoring and regular drills, it becomes a dependable bridge between clusters.