The plain-English idea first. The machinery follows in later chapters.

A region is a geographic area that holds one or more data centers, such as Mumbai, Singapore or Frankfurt. Cloud providers sell regions as separate places with separate power, cooling and networks. Two regions are far enough apart that a flood, a fire or a bad software update in one is very unlikely to touch the other.

A multi-region design is any system that keeps running parts, or copies of its data, in more than one region. There are two big families. In an active-passive design, one region does all the work and the other waits with a copy of the data. In an active-active design, every region does real work at the same time: it accepts new data from local applications, serves local readers, and shares its data with the other regions.

What Kafka brings to the picture

Apache Kafka is a system that stores streams of events, such as “order placed” or “payment received”, in an ordered log, and lets many programs read them. Kafka was designed to run inside one data center or one region, where machines are close together and the network is fast and reliable. Spreading it over regions means dealing with slower links, longer delays and the chance that the link between regions breaks while both sides stay alive.

That single fact, that the link between regions can be slow or broken while both regions are healthy, drives almost every decision in this guide.

A short tour of the parts of Kafka that the rest of this guide keeps pointing at.

If you already know Kafka well, skim this chapter. If not, these few ideas are all you need. Each has a small card below, and the paragraphs after the cards show how the ideas fit together.

Replication inside a cluster versus between clusters

Kafka copies each partition to several brokers. One broker is the leader and takes all writes. The others are followers that copy the leader. The copies that are keeping up are called the in-sync replicas, or ISR. A producer can ask the leader to wait until all in-sync copies have the record before saying “done”. This setting is called acks=all, and together with a minimum in-sync count it is what makes a single cluster safe.

That copying is fast and tightly controlled because it happens inside one cluster over a fast network. Copying between clusters is a different job, done by a different tool, over a slower network, and with weaker promises. The whole art of multi-region Kafka is understanding this difference.

Offsets belong to a cluster

An offset only has meaning inside one partition of one cluster. Record number 5,000 in the orders topic of the Mumbai cluster has nothing to do with record number 5,000 in the copy of that topic in the Singapore cluster. They may hold different records, because the copying tool may skip, retry, or interleave data. This detail causes many of the surprises you will see later, especially when consumers have to move from one region to another.

Three very different pressures push teams toward more than one region. Knowing which one is yours changes the design.

Nobody builds multi-region systems for fun. The cost in money and effort is high. Teams do it because one of three pressures leaves no better choice.

Pressure one: distance is slow

Light travels through fibre at roughly two thirds of its speed in a vacuum. A signal from Mumbai to Frankfurt cannot arrive in under about 60 milliseconds one way, no matter how much money you spend. If a customer in Frankfurt has to wait for a round trip to Mumbai on every click, the app feels sluggish. Serving that customer from a nearby region is the only cure, and that means the data must live nearby.

Pressure two: whole regions fail

Regions fail less often than single machines, but they do fail. A bad configuration change, a power problem, a cooling failure, a cut cable or a software bug in the cloud provider’s own control system can make a region unreachable for minutes or hours. Companies with strict uptime promises cannot say “our provider was down”. They need a second region ready to carry the traffic.

Pressure three: laws about where data lives

Some laws and contracts require that data about citizens of a country, or customers of a certain kind, stays inside a geographic boundary. This is called data residency. Multi-region designs must keep such data in the right regions and share only what is allowed, such as anonymous totals.

Three ways to lay out Kafka, from simplest to hardest, with what each one buys you.

Before choosing active-active, compare it fairly with the other layouts. Many teams discover that a simpler layout meets their real needs.

flowchart TB
  subgraph SR["Single Region"]
    A1["Apps"] --> K1["Kafka Cluster
3 zones"]
  end
  subgraph AP["Active-Passive"]
    A2["Apps in Region A"] --> K2["Kafka A
active"]
    K2 -- "one-way copy" --> K3["Kafka B
standby"]
    A3["Apps in Region B
idle until failover"] -.-> K3
  end
  subgraph AA["Active-Active"]
    A4["Apps in Region A"] --> K4["Kafka A"]
    A5["Apps in Region B"] --> K5["Kafka B"]
    K4 -- "copy" --> K5
    K5 -- "copy" --> K4
  end

The diagram shows the three layouts side by side. In the single-region layout, one cluster spreads its brokers across several zones inside a region. It survives the loss of a machine or a zone, but not the loss of the region. In the active-passive layout, a second cluster receives a one-way copy and stays quiet until a disaster. In the active-active layout, both clusters take writes, and each one copies to the other.

The first way to use two regions: pretend they are one big cluster. It is elegant and it is expensive in delay.

In a stretch cluster, the brokers of a single Kafka cluster are placed in several regions or data centers. Kafka does not know or care that some brokers are far away. Each partition has replicas in different locations, and the normal replication rules keep them in step. Kafka’s own rack awareness feature lets you label each broker with its location, so that the replicas of a partition are spread across locations.

flowchart LR
  subgraph R1["Location 1"]
    B1["Broker 1
leader of P0"]
    B2["Broker 2"]
  end
  subgraph R2["Location 2"]
    B3["Broker 3
follower of P0"]
    B4["Broker 4"]
  end
  subgraph R3["Location 3"]
    B5["Broker 5
follower of P0"]
    Q["Metadata quorum
voter"]
  end
  P["Producer"] --> B1
  B1 -- "replicate" --> B3
  B1 -- "replicate" --> B5
  C["Consumer"] --> B1

What you gain

Replication is synchronous, meaning the producer is told “done” only after enough copies exist. If the leader’s location vanishes, another location already holds every acknowledged record, so RPO is zero for acknowledged writes. Applications see one cluster, one set of topics, one set of offsets. Consumers do not need offset translation, because the offsets are the same everywhere. There is no second tool to run, and no naming tricks.

What you pay

Every write with acks=all waits for followers in other locations. If the round trip between locations is 80 milliseconds, every write takes at least that long. Throughput per producer drops, and tail latency rises when the link is congested. Cross-region traffic is also billed by cloud providers, and a stretch cluster sends every byte across regions at least twice for replication.

There is a second, quieter cost: quorum. The cluster’s metadata controllers must agree by majority. With only two locations, losing the location that holds the majority stops the cluster from making decisions, even though the other location is healthy. That is why serious stretch clusters use three locations, where losing any one leaves a majority.

The second way: every region owns a full cluster, and a copying tool moves data between them. This is what most active-active systems use.

In the multiple-cluster design, each region runs its own independent Kafka cluster. Applications talk only to the cluster in their own region. A separate tool reads records from one cluster and writes them into the other. Because the copying happens after the local write has already succeeded, it is asynchronous: the local application is never slowed down by the distance.

flowchart LR
  subgraph RA["Region A"]
    PA["Producers A"] --> KA["Kafka Cluster A"]
    KA --> CA["Consumers A"]
  end
  subgraph RB["Region B"]
    PB["Producers B"] --> KB["Kafka Cluster B"]
    KB --> CB["Consumers B"]
  end
  KA -- "replicator A to B" --> KB
  KB -- "replicator B to A" --> KA

Why this is the common choice

Local writes stay fast, because a producer in Region A only waits for its own cluster. Failures are contained: a broken cluster in Region B does not stop Region A from taking writes. Each cluster can be upgraded on its own schedule, which limits the damage of a bad release. And each region can keep some topics private, replicating only what is allowed.

What you give up

Because copying is asynchronous, there is always a delay between the moment a record is written in Region A and the moment it appears in Region B. That delay is usually well under a second in a healthy system, but it can grow to minutes or hours during trouble. If Region A is destroyed, the records that had not yet been copied are lost. That is why RPO is greater than zero. You must also handle topic naming, offset translation, duplicates and conflicts yourself, which are the topics of the next chapters.

Several tools can move data between Kafka clusters. They differ in what they copy, how faithfully, and who supports them.

A replicator is a program that reads records from a source cluster and writes them to a target cluster. It sounds simple, and at small scale it is. At large scale, it must also handle topic creation, partition counts, configuration, access rules, consumer positions, restarts and loop prevention. Over the years, several tools have grown to cover these jobs.

The most important difference: offsets

Classic mirroring tools, including MirrorMaker 2, read records and write them again on the target. The target cluster assigns new offsets, so record 5,000 on the source may become record 4,982 on the target. Byte-for-byte linking tools copy the log itself and keep the offsets identical. Identical offsets make consumer failover far simpler, but they also mean the mirror topic is read-only and must be promoted before applications can write to it. That does not fit a design where both regions must write into the same topic name, so active-active designs still use separate topics per region, which the next chapters explain.

MirrorMaker 2 is a set of connectors that run inside Kafka Connect. Each connector has one small job.

Kafka Connect is a framework that runs connectors, which are plug-in programs that move data in and out of Kafka. It handles restarts, scaling across workers and storing its own progress. MirrorMaker 2 (MM2) is a family of three connectors that ride on top of this framework.

flowchart LR
  subgraph SRC["Source cluster A"]
    T1["orders topic"]
    G1["Consumer group positions"]
    TA["Topic configs and ACLs"]
  end
  subgraph MM["MirrorMaker 2 on Kafka Connect"]
    MS["MirrorSourceConnector
copies records"]
    MC["MirrorCheckpointConnector
translates group offsets"]
    MH["MirrorHeartbeatConnector
sends heartbeats"]
  end
  subgraph TGT["Target cluster B"]
    T2["A.orders mirror topic"]
    OS["offset-syncs topic"]
    CP["A.checkpoints.internal"]
    HB["heartbeats topic"]
  end
  T1 --> MS --> T2
  MS --> OS
  G1 --> MC
  OS --> MC --> CP
  MH --> HB
  TA --> MS

The three connectors

How offset syncs work

Every so often, the source connector records a pair of numbers: “record 5,000 on the source became record 4,982 on the target”. These pairs go into an internal topic. The checkpoint connector uses them like landmarks on a map. When it sees that a group has read up to source offset 5,010, it finds the nearest landmark at or before that point and computes a safe target offset. Because landmarks are sparse, the answer is slightly behind. That is deliberate: it is better to re-read a few records than to skip any.

Running it

MM2 can run in three ways: as a dedicated cluster using its own driver program, as connectors on an existing Kafka Connect cluster, or in a standalone mode for tests. For production, teams usually run a dedicated Connect cluster in the target region, so that the process reads across the long link and writes locally. Reading remotely and writing locally tends to be more forgiving, because a remote read that times out can simply be retried, while a remote write that times out may leave doubt about whether the record arrived.

A closer look at the engine room, since its behaviour explains most lag and duplicate stories.

The source connector splits its work into tasks. Each task is assigned a set of source partitions. A task acts like an ordinary consumer for those partitions and like an ordinary producer for the target. It has no group of its own that Kafka rebalances in the usual way. Instead, Kafka Connect hands out the partitions and stores each task’s progress in its own internal offsets storage.

Partition mapping

MM2 writes each record to the same partition number on the target that it came from on the source. That is why the partition counts of the source and mirrored topics must match: key-to-partition placement stays the same, and order inside every partition is kept. If someone adds partitions to the source topic later, the connector notices on its next refresh and adds partitions to the mirror. Records with the same key can then move to a different partition on the source side, and the mirror will follow, but old and new records for that key are split between partitions in both places.

Progress and restarts

The connector commits its progress at intervals. The commit says, “I have safely written everything up to source offset N”. If it crashes between writing a batch and committing, it restarts from the last commit and copies part of the batch again. This is the exact mechanism that produces duplicates. Shorter commit intervals make duplicates rarer and smaller, at the cost of more bookkeeping traffic.

Headers, keys and timestamps

The copier keeps the record’s key, value, headers and, in normal settings, its original timestamp. This is useful: a reader in the target region can compute how old a record is, and windowed jobs based on event time keep working. Log-append timestamps, set by the broker, are not preserved, because the target broker assigns its own.

If both regions copy everything to each other, a record could bounce back and forth forever. One naming rule stops that.

Imagine Region A copies the orders topic to Region B, and Region B copies the orders topic back to Region A. A record written in Region A is copied to B. B’s copier sees a new record in its orders topic and copies it back to A. A’s copier sees it and copies it to B again. The record circles until the disks fill up. This is a replication loop, and it is the most dangerous mistake in active-active Kafka.

The prefix rule

MM2’s default answer is to rename topics when it copies them. A topic called orders in Region A appears in Region B as a.orders, where a is the alias of the source cluster. The copier in Region B is told to copy only topics that do not already carry a prefix from the other region. So a record created locally in A, copied to B as a.orders, is not copied back, because a.orders is recognised as a remote topic that came from A. Loops become impossible by design.

flowchart LR
  subgraph A["Cluster A"]
    AO["orders
local writes"]
    ABO["b.orders
copy from B"]
  end
  subgraph B["Cluster B"]
    BO["orders
local writes"]
    BAO["a.orders
copy from A"]
  end
  AO -- "copied as a.orders" --> BAO
  BO -- "copied as b.orders" --> ABO
  ABO -. "not copied back" .-> X["stop"]
  BAO -. "not copied back" .-> X

What this means for applications

Producers in every region write to the plain orders topic in their own cluster. To see the whole world, a consumer subscribes to a pattern that matches orders and every prefixed version of it. A consumer in Region A then reads its own orders plus b.orders. Each topic has a single writer region, which is a very important property: within any one topic, only one region ever writes. This makes ordering, ownership and debugging much easier.

Follow a single order from the moment it is written in Mumbai to the moment a reader in Singapore sees it.

sequenceDiagram
  participant P as Producer in Region A
  participant KA as Kafka A
  participant MM as MirrorMaker 2
  participant KB as Kafka B
  participant C as Consumer in Region B
  P->>KA: 1. Write order 1001 to orders
  KA-->>P: 2. Acknowledge after local replicas agree
  MM->>KA: 3. Fetch new records
  KA-->>MM: 4. Return order 1001
  MM->>KB: 5. Write order 1001 to a.orders
  KB-->>MM: 6. Acknowledge
  MM->>KB: 7. Record an offset sync pair
  C->>KB: 8. Fetch from a.orders
  KB-->>C: 9. Return order 1001

Where the record can go wrong

Between steps 3 and 5, the copier might crash after writing the record but before saving its progress. On restart, it copies the record again, and now the target holds it twice. This is the source of duplicates, discussed shortly. If the source cluster is destroyed between steps 2 and 3, the record is lost from the world even though the producer was told it was safe. This is the replication gap, the reason RPO is not zero.

When a consumer moves to another region, it must find its place again. Offset translation is the map for that.

A consumer group remembers its progress as a saved offset for every partition. If a whole application moves from Region A to Region B, the saved offsets in A’s cluster are useless in B, because B’s copy of the topic has different offsets. Without help, the application has two bad choices: start from the newest records and skip data, or start from the beginning and re-process everything.

flowchart TD
  S["Consumer group reads a.orders
saved position in Cluster A"] --> CK["Checkpoint connector reads group position"]
  CK --> LM["Finds nearest offset sync landmark"]
  LM --> TR["Computes translated offset for Cluster B"]
  TR --> W["Writes checkpoint to the checkpoints topic"]
  W --> O{"Sync group offsets
enabled?"}
  O -- "yes" --> AP["Applies offsets to the same group name in Cluster B automatically"]
  O -- "no" --> MAN["Failover script reads checkpoints and sets offsets"]
  AP --> R["After failover the group resumes near its old place"]
  MAN --> R

Two ways to use the translated offsets

Automatic: newer Kafka versions can apply translated offsets straight into the target cluster’s group, as long as that group has no active members there. When the application starts in the target region, it simply resumes. Manual: the checkpoints topic is read by a failover script or by the application itself, which then sets its own starting positions. The manual route gives more control, and the automatic route gives less work at the worst possible moment.

Things that surprise people

• Translation only covers groups that consumed the mirrored topics. A group that reads the local topic in Region A is translated for the mirrored copy of that topic in Region B, which is where the group would go after failover.
• A group that is active in both clusters at once with the same name can fight the automatic sync. Keep group names clear about where they run, or make sure only one side is active for each group.
• The translated position lags behind the real one by up to one landmark gap. Expect a small burst of duplicates after failover, and design consumers so that repeats are harmless.
• Checkpoints are written on a timer. If the region dies just after a big burst of progress, the last checkpoint may be a few seconds old.

What Kafka promises inside one cluster is stronger than what you get across regions. Knowing the difference prevents nasty surprises.

Inside a single cluster, Kafka guarantees that records in one partition keep their order, and with the right settings, that acknowledged records are not lost when a broker fails. With transactions, a group of writes can succeed or fail together. Across regions, these promises weaken.

The consequence: make consumers idempotent

An operation is idempotent when doing it twice has the same effect as doing it once. Setting a customer’s address to a given value is idempotent. Adding 100 rupees to a balance is not. Since duplicates can appear on the far side, and since consumers moving between regions re-read a few records, the reliable defence is to make the consuming code tolerate repeats. The usual method is to give every event a unique event ID and to remember the IDs already handled, or to write results with an operation that overwrites by key.

The heart of active-active difficulty. Copying is easy. Agreeing is hard.

A conflict happens when two regions make different changes to the same piece of information before either has heard about the other’s change. Suppose a customer’s shipping address is updated from the phone app, handled in Region A, and at almost the same moment from the website, handled in Region B. Each region writes its new address locally. Then the replicators carry each change to the other side. Now each region holds both updates, but they arrive in opposite order. Which address wins?

sequenceDiagram
  participant A as Region A
  participant B as Region B
  Note over A,B: Both start with address Old
  A->>A: Set address to Street 1 at 10:00:00.100
  B->>B: Set address to Street 2 at 10:00:00.200
  A-->>B: Replicate Street 1
  B-->>A: Replicate Street 2
  Note over A: Sees Street 1 then Street 2
  Note over B: Sees Street 2 then Street 1
  Note over A,B: Without a rule, A ends with Street 2 and B ends with Street 1

Strategies for resolving conflicts

The cleanest approach is always avoidance. If a record can only be changed in one region, no conflict can exist. The next chapter shows how to build that, because it is the pattern behind most successful active-active systems.

The most reliable way to avoid conflicts is to make sure every piece of data has exactly one place where it may change.

In a home-region design, each entity, such as a customer, an account or a device, is assigned to one region. That region is its home. All changes to that entity are written by applications running in the home region, into the home region’s local topic. Other regions receive those changes through replication and treat them as read-only facts.

flowchart TD
  U["User request arrives
at nearest region"] --> R{"Is this user's
home region local?"}
  R -- "yes" --> L["Handle locally
write to local orders topic"]
  R -- "no" --> F["Forward request to home region
or write to a request topic"]
  F --> H["Home region handles it
writes to its local topic"]
  L --> RP["Replicator copies to other regions
as prefixed topics"]
  H --> RP
  RP --> V["Every region builds its read view
from all regional topics"]

How the home is chosen

The home can be based on where the customer lives, which is the natural choice and gives the lowest delay for most requests. It can be based on a hash of the customer ID, which spreads load evenly but may put a customer far from their home. Or it can come from a small lookup table that says which region owns which customer range. Whatever the rule, it must be stable: if two regions disagree about a customer’s home, conflicts return.

What happens when the home fails

If a home region is down, its customers cannot change their data unless another region takes over. Takeover means updating the ownership table, so that another region becomes the home. This must be a deliberate, coordinated step, ideally through a small strongly consistent system such as a metadata store, and not a decision each region makes alone. Otherwise, two regions may both believe they are the new owner. Some teams accept a short read-only period for affected customers, rather than risk a split.

Once data lives in several prefixed topics, consumers need a plan for what to read. Three plans cover most needs.

Building the aggregate view

A consumer with the aggregate pattern subscribes to a name pattern that matches the local topic and every mirrored copy. All the records arrive in one stream, but they come from different partitions and different regions, so the consumer must not assume any global order. If it needs an order, it should sort by an event time or a version number carried inside the record, and accept that late records can still arrive.

The same-shape rule

The local and mirrored topics must contain the same kind of record, with the same schema. That means schemas need to be shared between regions and evolved carefully, which is covered in the supporting systems chapter. If one region starts writing a new field that others cannot read, every aggregate reader breaks at the same moment.

Programs that keep running totals hold their own state. In two regions, that state can drift apart unless you plan for it.

Kafka Streams is a library for building applications that read topics, transform events, keep state such as counts or balances, and write results. The state lives in local stores backed by internal Kafka topics. In an active-active design, a Streams application usually runs in every region, each with its own state stores and internal topics in its own cluster.

The drift problem

Suppose each region runs a job that counts orders per product. Each job reads its local orders topic and the remote prefixed topic. If both jobs read exactly the same records, they end with the same counts. But the two jobs read the records in different orders, at different times, and after a failover may re-read some records. If the counting logic is not designed for this, the two answers slowly disagree.

Practical rules

• Prefer logic where the result does not depend on arrival order.
• Remove duplicates by event ID inside the job, with a store that expires old IDs after a safe period.
• Give every application its own application ID in each cluster and never mix internal topics between regions.
• Do not replicate internal changelog and repartition topics as a substitute for running the job in both regions. Restoring from mirrored changelogs across different offsets is fragile.
• Where order matters, let one region own the key, and let the other regions only read results.

Kafka is rarely alone. Everything around it must also work in more than one region, or the design has a hidden single point of failure.

Schema registry

A schema describes the shape of a record, such as its field names and types. A schema registry stores schemas and gives each one an ID that is embedded in every record. A reader looks up the ID to know how to decode the bytes. If Region B receives a record that carries schema ID 42 but its own registry has never heard of ID 42, the reader cannot decode it. Multi-region setups therefore need a registry whose IDs mean the same thing everywhere, either through a primary registry in one region with read-only followers elsewhere, or through a feature that links registries and keeps IDs aligned. Schema changes must be made in one place and be compatible in both directions.

Kafka Connect and source systems

Connectors that pull from databases or push to storage must also be planned. If both regions run the same connector against the same database, the same rows are published twice. Decide which region owns each source, run the connector only there, and let replication share the results. Sinks are similar: a sink that writes to a database in each region must write idempotently, since it will see duplicates.

Access rules and settings

Access control lists, called ACLs, say who may read or write which topic. Topic settings, such as retention and compaction, say how long data lives. MM2 can copy both, but you should review what it copies, since the mirrored topic name changes and some rules must be rewritten for it. Also decide whether topic settings and partition counts should match across regions. Matching partition counts is important because it preserves key-to-partition mapping.

A design is only as good as its behaviour on the worst day. Walk through the main failures before they happen.

flowchart TD
  X["Alert: replication lag rising or region unreachable"] --> Q1{"Is the region itself
healthy for its own users?"}
  Q1 -- "yes" --> Q2{"Is the link between regions broken?"}
  Q2 -- "yes" --> LC["Link cut: both regions keep working
backlog grows on the copier"]
  Q2 -- "no" --> SL["Copier too slow or under-sized
scale workers, check throttles"]
  Q1 -- "no" --> Q3{"Confirmed by more than one
independent signal?"}
  Q3 -- "yes" --> RO["Region outage: shift traffic
to the surviving region"]
  Q3 -- "no" --> MON["Monitoring blind spot: verify before acting"]
  LC --> HEAL["Link returns: backlog drains, conflicts resolved by rules"]
  RO --> FB["Later: recover the region and fail back"]

Split brain

A split brain happens when two regions each believe they alone are in charge and both act as the owner of the same data. It usually follows a network split, when each region thinks the other is dead. Home-region design plus a single authority for ownership changes prevents most of these. If the ownership rules are decided by a small strongly consistent service, only the side that can reach a majority of that service is allowed to take over.

A written, rehearsed sequence of steps beats improvisation at three in the morning.

Failover means moving work from a failed region to a healthy one. Failback means moving it back after repair. In active-active, the surviving region is already running, so failover is mostly about redirecting users and taking over ownership. The steps below assume a home-region design with translated offsets.

Cross-region traffic is the slowest and most expensive part of the system. Most tuning is about moving fewer bytes, more efficiently.

Where the time goes

A replicated record pays for a fetch from the source, a trip over a long link, and a write on the target. The long link has high latency, so the copier must keep many bytes in flight at once. TCP connections with small buffers can leave a fast link mostly idle. Larger socket buffers, bigger batches and compression usually raise throughput far more than adding more copier tasks.

Cost is a design input

Cloud providers charge for data leaving a region. A design that replicates every topic in both directions to three regions multiplies transfer cost. Count it early: bytes per day per topic, times the number of destination regions, times the provider’s rate. Many teams are surprised to find that transfer costs exceed the cost of the brokers themselves.

Capacity planning for the survivor

When one region dies, the other must carry its traffic. If each region runs at 70 percent of capacity, the survivor needs 140 percent, and it will fail. Size each region to handle the whole load, or accept planned degradation, such as turning off non-critical features during an outage. The same holds for the replicator: after an outage, the backlog must drain at the same time as normal traffic, so it needs spare throughput.

Systems grow. Adding a third region is easier when you plan the steps and keep every change reversible.

Adding a region is not just building another cluster. Every existing region must start copying to it, it must start copying to every existing region, and the ownership table must learn about it. With two regions there are two replication directions. With three there are six. With four there are twelve. The count grows with the square of the regions, and so do the number of dashboards, alerts and ways to make a mistake.

Watch out for the quadratic growth

Some teams avoid the full mesh with a hub-and-spoke layout, where every region copies to and from a central hub cluster, not directly to every other region. This cuts links from the square to a straight line, but it adds an extra hop for data crossing between two spokes, and the hub becomes a critical component that must itself be highly available. Choose the mesh for small numbers of regions and low delay, and hub-and-spoke when the region count grows.

The settings that matter most, grouped by where they live. Names follow open-source Apache Kafka; check them against your version.

MirrorMaker 2 settings

Producer, consumer and broker settings that matter

Data crossing regions crosses trust boundaries, legal boundaries and public networks. Each one needs a lock.

Protect the link

All cross-region traffic must be encrypted in transit with TLS, and both ends must prove who they are. Mutual TLS or a strong authentication mechanism, such as SASL with short-lived credentials, prevents an attacker from posing as a replicator. Prefer private connectivity, such as peering or dedicated links between regions, over the public internet.

Give the replicator the smallest rights

The replicator needs to read the topics it copies and write to the mirrored topics, and to manage its own internal topics and groups. It should not have administrator rights over the whole cluster. Separate identities per direction help: the identity that writes into Region B should not be able to write to local topics that only local applications own.

Respect data boundaries

If some data must stay in a country, do not replicate its topic, or replicate only a filtered, masked version. A common design keeps personal topics regional and publishes derived, anonymous events for global use. Test the filter, because a missing filter is a legal problem, not a bug.

Other important points

• Encrypt data at rest in every region with keys managed in that region, or with keys that each region can reach even when another region is down.
• Audit who reads mirrored topics; the copy in Region B is as sensitive as the original in Region A.
• Rotate credentials in both regions on a schedule, and rehearse it. A certificate that expires in one region can silently stop replication.
• Apply retention rules consistently. Deleting a customer’s data means deleting it in every region, including mirrors and backups.

You cannot manage what you cannot see. A few well-chosen signals tell you whether both regions are healthy and in step.

flowchart LR
  subgraph RA["Region A"]
    KA["Kafka A metrics"]
    MMA["Replicator metrics"]
  end
  subgraph RB["Region B"]
    KB["Kafka B metrics"]
    MMB["Replicator metrics"]
  end
  HB["Heartbeat topic
end-to-end delay"]
  KA --> DASH["Dashboards and alerts
hosted outside both regions"]
  KB --> DASH
  MMA --> DASH
  MMB --> DASH
  HB --> DASH
  DASH --> ONC["On-call engineer"]

Measure RPO directly

Replication lag expressed in records is hard to interpret. Convert it into time: “the far region is 4 seconds behind”. That number is your live estimate of how much would be lost if the source region vanished right now. Track it, graph it, and alert when it exceeds the business target.

Logging that helps later

Log every ownership change, every failover step, and every conflict decision with the event ID and region. When a customer asks why their update disappeared, this log is the only way to answer.

A few shapes recur in every successful design, and a few mistakes recur in every painful one.

Patterns worth using

Anti-patterns to avoid

How well-known companies have described their multi-region streaming setups in public, in general terms.

A checklist to review before going live, and a list of mistakes that show up again and again.

Mistakes seen most often

• Treating asynchronous as synchronous. Teams tell the business that no data can be lost, then discover the far copy was seconds behind.
• Forgetting the small services. Kafka is multi-region, but the schema registry, identity provider or configuration store sits in one region.
• Testing only the happy path. Failover is tried once in a calm test, never under load, never with a partial network failure.
• Unbounded reprocessing. After failover, a consumer starts from the beginning of a topic and overloads downstream systems.
• Silent filters. A replicator topic filter that does not match a new topic means the data is never copied and nobody notices for weeks.

Confidence in a multi-region design comes from breaking it on purpose while everyone is watching.

A game day is a planned exercise in which the team injects a failure into a real or production-like environment and observes what happens. The point is not to prove that the design works. The point is to find what does not, while the cost is low.

All the ideas together in one made-up system. The company and numbers are invented for teaching.

CartCloud sells goods to customers in India, Southeast Asia and Europe. It runs three regions, which we call Mumbai, Singapore and Frankfurt. Its goals are simple: pages must feel fast everywhere, the shop must survive the loss of any one region, and customer data of European customers must stay in Europe.

flowchart TB
  subgraph MUM["Mumbai"]
    AM["Cart and order services"] --> KM["Kafka Mumbai"]
  end
  subgraph SIN["Singapore"]
    AS["Cart and order services"] --> KS["Kafka Singapore"]
  end
  subgraph FRA["Frankfurt"]
    AF["Cart and order services"] --> KF["Kafka Frankfurt"]
  end
  KM -- "catalog and price events" --> KS
  KS -- "catalog and price events" --> KM
  KM -- "catalog and price events" --> KF
  KF -- "catalog and price events" --> KM
  KS -- "catalog and price events" --> KF
  KF -- "catalog and price events" --> KS
  KM -- "anonymous order stats only" --> AGG["Global analytics cluster"]
  KS -- "anonymous order stats only" --> AGG
  KF -- "anonymous order stats only" --> AGG

Decisions and reasons

A bad day: Singapore goes dark

Answer a few honest questions and the right layout usually becomes obvious.

flowchart TD
  S["Start: do you need more than one region?"] --> Q1{"Must the service survive
a whole region loss?"}
  Q1 -- "no" --> SR["Single region across 3 zones"]
  Q1 -- "yes" --> Q2{"Is zero data loss required
and are sites close together?"}
  Q2 -- "yes" --> ST["Stretch cluster over 3 nearby sites"]
  Q2 -- "no" --> Q3{"Do users in several regions
need fast local writes?"}
  Q3 -- "no" --> AP["Active-passive with replication
and rehearsed failover"]
  Q3 -- "yes" --> Q4{"Can each entity have
one home region?"}
  Q4 -- "yes" --> AA1["Active-active with home regions
simplest and safest"]
  Q4 -- "no" --> AA2["Active-active with merge rules
highest effort, use with great care"]

Short answers to questions that come up in almost every design discussion.

The whole guide in a few sentences.