Before we talk about the problem, we need to understand why teams accept it in the first place.

Apache Kafka is a system that stores streams of events, such as orders, payments, sensor readings, or clicks. A group of Kafka servers working together is called a cluster. A single cluster is usually placed inside one data center or one cloud region. That is fast and simple, but it also means the cluster is only as safe as the place it lives in.

Cross-cluster replication means continuously copying events from one Kafka cluster to another, often in a different building, region, or even cloud provider. It exists because some needs cannot be met by one cluster alone. A region can lose power, a legal rule can demand that data stay in one country, or users on another continent can demand faster reads.

The Main Reasons Teams Replicate

Each of these reasons ends with the same practical question: when an application moves from the source cluster to the target cluster, where should it continue reading? Answering that question is called offset translation, and it is the heart of this guide.

An offset is just a number, but almost everything about reliable Kafka reading depends on it.

A Kafka topic is a named stream of events, such as a topic called payments. To allow many servers to share the work, a topic is split into partitions. A partition is an ordered, append-only log. New events are always added at the end and are never inserted in the middle.

Every event inside a partition receives a sequence number, starting at zero and growing by one. That number is the offset. The offset is unique only within one partition of one topic, so the full address of an event is the topic, the partition, and the offset together.

Consumers, Groups, and Committed Offsets

A consumer is an application that reads events. Consumers usually work in a consumer group, a team of consumer instances that share the partitions of a topic so that each partition is read by only one member at a time. This lets you scale reading by adding more members.

To avoid rereading everything after a restart, the group records how far it has read. This progress marker is the committed offset, and it means the next event to read. Kafka stores committed offsets in an internal topic named __consumer_offsets, keyed by group, topic, and partition.

That last term is important for our story. If a consumer group arrives at a new cluster and finds no committed offset, or an offset that makes no sense, it falls back to the reset rule. Starting at the beginning replays everything, and starting at the end skips everything that arrived before the consumer connected. Both are painful in production.

The word replication hides two separate mechanisms, and only one of them keeps offsets identical.

Inside a Cluster: Follower Replication

Inside one cluster, every partition has a leader and several followers. Followers fetch the leader’s log byte by byte, so their copy is identical, including offsets. The set of followers that are fully caught up is called the in-sync replicas. When a leader fails, a follower is promoted, and consumers continue with the same offsets because the log is the same.

Between Clusters: Mirroring

Between two independent clusters, there is no built-in follower relationship. The replication tool acts as an ordinary consumer of the source cluster and an ordinary producer to the target cluster. It reads an event, then writes it as a brand-new event. The target cluster does not know the event has any history, so it hands out its own next offset.

Because mirroring is a fresh write, three things can differ between the copies: the offset of each event, the presence of some events, and the layout of partitions. Offset translation exists to bridge the first difference, and it must survive the other two.

Offsets rarely match, and the reasons are ordinary, not exotic.

It is tempting to believe that if you start replication on day one and nothing breaks, the offsets will match forever. In practice, several forces act on every real system, and any one of them is enough to break the match. Understanding each cause helps you predict how large the gap will be for your workload.

flowchart LR
  A["Source partition: offsets 0 to 9"] --> B["Mirror tool: consume then produce"]
  B --> C["Target partition: offsets 0 to 6"]
  A -. "aborted transaction, markers, compacted keys" .-> D["Events never copied one to one"]
  D -.-> C
        

The result is a mapping between two number lines. Source offset 500 might correspond to target offset 430 today and target offset 470 an hour later, because the gap between the two lines changes over time as markers appear or old data is deleted. The mapping is not a fixed formula, which is why tools must keep records of it while replication runs.

The open-source Kafka mirroring tool solves translation by keeping a running diary of how the two clusters line up.

MirrorMaker 2 is the mirroring framework shipped with Apache Kafka. It runs on Kafka Connect, which is a framework for moving data in and out of Kafka using pluggable workers called connectors. MirrorMaker 2 is made of three cooperating connectors, each with a narrow job.

The Three Bookkeeping Topics

The tool keeps its notes in ordinary Kafka topics. The first is the offset-syncs topic, which stores pairs saying that source offset X was written as target offset Y. The second is the checkpoints topic, which stores, for each consumer group and partition, the last source position the group committed and the target position considered equivalent. The third is the heartbeats topic, which proves the pipeline is alive.

Remote topics also follow a naming convention. By default, a topic called orders on the cluster with alias primary appears on the target as primary.orders. The prefix records where the data came from and prevents infinite loops in two-way setups. Some deployments use an identity policy that keeps the original name instead, which is convenient for applications but removes that loop protection and requires extra care.

flowchart TD
  S["Source cluster topics and consumer groups"] --> M["Source connector"]
  M --> T["Remote topic on target cluster"]
  M --> O["Offset-syncs topic"]
  O --> C["Checkpoint connector"]
  S --> C
  C --> K["Checkpoints topic on target"]
  K --> F["Failover tools and group sync"]
        

The translation is an estimate built from sparse sample points, and that design choice explains most of its limits.

Writing a mapping record for every single event would double the write volume and add heavy load, so MirrorMaker 2 does not do that. It records a sample point only occasionally. A sample point is a fact such as: source offset 1,000 became target offset 940. Between two sample points, the tool knows nothing exact.

This produces an important property: translation is conservative and favors at-least-once delivery. You will usually see some duplicates after failover, but you should not see silent loss, provided the assumptions behind the sample points still hold. The rest of this guide is largely about the situations where that promise gets weaker.

A Worked Example With Real Numbers

Numbers make the idea stick, so let us walk through a small, invented example. A topic partition on the source cluster holds events from offset 0 upward. The mirroring tool started late, when the source already had 5,000 events. It began copying from the oldest event it could still read, and the target started numbering from zero. For simplicity, imagine the source had trimmed everything below offset 5,000, so the first mirrored event is source offset 5,000 and target offset 0.

Now suppose a consumer group has committed source offset 5,300. The checkpoint connector searches for the newest sample whose source offset does not exceed 5,300. That is Sample C, at source offset 5,260, which maps to target offset 256. The checkpoint therefore says: for this group, resume the target partition at about offset 256. The real equivalent of source offset 5,300 might be target offset 294, but the tool cannot know that exactly. The group will process roughly forty events twice, and nothing is skipped.

Notice how the difference between the two number lines changed across the samples: it was 5,000 at the start, then 5,002, then 5,004, then 5,008. A growing difference signals that the source contains offsets the target does not. If you copied the raw number 5,300 into the target, you would land far beyond the end of the partition, since the target might only have a few hundred events. Depending on the reset rule, the group would either fail or jump to the end.

The sparser the samples, the larger the replay after failover.

Suppose sample points exist at source offsets 1,000 and 1,100. A group has processed up to offset 1,099 when the disaster hits. Translation can only use the sample at 1,000, so the group restarts on the target at the equivalent of 1,000. It will process about a hundred events a second time. With a small threshold, the duplicate window is small, but the extra bookkeeping traffic grows.

The duplicate window has a second cause that is easy to forget: timing. Checkpoints are calculated on a timer. If the checkpoint was written a minute ago and the group has advanced since, the translated position reflects the old state. In a busy topic processing thousands of events per second, one minute of staleness can mean tens of thousands of extra replayed events.

The Tuning Trade-off

Every tuning knob in this area moves cost from one place to another. Shrinking the sample spacing gives finer translation but produces more bookkeeping records. Shortening the checkpoint interval keeps checkpoints fresh but makes the checkpoint connector read consumer group state more often, which adds load on the source cluster, especially when hundreds of groups are monitored. Frequent automatic group syncing adds write traffic to the target cluster’s internal offsets topic.

A sensible approach is to start with defaults, measure real replay in a drill, and tighten only when the measured window is larger than the business accepts. Teams with thousands of consumer groups often restrict monitoring to the groups that truly matter, using include and exclude patterns, so the checkpoint work stays affordable.

Translation can only point at data that actually reached the target.

Mirroring is asynchronous. The source acknowledges a write to its producer before the mirror has copied it. If the source cluster fails, the newest events that had not yet crossed the network are lost from the target’s point of view. This is a data-loss question, not a translation question, but it affects translation in a subtle way.

A consumer group on the source may have already read and processed events that never reached the target. After failover, the checkpoint for that group points to the last sample that exists on the target. The group will continue from there, but the events in the missing tail are gone. Downstream systems that already saw them, such as a database updated by the consumer, may now hold facts that the target cluster has never recorded.

What This Means for Recovery

The gap between what the source had and what the target received is measured as replication lag. It equals the recovery point, the amount of business data you can lose in a disaster. Offset translation cannot shrink this gap. Only faster or more parallel replication, better network paths, or synchronous designs can. Therefore, a good failover plan states two different numbers: how much data may be missing, and how much data may be replayed.

A partition’s offset line is not always a solid line. Sometimes it has gaps.

Kafka topics use one of two cleanup styles. Delete-style topics remove whole old segments after a time or size limit. Compacted topics keep only the latest event for each key and remove older ones in the background. In both cases, offsets are never reused, so the surviving events keep their old numbers and the sequence develops holes.

Compaction Makes Alignment Harder

On a compacted topic, the source and the target run their cleanup independently, at different moments, with different amounts of data to compact. A given key might already be reduced to its latest value on the source while the target still holds several older versions, because the mirror copied them before compaction ran. The two logs therefore contain different numbers of events for the same time period, and the distance between equivalent offsets keeps shifting.

Sample points remain valid for the moments they were recorded. The trouble appears when a group’s committed offset points into a region that has since been compacted away, or when the group’s position falls between two sample points across a large hole. The estimate is still safe in the backward direction, but the replay could be much larger than the number of live events suggests.

Retention Mismatch

If the target keeps data for a shorter time than the source, a group that was far behind on the source may translate to a position that the target has already deleted. The consumer then hits an out-of-range error and falls back to its reset rule, replaying everything or skipping to the end. Always make the target’s retention at least as long as the source’s for topics whose consumers might lag.

Exactly-once on one cluster does not automatically survive a trip to another.

A Kafka transaction lets a producer write to several partitions and make all writes visible together or not at all. When the producer commits or aborts, the cluster writes a small control record, called a marker, into each partition involved. Markers take up offsets, but consumers never see them as data. That is one reason source offsets can jump by more than the number of visible events.

What Changes During Mirroring

The mirroring tool reads events and writes them as ordinary events. It does not recreate the original transaction boundaries or markers on the target. Which records it reads also depends on how its consumer is configured. A consumer that only reads committed data skips events from aborted transactions entirely. A consumer that reads everything copies them as if they were normal, which could expose data that the source considered cancelled. Either choice moves target offsets away from source offsets.

The practical lesson is to treat cross-cluster failover as an at-least-once world even if your applications use transactions locally. Design consumers to tolerate replays and, when strict correctness matters, keep a durable record of processed identifiers outside Kafka. Also consider the isolation level of the mirror’s own consumer as a deliberate design decision, not a default you never looked at.

Even a correct translation is useless if it lands at the wrong moment.

Checkpoints only tell you where a group could resume. Something still has to apply that position to the group on the target cluster. There are two common ways. In the first, an operator or script reads the checkpoints during failover and sets the group’s position before the applications start. In the second, the mirroring tool can automatically write the translated positions into the target’s committed offsets on a schedule, if that feature is enabled. It is not enabled by default in many versions, so check your release.

The Active Group Rule

Automatic synchronization generally refuses to overwrite a group that is actively running on the target. This protects a live group from having its position yanked backward. The consequence is a trap: if some instance of your application already started on the target for testing, or was started too early during the incident, the automatic sync may stop updating that group. Later the group begins from whatever position it created for itself, not from the translated one.

Group Names and Topic Names

Translation is keyed by group name and topic name. If the target’s topic carries a source prefix, consumers must subscribe to the prefixed name after failover, and the translated positions apply to that name. If applications hard-code the original topic name, they will connect to an empty or different topic. Many teams solve this with subscription patterns that match both the local and the prefixed topics, so the same application works in both places.

Offsets belong to partitions, so anything that changes partitions changes the whole translation problem.

Offsets are per partition, and translation maps a source partition to a target partition with the same number. This assumes the target topic has the same partition count as the source. If the counts differ, one source partition would have to feed several target partitions, and there is no single target offset that corresponds to a source position.

Partitioners and Key Ordering

Events with the same key normally go to the same partition, which is how Kafka preserves order per key. The mirroring tool by default keeps events in their original partition numbers. If a team changes it to re-partition using keys on the target, the per-key order may still hold, but the partition-to-partition mapping that translation depends on is lost. Any design that repartitions during mirroring must give up the standard offset translation and rely on timestamps or business keys instead.

Failing over is only half the journey. Getting back is where most plans fall apart.

In an active-passive design, one cluster serves traffic and the other waits. After a failover, the old primary is behind, because it missed everything written during the outage. Returning to it is called failback. It requires replication in the opposite direction and translation from the new primary’s offsets back into the old primary’s offsets. Every difficulty we have discussed now applies a second time, in reverse.

In an active-both design, applications on both sides produce to their local clusters and each cluster mirrors to the other. Each mirrored topic appears with a source prefix on the far side, and consumers that need the global view read both the local topic and the prefixed remote topic. This layout avoids replication loops, since a prefixed topic is never mirrored back to where it came from.

sequenceDiagram
  participant App as Consumer application
  participant A as Cluster A primary
  participant B as Cluster B standby
  App->>A: Reads and commits offsets
  A->>B: Mirror events and alignment records
  Note over A: Region failure
  App->>B: Failover using translated offsets
  B->>B: New events written locally
  Note over A: Cluster A recovers
  B->>A: Reverse mirror from failover point
  App->>A: Failback using reverse translation
        

The Split Brain of Offsets

If both clusters accept writes for the same logical stream during a network split, they create two different histories. When the link returns, merging them is not an offset problem but a business problem. Events may conflict, and no offset translation can decide which is right. Active-active designs therefore work best when data ownership is clear, for example, each region owns its own customers, and the other region only reads.

Translation behavior has changed across releases, so the version you run matters.

Offset translation in MirrorMaker 2 has been improved several times since its introduction. Early versions could translate reliably only for consumer positions at or after the most recent sample point. A group that was lagging well behind the newest sample could fail to get a usable checkpoint. Later releases keep more history of sample points so lagging groups can be translated too. Other improvements added automated syncing of group positions, more flexible placement of the bookkeeping topics, and identity naming policies.

Because these details change, the safest approach is to read the release notes and configuration reference of the exact Kafka version you run, then test the behavior in a staging environment with a deliberately lagging consumer group. Never assume that a blog post, including this one, describes your version’s defaults.

If translated offsets are too coarse for you, there are other ways to choose a safe restart point.

Timestamp-based recovery deserves a special note. It relies on the timestamps stored inside each event. If the topic is configured to use the time the event was appended to the log instead of the time the producer created it, then the mirror’s write time replaces the original time, and the timestamps reflect the copy moment. Check the timestamp setting and ensure it is consistent on both clusters before relying on this method.

You cannot trust a failover plan that you cannot measure.

Translation quality is invisible until you need it, which is why monitoring must check it in advance. The goal is to answer three questions at any moment: is data flowing, how far behind is it, and are checkpoints fresh enough to be useful?

Logging That Helps During an Incident

Log every failover step with the group name, partition, source position, translated position, and the age of the checkpoint used. After the incident, these logs let you estimate how many events were replayed and confirm whether anything was missed. Also record who applied the positions and when, because manual steps are the most common place for human error.

Rehearsals as a Metric

Schedule regular failover drills and record how long each takes, how many events were replayed, and what surprised the team. The trend across drills is the best single indicator of whether your offset strategy is healthy. A plan that has never been rehearsed is only a hope.

Keeping the Bookkeeping Topics Healthy

The offset-syncs and checkpoints topics are easy to forget because they are internal. Yet they are the memory of the whole translation system. If the offset-syncs topic loses its recent data because of aggressive retention, the checkpoint connector may lose the samples it needs. If the checkpoints topic is deleted, every group loses its translated position until new checkpoints are produced, which can take a full interval or more.

Give these topics enough replicas, sensible retention, and proper alerts. Include them in backup and configuration management, and never let cleanup scripts delete them by accident. When the source cluster is the home of the offset-syncs topic, consider what happens if that cluster disappears entirely, because the samples you need at failover time would then be unreachable.

Replication tools hold powerful keys to two clusters at once.

A mirroring deployment connects to both clusters with credentials that can read many topics on one side and write many topics on the other. If those credentials leak, an attacker gains wide access. Treat the replication identity as a high-value account with the smallest set of permissions that still works, and give it separate credentials from ordinary applications.

The checkpoints topic deserves extra attention. It looks harmless, but a tampered checkpoint could push a group far backward, causing a large replay of sensitive events, or far forward, causing loss. Treat it as control data, monitor unexpected changes, and audit who applies translated positions.

Teams that survive failovers tend to share a handful of habits.

A short checklist can prevent most painful surprises.

The same pressures appear across very different industries.

Cross-cluster mirroring has been a core need almost since Kafka became popular. The original MirrorMaker tool was a simple consumer-producer pair with no offset translation. It worked for moving data but left consumers with no help at failover. Large companies built their own replicators to cope, and the open-source community later designed MirrorMaker 2 to add topic and configuration syncing, offset translation, and heartbeats. Commercial vendors also offer their own replication and linking products with different trade-offs.

A Typical Migration Walkthrough

Migration is the friendliest setting for offset translation, because nothing is on fire and you can pause at each step. Teams that do it well move one consumer group at a time and check the result before touching the next.

Each cluster assigns offsets when it stores an event. A mirroring tool writes new events, so the target chooses its own numbers. Only follower replication inside one cluster copies the log exactly.

Translation itself rounds backward to avoid skipping events. Data loss usually comes from asynchronous replication lag, where the newest events never reached the target, or from mismatched retention and reset rules.

Because translated positions come from sample points and timed checkpoints. The group restarts slightly earlier than its true position, so a window of events is processed again.

Not through mirroring alone. Kafka’s transactional guarantees apply inside one cluster. Across clusters, design for at-least-once and add idempotent processing or a store of handled identifiers.

Automatic sync generally avoids overwriting an active group. You may need to stop all members, apply positions, and restart. Confirm the exact behavior in your version.

Compaction creates holes in the offset line that differ between clusters. Translation stays safe but may replay more than expected, and event counts will not match between source and target.

It is simpler and independent of offsets, but less precise and dependent on timestamp quality. Many teams keep it as a fallback rather than the main method.

Standard translation no longer applies, because partitions cannot be mapped one to one. Use identical counts, or rely on timestamps and idempotent consumers.

Generally no. Checkpoints are recorded under the original group name, and a new name starts without any translated position.

Often enough that the runbook stays accurate and staff stay familiar with it, commonly a few times a year, and after any major change to topology, versions, or partitioning.

The tool only has sample points and chooses the newest one that does not pass the group’s committed position. Being slightly behind is the safe direction, because it can only cause replay, never skipped events.

It usually means the target lost or gained events relative to the source, for example after a topic recreation, a large aborted transaction, a mirror restart from another position, or a heavy compaction run. Investigate before trusting old checkpoints.

Not directly. Heartbeats prove that data flows end to end and let you measure latency. That measurement tells you how large the missing tail could be, which is an important companion to translation quality.

Yes. With prefixed remote topics, a consumer can subscribe to the local topic and the remote topic together using a pattern. Ordering across the two topics is not guaranteed, so downstream logic must not depend on it.

Idempotent consumers. If reprocessing an event is harmless, then imprecise translation, stale checkpoints, and even manual timestamp resets stop being emergencies and become routine operations.

Offsets are positions inside a single copy of a partition, so they cannot be treated as universal addresses. Mirroring between clusters creates new events and therefore new numbering, which then drifts further because of late starts, retention, compaction, transaction markers, retries, and configuration differences. Tools such as MirrorMaker 2 respond by keeping sparse sample points and computing checkpoints that map consumer positions from one cluster to the other. That mapping is deliberately conservative, favoring replay over loss.

The hard parts appear at the edges: stale checkpoints, active groups on the target, mismatched partitions, missing replication tails, and the return trip during failback. The most resilient organizations combine sensible translation with idempotent consumers, uniform topic contracts, strong monitoring, and rehearsed runbooks.