Cross-Cluster Replication in Elasticsearch
How Elasticsearch keeps live copies of your data in a second cluster, in another building, city or continent, so your search stays close to your users and survives the worst day.
Picture a company that runs its online shop from a data center in Frankfurt. Every product, order and customer search lives in one Elasticsearch cluster there. On an ordinary day, everything is fine. Then a power failure, a cut cable or a mistake by a contractor takes the whole building offline. The shop goes dark, customers cannot search, and every minute of silence costs money and trust.
Now picture a second company with customers in Tokyo, London and New York, all searching one cluster in Virginia. The Tokyo customer waits far longer for each search than the New York customer, simply because the data travels half the planet. Nothing is broken, yet the experience is unequal and slow.
Both problems have the same cure: keep a live copy of the data somewhere else. Elasticsearch has a built-in feature for exactly this, called Cross-Cluster Replication, or CCR. This guide explains CCR from zero. Every new word is explained the first time it appears, and each idea builds on the last. By the end you will know what CCR is, how it works inside, how to plan for it, what it costs, and which traps to avoid.
1What Is Cross-Cluster Replication?
We start with the words in the name, because each one carries part of the meaning.
Cluster, Replication and Cross
A cluster is a group of Elasticsearch servers, called nodes, that work together and share the job of storing and searching data. Replication means keeping copies of the same data in more than one place. Cross-cluster means the copies live in different clusters, not just on different servers of the same cluster.
Put together, Cross-Cluster Replication is an Elasticsearch feature that automatically copies indices from one cluster to another and keeps the copy up to date as the original changes. An index is the basic container for documents in Elasticsearch, a bit like a table in a database. So CCR copies whole indices, continuously, from a source cluster to a target cluster.
Think of a national newspaper. The head office in the capital writes and edits every story. Instead of making readers in every city wait for trucks, the paper sends each finished page electronically to printing presses in other cities. Local readers get the same paper, printed nearby, only a few moments after the head office finished it. If the head office loses power, the regional presses still hold the latest pages. CCR is the electronic link, and the regional presses are the follower clusters.
Leader and Follower
CCR uses two simple roles. The index that gets changed by applications is the leader index, and it lives in the leader cluster. The copy is the follower index, and it lives in the follower cluster. The follower watches the leader and copies every change. The follower never changes the leader.
Cross-Cluster Replication (CCR) is an Elasticsearch feature in which a follower index in one cluster continuously pulls every change from a leader index in another cluster, so that the follower stays a near-real-time, read-only copy of the leader.
Three Words to Remember
- Asynchronous: the leader does not wait for the follower. It saves the data locally, tells the application “done”, and the follower catches up a moment later.
- Pull-based: the follower asks the leader for new changes. The leader does not push anything out.
- Read-only copy: applications can search the follower, but they cannot write to it while it is following.
What CCR Is Not
CCR is not a backup. If someone deletes an index by mistake on the leader, that deletion is copied to the follower too, because the follower faithfully copies everything. CCR is also not a way to write to two clusters at once for the same index. And it is not the same as searching several clusters together, which is a different feature called cross-cluster search. We will compare all of these later.
2Why Copies Inside One Cluster Are Not Enough
Elasticsearch already keeps copies of your data. To see why we need CCR as well, we must understand what those copies protect against and what they cannot.
Shards and Replicas Inside a Cluster
An index is split into pieces called shards, and shards are spread across the nodes of a cluster. Each shard normally has one or more replica shards, which are extra copies placed on different nodes. If one node crashes, a replica on another node takes over, and no data is lost. This is Elasticsearch’s everyday safety net, and it works very well for single-machine failures.
Where the Safety Net Ends
All those replicas still live inside one cluster, usually in one data center or one cloud region. That means they share the same fate in several situations:
- Site failure: a fire, flood, power cut or network outage at the data center takes every node down together.
- Regional cloud outage: a problem in one cloud region can affect all zones or the services they rely on.
- Distance: users far from the cluster always pay the price of long network trips, no matter how many replicas exist.
- Rules about location: some laws and contracts require data to stay in a certain country or to be served from inside it.
- Growth of many small clusters: an organization with dozens of separate clusters at branches or factories has no built-in way to see all their data in one place.
Keeping three copies of an important document in three drawers of the same office protects you if one drawer jams. It does not help if the whole office burns down. To be truly safe, you also keep a copy in another building. Replicas are the extra drawers, and CCR is the second building.
Why Not Just Stretch One Cluster Across Regions?
A natural idea is to place some nodes of one big cluster in Frankfurt and others in Virginia. Elasticsearch is not designed for this. Nodes in one cluster talk to each other constantly and expect fast, steady links. Across continents the delay is high and the network can wobble. A cluster spread this way can become slow, unstable, or unable to agree on who is in charge when the link between regions breaks. Elastic’s guidance is to keep a cluster within one low-latency location and to use CCR to connect separate clusters.
A stretched cluster looks cheap and simple on a diagram, but it turns every network hiccup between regions into a cluster-wide problem. Two independent clusters joined by CCR fail separately, which is what you want in a disaster.
3Core Concepts You Need First
A few terms appear in every CCR discussion. Learn them here and the later chapters will feel easy.
Remote Cluster
A saved connection from one cluster to another, with a short nickname. The follower cluster keeps a remote cluster entry that points to the leader.
Leader Index
The original index that applications write to. Every change on it gets a number that tells the order of events.
Follower Index
The read-only copy in the follower cluster. It keeps asking the leader for changes and applying them.
Sequence Number
A counter that Elasticsearch attaches to every change on a shard. It lets a follower know exactly which changes it has and which it still needs.
Global Checkpoint
The highest sequence number that is safely stored on all in-sync copies of a shard. It marks how far a shard can be trusted.
Soft Deletes
A setting that makes a shard remember recent changes, including deletions, for a while, so followers can read the history they missed.
Auto-Follow Pattern
A rule that says “whenever a new leader index with a name like this appears, create a follower for it automatically.”
Replication Lag
The gap between the leader’s latest change and the follower’s latest applied change. Small lag means a fresh copy.
Two Clusters, One Direction
flowchart LR
APP["Applications writing data"] --> L["Leader cluster: leader index"]
L -. "changes pulled over the network" .-> F["Follower cluster: follower index"]
F --> READ["Applications searching nearby"]
The dotted arrow is the heart of CCR. It is drawn from the leader toward the follower, but remember who starts the conversation: the follower reaches out and pulls. That design makes the leader simpler, because it does not have to keep track of every place its data should be pushed to.
Why the Numbering Matters
Every change on a shard gets a sequence number, counted upward like the page numbers of a diary: 1, 2, 3, and so on. Because of this, a follower can say precisely, “I have everything up to number 4,200. Please send me 4,201 onward.” There is no guessing and no need to compare whole indices. This tiny idea is what makes continuous copying both accurate and efficient.
Imagine a friend who writes a numbered diary and lets you copy it. When you visit again, you do not reread the whole diary. You tell them the last entry number you copied, and they hand you only the newer pages. Sequence numbers do the same for shards.
4How Replication Styles Differ
Systems copy data in several different ways. Knowing the options explains why CCR was designed the way it was.
Synchronous or Asynchronous
In synchronous replication, the system does not tell the application “saved” until every copy, including the faraway one, confirms it. This gives strong safety, because no confirmed write can be lost. But every write must wait for the slowest, farthest copy. Across continents, that delay is painful, and if the far link breaks, writes may stall completely.
In asynchronous replication, the system confirms the write after saving locally, and copies to the far site afterward. Writes stay fast and keep working even when the far link is down. The price is a small window during which the far copy is slightly behind. CCR chooses this style.
Push or Pull
In a push design, the source sends changes to each destination. In a pull design, each destination asks the source. CCR is pull-based. The follower decides how fast to read, how much to read at once and when to retry after a failure, which naturally protects the leader from being overwhelmed.
| Style | How It Works | Strength | Weakness |
|---|---|---|---|
| Synchronous | Write waits until all copies confirm | No confirmed write is lost | Slow over long distances and fragile when links fail |
| Asynchronous | Write confirms locally, copies follow later | Fast writes and tolerant of link problems | The copy can lag, so a sudden disaster may lose the last few moments |
| Push | Source sends changes to destinations | Very low delay for simple setups | Source must track and serve every destination |
| Pull | Destination asks the source for changes | Destination controls pace and retries | Delay depends on how often and how much it pulls |
Synchronous replication is like a bank clerk who refuses to hand you a receipt until the head office in another country has phoned back to confirm. Asynchronous replication is like the clerk handing you the receipt right away and sending the paperwork by courier for the head office to file. The second way is much faster, but if the courier’s truck crashes, the head office may briefly lack the newest paperwork.
Because CCR is asynchronous, the most useful number to plan around is the recovery point objective, often shortened to RPO. It answers the question “how much recent data can we afford to lose if the leader vanishes suddenly?” With CCR, that answer is usually small, often seconds, but it is never exactly zero.
5Architecture and Components
Let us open the box and look at the parts that make CCR work, from the user’s browser all the way to the shards.
The Parts of the System
API Gateway and Load Balancer
The gateway is the single entrance for requests, handling identity checks and rate limits. The load balancer spreads requests over several application servers. During a failover, this layer is often where traffic is redirected to the other cluster.
Search and Indexing Services
Your own code. Indexing services write to the leader cluster. Search services read from the leader or, for nearby reads, from the follower.
Leader Cluster and Shards
The cluster where data is created. Each leader shard keeps a numbered history of recent changes that followers can read.
Follower Cluster and Shards
The cluster that holds the copy. Each follower shard runs a small background task that reads changes from the matching leader shard and applies them.
Remote Cluster Connection
A secured network link, with a saved nickname, that lets the follower cluster talk to the leader cluster. Encryption and credentials protect it.
Monitoring and Management
Stats interfaces and dashboards that show lag, errors and follower health, plus Kibana screens for managing followers and auto-follow patterns.
flowchart TD
U["Users on web and mobile"] --> G["API Gateway"]
G --> LB["Load Balancer"]
LB --> IS["Indexing Service"]
LB --> SS["Search Service"]
IS --> LC["Leader Cluster: coordinating node"]
LC --> LP["Leader primary shards"]
LP --> LR["Leader replica shards"]
RC["Remote cluster connection with TLS and credentials"]
FT["Follower shard tasks"] --> RC
RC --> LP
FT --> FP["Follower primary shards"]
FP --> FR["Follower replica shards"]
SS --> LC
SS -. "nearby reads" .-> FC["Follower Cluster: coordinating node"]
FC --> FP
MON["Monitoring: lag, errors, stats"] --> FT
Reading the Diagram
Follow the top of the picture first. Users reach the gateway and load balancer, which hand requests to the indexing service and the search service. The indexing service sends writes to the leader cluster. There, a coordinating node routes each write to the right primary shard, which copies it to its replicas inside the same cluster.
Now look at the follower side. For each follower shard there is a background shard follow task. This task reads changes from the leader’s primary shard through the secured remote cluster connection. It then writes those changes into its own follower primary shard, which copies them to follower replicas. The search service can send reads to the follower cluster when the follower is closer to the user. Monitoring watches the follow tasks, since they carry the health of the entire copy.
The follower index must have the same number of primary shards as the leader index, because each follower shard follows exactly one leader shard. The number of replica shards on the follower, however, is your own choice and can differ from the leader. Plan the shard count carefully at the start, since it cannot be changed later without rebuilding the follower.
6Internal Working: Life of One Write
Follow one document from the moment an application saves it to the moment it is searchable in the follower cluster.
sequenceDiagram
participant App as Application
participant LP as Leader primary shard
participant LR as Leader replica shard
participant FT as Follower shard task
participant FP as Follower primary shard
App->>LP: Index a document
LP->>LP: Assign sequence number and store change
LP->>LR: Copy change to replica
LR-->>LP: Confirm
LP-->>App: Write acknowledged
FT->>LP: Ask for changes after my last checkpoint
LP-->>FT: Send batch of changes
FT->>FP: Apply changes with same sequence numbers
FP-->>FT: Confirm and advance checkpoint
The write arrives on the leader
An application sends a document to the leader cluster. The coordinating node routes it to the correct primary shard, based on the document’s identifier.
The primary numbers and stores the change
The primary shard gives the operation the next sequence number, writes it into its transaction log for safety, and applies it to the search data.
Replicas inside the leader cluster copy it
The primary forwards the operation to its replica shards. Once the in-sync copies have it, the global checkpoint can move forward and the application receives its confirmation.
The follower task asks for news
The follower’s shard task knows the last sequence number it has applied. It sends a read request to the leader shard asking for everything newer. If there is nothing new yet, the leader holds the request open for a while and answers as soon as something arrives, instead of making the follower ask again and again.
The leader returns a batch of operations
The leader shard reads the requested operations from its retained history and sends them in a batch. Only operations up to its global checkpoint are handed out, so the follower never receives changes that could still be rolled back.
The follower applies them in place
The follower’s primary shard applies the operations, keeping the same sequence numbers the leader gave them. It then copies them to its own replicas inside the follower cluster.
The change becomes searchable
After the next refresh on the follower, the document can be found by searches. Then the follower’s checkpoint moves forward, and the loop starts again from step 4.
Why the Leader Must Remember History
The leader can only hand out changes that it still remembers. Elasticsearch keeps this history using a feature called soft deletes. Normally, when a document is deleted or updated, the old version simply disappears at some point. With soft deletes, the shard keeps a lightweight record of recent operations, including deletions, so that followers can read them later.
To avoid keeping history forever, Elasticsearch uses retention leases. Each follower holds a lease on the leader shard that says “please keep everything after my checkpoint.” The follower renews the lease as it reads. If a follower disappears for too long, its lease expires, by default after about twelve hours, and the leader is free to discard old history.
A shop keeps its old receipts for a limited time. A regional auditor can ask to see the receipts since their last visit, as long as they come back within the retention period. If the auditor is away too long, the shop may have thrown the oldest receipts away, and the auditor needs a full inspection instead of a quick update. A retention lease is the auditor’s signed note saying “please keep my receipts.”
If a follower is paused or cut off for longer than the retention period, it may no longer be able to catch up from history. In that case it has to be rebuilt from scratch. Long outages of the link or long pauses are therefore a planning matter, not just a nuisance.
7Bootstrapping: Making the First Copy
A new follower cannot begin by reading a live stream of changes, because it is missing everything that happened before. It first needs a full starting copy.
Two Phases: Copy, Then Follow
When you create a follower index for an existing leader index, two phases happen. In the first phase, called remote recovery, the follower copies the leader’s data files over the network, much as a replica shard would recover from its primary. This gives the follower a snapshot of the leader as it looked at one moment.
In the second phase, the follower switches to following. It asks for every operation that happened after the moment of the snapshot, applies them, and from then on keeps reading new operations continuously. Sequence numbers make the hand-off exact, so nothing is missed and nothing is applied twice.
Suppose you want to keep a mirror copy of a shared notebook. First you photocopy all the existing pages, which takes a while. During the copying, the owner adds new pages. After finishing, you ask for every page numbered higher than the last one you copied. From then on you check for new pages regularly. Bootstrapping is the photocopying, and following is the regular checking.
The Cost of the First Copy
For a large index, the first phase can move hundreds of gigabytes or even terabytes across the network. This takes time and uses bandwidth on both sides. Elasticsearch limits the speed of this recovery by default, so it does not swamp the leader’s network or slow its normal work. You can tune that limit, but remember that a faster copy uses more of the leader’s resources at the same moment it is serving real users.
Planning the Start
Plan bootstrapping like a move to a new house. Schedule it for a quiet time, check that the network link has enough room, and make sure the follower cluster has enough disk space for the full copy plus its own replicas. Start with the most important indices first, and add others in waves rather than all at once. Watch the recovery progress and the leader’s load while it runs.
For a first rollout, choose one small, non-critical index and follow it end to end. Practice creating it, watching its lag, pausing it and promoting it. The skills you build on a small index carry directly to the large ones.
8Setting Up CCR Step by Step
Setting up CCR follows a clear order. Each step depends on the one before it.
Check the license and versions
CCR is a feature of Elastic’s paid subscription levels, and both clusters must meet its license and version requirements. Confirm this first in Elastic’s current documentation, because it decides whether the project can start at all.
Open the network path
The follower cluster must be able to reach the leader cluster over the network on the port used for cluster-to-cluster communication. Work with your network team on firewall rules, private links or peering between the two locations.
Prepare security
Encrypt the connection with TLS and create credentials for the follower to use. Give those credentials only the permissions needed to read from the leader index, following the least privilege rule.
Register the remote cluster
On the follower cluster, save a remote cluster connection with a short nickname and the address of the leader cluster. Elasticsearch checks the connection and reports whether it is healthy.
Confirm the leader index is ready
The leader index must keep its operation history, which means soft deletes must be on. In current versions this is the default for new indices. Older indices that were created without it need to be rebuilt first.
Create the follower index
On the follower cluster, create a follower index by naming the remote cluster and the leader index. Elasticsearch performs the bootstrapping copy and then starts following.
Verify and monitor
Check that the document counts match, that the lag is small and that the follow tasks show no errors. Add these checks to your monitoring before you rely on the follower for anything important.
Two Ways to Create Followers
You can create followers by hand, one index at a time, which suits a few long-lived indices such as a product catalog. Or you can define an auto-follow pattern, which suits indices that appear over time, such as one new log index each day. We look at auto-follow in the next chapter.
Using Kibana or the API
Elasticsearch exposes all of these steps through its programming interface, so they can be automated with scripts and infrastructure tools. Kibana also has a management screen for cross-cluster replication, where you can register remote clusters, create followers and auto-follow patterns, and see their status. Small teams often start with the screen to learn the concepts, and move to automation once the design is stable.
Follower index names must be different from any existing index in the follower cluster, and the shard count must match the leader. Teams often trip on naming clashes when they replicate the same index names from several leader clusters into one follower cluster. Add a prefix or suffix that identifies the source.
9Auto-Follow Patterns and Data Streams
Many real systems create new indices all the time. Creating each follower by hand would be tiring and easy to forget, so Elasticsearch can do it for you.
The Problem With Rolling Indices
Log and metric systems usually write to a new index every day or whenever the current one gets big. A team might have indices named like “logs-web-2026.09.21”, then “logs-web-2026.09.22”, and so on. If you had to create a follower for each new index by hand, someone would eventually forget, and the follower cluster would silently miss a day of data. That gap would only be discovered during a disaster, the worst possible moment.
What an Auto-Follow Pattern Does
An auto-follow pattern is a standing instruction stored on the follower cluster. It names a remote cluster and one or more name patterns with wildcards, such as “logs-web-*”. Elasticsearch checks the leader cluster regularly. Whenever it finds a new index whose name matches, it creates a follower index for it, following all the usual steps automatically. You can also decide how follower indices should be named, for example by adding a prefix that shows where they came from.
Think of a magazine subscription. You could walk to the shop every month and buy the newest issue by hand, and one busy month you would forget. Or you can subscribe once, and every new issue is delivered automatically. An auto-follow pattern is a subscription for new indices.
Name Patterns
Wildcards choose which leader indices are followed. Precise patterns avoid copying test or scratch indices that nobody needs.
Follower Names
A prefix or suffix can be added to each follower name. This keeps indices from different leader clusters apart when they land in one follower cluster.
Pause and Resume
A pattern can be paused during maintenance and resumed later. While paused, new leader indices are not followed.
Auto-Follow Stats
Elasticsearch reports how many indices were followed successfully, how many failed and the most recent errors, which helps you notice trouble early.
Data Streams
A data stream is a newer Elasticsearch structure for time-based data such as logs and metrics. Behind one name, it manages a series of hidden backing indices and creates new ones as the old ones fill up. CCR can follow data streams through auto-follow patterns, which is the recommended way to replicate time-series data. New backing indices are picked up automatically, and the follower cluster ends up with a matching data stream that readers can query by its name.
Write your auto-follow patterns with care. A pattern that is too broad may copy huge, unimportant indices and fill the follower’s disks. A pattern that is too narrow may skip something important. Review the list of followed indices after any new naming rule is introduced on the leader side.
Auto-follow only acts when it can reach the leader. If the link is down for a while, new leader indices may appear during the outage. Check after any outage that all expected followers exist, and use the auto-follow stats to see whether any attempts failed.
10Consistency and Ordering: What You Can Promise
Because CCR is asynchronous, the follower is always slightly behind. This chapter explains exactly what is guaranteed and what is not.
Eventually Consistent
A system is eventually consistent when copies may differ for a short time, but all copies will become the same once the changes have arrived. That is the promise of CCR. If writes to the leader stopped right now, the follower would soon hold exactly the same data.
In practice this means that a document indexed on the leader a second ago might not yet be visible in the follower. During that moment, a search on the follower gives an answer that was true a short while earlier. Most search use cases accept this happily, since a product added two seconds ago missing from a far-away search is rarely a problem.
Order Is Preserved Per Shard
Each follower shard applies operations from its leader shard in sequence-number order, so the story of each individual document is told in the right order. If a document was created, updated twice and then deleted, the follower sees those events in that order and ends in the same state as the leader.
Across different shards, however, the follower does not keep a global order. Shard 1 may be a little further ahead than shard 2. A search that touches both could briefly see a mix in which shard 1 has a change that shard 2 does not yet have. For related documents that live on different shards, this can produce a momentary inconsistency.
Imagine five clerks each copying a numbered ledger by hand from head office. Each clerk copies the pages of their own ledger in order, so each ledger is always correct up to some page. But one clerk may be on page 90 while another is only on page 85. At any given second, the five ledgers together do not show the same moment in time, although each one is internally tidy.
What You Cannot Assume
- No read-your-own-writes on the follower. An application that writes to the leader and immediately reads the same data from the follower may not find it yet. Read from the leader when freshness matters.
- No zero data loss on sudden failure. Operations that reached the leader but had not yet been pulled by the follower are lost if the leader is destroyed at that instant.
- No cross-shard snapshot of a single moment. Queries that combine data from several shards see each shard at its own point.
Decide early which reads can tolerate slightly old data and which cannot. Browsing a product list can be served by a follower. Checking whether a limited item is still in stock during checkout should go to the leader.
Deletes and Updates
CCR copies everything that happens to a document, including updates and deletions. This is good for fidelity and bad for protection against mistakes. If an engineer accidentally deletes documents on the leader, the follower will mirror the deletion within moments. Keep this in mind whenever someone suggests that CCR “is our backup”.
11Use Case One: Disaster Recovery and Failover
The most common reason to use CCR is to survive the loss of an entire site. This chapter shows how a follower becomes the new primary system.
Two Numbers That Matter
Disaster planning uses two standard measures. The recovery point objective (RPO) is how much recent data you can afford to lose, measured in time. The recovery time objective (RTO) is how long you can afford to be without service. A business might say, “We can lose at most a minute of data and be down for at most thirty minutes.” CCR mainly helps with the RPO, by keeping the copy fresh, and helps with the RTO, by having the copy already loaded and ready.
Every large hospital has a backup generator. The RPO is how much of the day’s records could be lost when the power dies, and the RTO is how many seconds pass before the lights come back. A generator that is always warm and ready gives a short RTO. A backup that must first be delivered and installed gives a long one. A follower cluster is a warm generator for your search.
The Failover Story
flowchart TD
A["Normal: writes go to Leader cluster in Region A"] --> B["Region A fails"]
B --> C["Decide to fail over"]
C --> D["Pause following on Region B follower indices"]
D --> E["Close each follower index"]
E --> F["Unfollow to turn it into a normal index"]
F --> G["Reopen the index in Region B"]
G --> H["Redirect applications to Region B"]
H --> I["Region B now accepts writes"]
Detect and decide
Monitoring shows the leader region is unreachable, and the people in charge decide the outage is serious enough to fail over. This decision is usually made by a person or a carefully tested automation, because switching sites by mistake causes its own trouble.
Stop following
On the follower cluster, each follower index is paused. This stops the background tasks that keep trying to reach the failed leader.
Promote the follower
The follower index is closed, then converted into a regular index by an “unfollow” step, then reopened. It now has all the data it had received, and it accepts writes like any other index.
Redirect the traffic
Applications are pointed to the second cluster, often by changing a DNS name, a load balancer target or a setting in the gateway. From here on, Region B is the primary.
Rebuild for the return
When Region A is healthy again, its copy of the data is out of date and possibly different from Region B, since some final writes may never have been copied. The clean way back is to delete the old indices in Region A and create new followers there that follow Region B. After they catch up, you can fail back or simply keep Region B as primary.
Do not let applications write to both clusters after a partial failover. If some writers still reach the old leader while others reach the promoted follower, the two copies drift apart, and merging them later is painful. Make redirection complete and quick, and stop the old writers if you can.
Practice Makes the RTO
A disaster plan that has never been rehearsed is only a hope. Schedule a game day at least once or twice a year, in which you promote the followers, redirect a test application and measure how long each step takes. Write the steps in a short runbook, a plain checklist that someone half-awake at three in the morning can follow. Time the rehearsal, and use the real time as your RTO instead of a guess.
12Use Case Two: Data Locality and Faster Reads
A second, nearby copy can make searches faster for users far from the main site, and can help meet rules about where data may live.
Distance Costs Time
Network signals travel fast, but not instantly, and long routes across oceans pass through many devices. A search that takes 20 milliseconds inside a data center can take several times longer when the user is on another continent, before we even count the time for the response to come back. Multiply that by the several requests a single web page might make, and the delay becomes something users feel.
By placing a follower cluster in each major region, searches from that region are answered close to home. Writes still go to the leader, and reads are served locally from the follower. Because most search applications read far more than they write, this sharply improves the everyday experience.
A famous library has one original reference book. Instead of making people from every city travel to read it, the library sends photocopies to branch libraries in each city. The photocopies may be a few days behind the newest edition, but people can read them down the street. Followers are the branch copies.
Example: A Travel Booking Site
A booking site keeps its hotel catalog in a leader cluster in one region. Follower clusters in Europe, Asia and North America hold copies of the catalog index. Each region’s search service reads from its nearest follower, so a traveler in Singapore gets results quickly. Price changes and room availability that must be exact are checked against the leader at checkout, while browsing uses the local copy.
Rules About Location
Some regions require certain data to be stored or served inside their borders. A follower cluster in that region can hold a compliant local copy for serving users there. Whether a copy is allowed at all depends on the rules for your data and your industry. Speak with your legal and compliance teams before replicating personal or regulated data across borders, because the same feature that solves a speed problem can create a legal one.
Replicating data to another country is a decision about law and privacy, not only about performance. Never enable it for personal data until the right people have approved where that data may be stored and who may access it.
Use the Right Node Sizes
Follower clusters serving reads need enough memory and processing power for search traffic, but they do not need to handle heavy indexing from applications, since the only writes they perform are the replicated operations. This can allow slightly different hardware choices from the leader, tuned to reading. Many teams also choose fewer replicas on the follower in regions with lower traffic, to save cost.
13Use Case Three: Centralized Reporting
Sometimes the problem is the opposite of scattering data to many places. You have many small clusters and want everything in one.
Hub and Spoke
Imagine a retail chain with hundreds of stores, each running a small Elasticsearch cluster for local point-of-sale search and analysis. Head office wants to see sales across all stores without visiting each cluster one by one. With CCR, every store cluster acts as a leader for its own indices, and one large central cluster follows all of them. The central cluster becomes a complete, up-to-date reporting hub.
flowchart LR
S1["Store cluster 1: leader"] --> H["Central reporting cluster: many follower indices"]
S2["Store cluster 2: leader"] --> H
S3["Store cluster 3: leader"] --> H
S4["Store cluster N: leader"] --> H
H --> D["Dashboards and analysts"]
A single follower cluster can follow indices from many leader clusters at once. Each follower index stands for one leader index, so you will have many follower indices in the hub, one per source index. Naming becomes important here. If every store names its sales index the same way, the hub needs a naming rule such as adding the store code as a prefix, so that names do not collide.
Think of a national weather agency. Every local station records its own readings. Instead of asking each station by phone whenever someone wants the national picture, all stations send their data to a central office, where one team can chart the whole country. The stations stay small and simple, and the central office does the heavy analysis.
Why This Beats Visiting Each Cluster
Elasticsearch can also search many clusters at once through a different feature called cross-cluster search. That works well for occasional questions. For heavy, frequent reporting, having the data already in one place is faster, because the queries run locally in the hub without waiting for hundreds of remote clusters to answer. It also keeps reports working even if a store’s cluster is temporarily offline, since the hub already holds the latest copy.
Capacity for the Hub
The hub cluster must be able to store every followed index. Add up all the leader indices’ sizes, then the hub’s own replica shards, then room to grow. Many teams underestimate this and discover disk pressure after a few months. Combine storage planning with a retention plan, so that old data leaves the hub on a schedule.
When following many sources, stagger the first copies. Bootstrapping fifty leaders at once creates a huge wave of network and disk activity on the hub. Start with a few, confirm that everything is healthy, and then add more in batches.
14Active-Active and Bi-Directional Patterns
So far, data flows one way for each index. Some organizations want both sites to accept writes. CCR can support this, but only with a careful design.
One Index Cannot Have Two Leaders
A follower index is read-only, and an index can only follow one leader. That means you cannot have two clusters both writing to the same index and copying to each other. If both sides could change the same document at the same time, there would be no rule to decide whose version wins, and the two copies would disagree.
The Prefix Pattern
The safe design gives each cluster its own writable index, and each cluster follows the other’s index. For example, the European cluster writes to an index whose name has a “eu” prefix. The American cluster writes to an index with a “us” prefix. Each cluster also holds a follower of the other’s index. A search that needs everything reads across both the local writable index and the follower index, usually through a shared alias or a wildcard.
Two regional offices share a filing cabinet system. Each office writes only in its own colored folders, and copies the other office’s folders into a read-only shelf. Anyone in either office can read all the folders, and nobody ever edits the same page as someone else. Conflicts never happen, because each page has exactly one author.
| Cluster | Writes To | Follows | Reads From |
|---|---|---|---|
| Europe | Its own index for European data | The American index | Its own index and the follower of the American index |
| Americas | Its own index for American data | The European index | Its own index and the follower of the European index |
Where This Works Well
This pattern suits data with a natural owner, such as logs generated by each region, orders placed in each region, or sensor data from local sites. Each record is created in exactly one place, so nothing needs merging. It is less suited to shared records that anyone might edit, such as a single global customer profile, because those need a different design.
An active-active design multiplies the things that can go wrong: two writers, two followers, two sets of aliases and two failure modes. Use it only when you truly need writes in both regions. Many teams find that a simple active-passive setup meets their needs with far less complexity.
15CCR and Index Lifecycle Management
Data does not live forever. Elasticsearch has a feature that moves indices through stages as they age, and it interacts with CCR in useful ways.
A Quick Look at Index Lifecycle Management
Index Lifecycle Management, or ILM, is a feature that automatically moves indices through stages: hot for fresh, busy data, then warm, cold and finally deletion as data ages. It can also start a new index when the current one gets too large, an action called rollover. ILM saves money and effort by keeping recent data on fast storage and pushing old data to cheaper storage.
Leader and Follower Have Their Own Lives
The leader and the follower each live in their own cluster, and each can have its own lifecycle policy. The leader’s policy might roll over daily and delete after thirty days. The follower’s policy could keep data longer, for example ninety days, if the follower cluster is used for reporting and has cheap storage available. Retention on the follower is your own choice.
Rollover is the special case. Only the leader can roll over, because a follower is read-only. The follower gets its new indices by following the leader’s new indices, either through an auto-follow pattern or a data stream. ILM has a specific step designed for follower indices, called unfollow, which detaches a follower from its leader once the leader has finished with it. This turns an old, finished index into an ordinary index that can be moved to colder storage or deleted in the normal way.
Consider a newspaper archive. The head office decides when to close an issue and start a new one. Branch offices receive each closed issue and then decide for themselves how long to keep it on the shelf before moving it to the basement or throwing it out. The head office does not tell them what to do with their copies.
Make sure the leader does not delete an index before the follower has finished copying it. If your leader retention is very short and the link is slow or down, data could vanish from the leader before the follower ever received it. Keep leader retention comfortably longer than your worst expected outage.
16CCR Versus Other Ways to Move or Protect Data
Elasticsearch offers several ways to reach across clusters or to protect data. Each solves a different problem, and confusing them is a common mistake.
| Approach | What It Does | Best For | Main Limit |
|---|---|---|---|
| Cross-Cluster Replication | Continuously copies indices to another cluster | Disaster recovery, nearby reads, central reporting | Follower lags slightly, is read-only and copies mistakes too |
| Cross-Cluster Search | Runs one search across several clusters and merges the results, without copying data | Occasional searches over data that should stay where it is | Each search depends on remote clusters being reachable and fast |
| Snapshots | Saves a point-in-time backup to storage such as an object store | Protection against deletion, corruption and long-term archiving | Restoring takes time, and data since the last snapshot is missing |
| Stretched cluster | One cluster with nodes in several locations | Only very close, low-latency sites such as nearby zones | Unsafe across long distances because of delay and split-brain risk |
| Dual writing | The application or a pipeline sends every write to two clusters | Custom needs, such as different transformations per cluster | You must handle failures and mismatches yourself |
Cross-Cluster Search in One Paragraph
Cross-cluster search leaves data where it is and travels to it. A search request goes to a local cluster, which forwards it to remote clusters, gathers their answers, and merges them into one result list. It is like phoning several offices to ask the same question. CCR, by contrast, copies the data so that the answer is already at hand. The two are often used together: CCR brings key indices close, and cross-cluster search reaches the rest.
Snapshots and CCR Are Partners
Snapshots and CCR protect against different dangers. CCR protects against the loss of a place, and it does so with small data loss and fast recovery. Snapshots protect against the loss of correctness, such as accidental deletion or corrupted data, because a snapshot can be taken before the mistake and restored afterward. A serious system uses both.
CCR is like a live video feed to a second control room. If the first room floods, the second is already watching everything. A snapshot is like a photo album taken every night. If someone scribbles over the live records, the video feed shows the scribbles too, but the photo album still holds a clean earlier page.
Ask three questions. Do I need a live copy elsewhere? Use CCR. Do I only need to search data that lives elsewhere, occasionally? Use cross-cluster search. Do I need to undo mistakes or keep long-term history? Use snapshots. Most production systems answer yes to more than one.
17Performance and Scalability
CCR adds work to both clusters and uses the network between them. Knowing where that work comes from lets you keep it under control.
Where the Work Happens
On the leader side, the extra work is answering read requests from followers and keeping the history of recent operations. On the follower side, the work is applying those operations as if they were normal writes, then copying them to follower replicas. Between them, the network carries every change. If your leader handles a heavy stream of writes, the follower must be able to absorb the same stream.
Network Bandwidth
The link must carry the leader’s write volume, plus the initial copy. Undersized links are the number one cause of growing lag.
Read Batch Size
Followers can pull more operations per request. Bigger batches reduce round trips, but use more memory and can slow individual requests.
Parallel Reads
A follower shard can keep several read requests in flight at once. More parallelism helps on high-latency links, up to a point.
Follower Write Capacity
The follower must index operations as fast as they arrive. Fast disks and enough processing power keep it from falling behind.
Shard Layout
Each leader shard is followed by one follower shard, so a hot leader shard can become a bottleneck. Balanced shards spread the load.
Recovery Throttle
The speed limit on the initial bulk copy protects the leader. Raise it only when the leader has spare capacity.
Why Latency Matters as Much as Bandwidth
Bandwidth is how much data the link can carry at once. Latency is how long a round trip takes. A follower that reads one small batch, waits for the answer and then asks again will be limited by latency, even on a very fast link. This is why followers can keep several requests in flight and read bigger batches. Think of moving water with buckets between two villages: a wide road helps, but so does sending many carriers at the same time so nobody stands waiting.
Scaling Out
CCR scales with the number of shards. More shards mean more follow tasks working in parallel, each pulling from its own leader shard. A very busy index with many shards can therefore replicate faster than a single-shard index that funnels all changes through one task. At the same time, too many tiny shards waste memory and add overhead, so keep shard sizes in a healthy range.
Traffic spikes on the leader show up as lag on the follower a moment later. Test CCR under your busiest expected write load, not under average load. A follower that keeps up at noon on a quiet Tuesday may fall behind on the biggest sale day of the year.
Example: Fixing Growing Lag
A team notices that lag on a busy log index grows during evening peaks and only shrinks overnight. Investigation shows that the index has a single hot shard, which limits parallel copying, and that the link between regions is close to full. They split the index into more shards for new daily indices and upgrade the link. Within a week, lag stays under a few seconds even at peak.
18Data Flow and the Life of a Follower Index
A follower index moves through clear stages from birth to retirement or promotion. Knowing these stages makes operations far less mysterious.
flowchart LR
C["Created"] --> R["Remote recovery: bulk copy"]
R --> F["Following: reading new operations"]
F --> P["Paused"]
P --> F
F --> U["Unfollowed: becomes a normal index"]
P --> U
F --> X["Lagging or failing: needs attention"]
X --> F
X --> RB["Rebuilt from scratch"]
RB --> C
Created
The follower index is registered and linked to a leader index. No data has been copied yet.
Remote Recovery
The initial bulk copy of the leader’s data files. Searches on the follower may show incomplete data until this finishes.
Following
The normal state. Follow tasks read operations continuously, and the follower stays close to the leader.
Paused
Following is stopped on purpose, for maintenance or a failover step. The lease on the leader keeps history for a limited time only.
Unfollowed
The link to the leader is cut. The index becomes ordinary, writable and independent. This is what promotion means.
Failing or Lagging
An error such as a broken link or missing history has stopped progress. Someone needs to look at the reason and repair it.
The Data That Travels
The follower receives more than documents. When the leader’s mappings change, meaning the rules that describe the fields in the index, the follower picks up those changes as well, so both sides keep matching structures. Many index settings are also copied, so that the follower behaves like the leader. Settings that describe the follower’s own physical situation, such as the number of replica shards or where shards are placed, stay under the follower cluster’s control.
Consider an apprentice who copies a master baker’s recipe book. The apprentice copies the recipes and any new notes, so the books match. But the apprentice’s own kitchen has its own oven, shelves and staff, so the apprentice decides where ingredients are stored. The recipes follow the master. The kitchen setup is local.
Some index settings can only be changed while an index is closed. If a setting on the leader requires a closed index, the follower may need to be paused and closed briefly to accept it. Read the release notes for your version before planning changes to settings on followed indices.
19A Worked Example, Start to Finish
Concepts are easier to remember with a story. Here is a small made-up example that shows planning, lag and failover in numbers.
The Setting
An online marketplace runs its main Elasticsearch cluster in one region, which we will call Region A. It holds a product index of 400 gigabytes, and it receives about 800 changes per second on average, with peaks near 3,000 per second during sales. The business asks for a disaster recovery copy in Region B, with at most one minute of data loss and at most thirty minutes without service. All figures here are invented to illustrate the reasoning and are not from a real company.
Planning the Numbers
| Question | Plan | Reason |
|---|---|---|
| How big is the follower cluster? | Disk for 400 GB, plus one replica copy, plus growth room | The follower holds a full copy and its own replicas, so storage needs are roughly double the leader’s primary data |
| How long is the first copy? | Plan for several hours on a link that can spare about 200 megabits per second | 400 GB is about 3,200 gigabits, and even at full speed the copy needs time, so it is throttled to spare the leader |
| What lag do we expect? | A few seconds normally, and up to about 20 seconds at peak | The follower must absorb 3,000 operations per second during sales, so it is sized for the peak, not the average |
| Does it meet the one-minute RPO? | Yes, if peak lag stays under 60 seconds, with an alert at 30 seconds | If the leader vanished at the worst moment, we would lose up to the lag, which is inside the goal |
| Does it meet the 30-minute RTO? | Yes, if the runbook is rehearsed and takes under 15 minutes | Promotion involves pausing, closing, unfollowing, reopening and redirecting traffic, which is quick for a few indices |
A Day in the Life
At first, the team creates one follower index for the product catalog and watches it for a week. Lag stays around two seconds. During a big sale, lag rises to 18 seconds for a few minutes, then falls back. The team sets an alert at 30 seconds and moves on.
Three months later, a power failure takes Region A offline at 14:05. Monitoring shows the leader unreachable at 14:06. At 14:10 the incident lead decides to fail over. The team follows the runbook: pause the follower indices, close them, unfollow, reopen, and change the gateway so that applications point to Region B. By 14:22 the site accepts orders again. The last recorded lag before the failure was 4 seconds, so at most about four seconds of changes were lost, well inside the one-minute goal.
When Region A returns hours later, the team does not try to reconnect its old indices. They delete the stale ones and create new followers in Region A that follow Region B. After they catch up, the business decides to keep Region B as the primary for now and use Region A as the new standby.
The goals were set first, and the design followed from them. The team sized for the peak, alerted early, rehearsed the runbook and never tried to merge the old data back in. These habits matter more than any single setting.
20Security
Two clusters joined by CCR share data over a network that may cross the public internet or a provider’s backbone. Security is not optional here.
Protect the Link
All traffic between clusters must be encrypted using TLS, the same technology that puts the padlock in a web browser. Encryption stops anyone on the path from reading or changing data in transit. Where possible, also keep the link off the public internet by using private connections, peering or virtual private networks, and limit network rules so that only the follower’s addresses can reach the leader’s cluster port.
Prove Who You Are
The follower cluster must show credentials to the leader. Depending on your version and setup, these can be certificates, or newer cross-cluster API keys that are limited to specific access on the leader. In both cases, treat the credentials like keys to a safe. Store them in a secrets manager, rotate them on a schedule, and remove them when a cluster is retired.
Give Only the Permissions Needed
Elasticsearch roles have special permissions for CCR. On the leader side, the credentials need only enough rights to read the leader index and its changes. On the follower side, the person or automation creating followers needs rights to manage follower indices and to write into them. The follower does not need any ability to change the leader, and it should not have one. This is the principle of least privilege: every person or program receives no more access than its job requires.
A courier who collects documents from your office needs a key to the pickup room, not a key to the vault. If the courier’s key is lost, the damage is limited to what was in the pickup room. Cross-cluster credentials should be the pickup-room key.
Data Follows the Rules
A follower holds a complete copy of the leader’s data, including sensitive fields. Any access control based on documents or fields in the leader does not automatically apply to the follower unless you define roles on the follower cluster too. Set up users, roles and audit logging on the follower with the same care as on the leader. Encrypt disks in both clusters, and remember that a second copy of sensitive data doubles the places where it must be protected.
It is easy to secure the primary cluster carefully and then leave the disaster recovery cluster with weaker settings because “nobody uses it”. After a failover, that cluster becomes the production system overnight. Both sides deserve the same security standard from the start.
21High Availability, Failure Scenarios and Recovery
CCR exists to help when things fail, and CCR itself can also fail. Plan for both.
What CCR Does When the Link Breaks
If the network between clusters drops, follow tasks cannot read from the leader. They do not give up at once. They retry with growing pauses between attempts, and they resume from their last checkpoint when the link returns. Since the leader keeps history under retention leases, a short outage is repaired automatically, with lag growing during the outage and shrinking after it.
| Scenario | What You See | Likely Cause | What To Do |
|---|---|---|---|
| Short network outage | Lag grows, then falls after the link returns | Link failure or maintenance | Usually nothing; verify that lag returns to normal |
| Long outage beyond retention | Follower cannot resume and reports missing history | Retention lease expired on the leader | Rebuild the follower from scratch and consider longer retention |
| Follower falls behind at peak | Lag climbs during busy hours | Not enough follower write capacity or bandwidth | Add capacity, tune batch sizes or add shards for new indices |
| Leader cluster lost | Follow tasks fail with connection errors | Site disaster | Fail over by promoting followers and redirecting traffic |
| Follower cluster lost | No follower to serve nearby reads or hold the copy | Site disaster on the follower side | Leader keeps working; rebuild the follower cluster and recreate followers |
| Credentials expired | Follow tasks fail with authorization errors | Rotated or expired secrets | Update credentials on the remote cluster settings and resume |
| Leader index deleted by mistake | Follower index follows the deletion or stops with errors | Human error | Restore from a snapshot; CCR cannot undo this |
Split-Brain Thinking for Two Sites
Suppose the link between regions breaks, but both regions are healthy. Each side may believe the other has failed. If your automation promotes the follower while the leader is still alive and receiving writes, you now have two writable copies that will drift apart. This is a split-brain situation. Guard against it by making failover a deliberate decision, confirming from an independent viewpoint that the leader is really down, and making sure old writers cannot keep writing after the switch.
Two managers run branches of a shop. If the phone line between them dies, each might assume the other has closed and start ordering stock for the whole company. Both mean well, and the company ends up with double the orders. A clear rule such as “only the regional director may declare a branch closed” prevents that confusion.
Write your failover decision rules down before you need them: who decides, which signals count, how long to wait, and what to check. A calm rule written in advance beats a panicked debate during an outage.
22Monitoring, Logging and Metrics
A follower that silently stops following looks healthy until the day you need it. Good monitoring makes silent failures loud.
The Numbers That Matter
Operations Behind
The difference between the leader’s global checkpoint and the follower’s. This is the single most important number, because it tells you how much you could lose.
Time Since Last Read
How long ago each follow task last read from the leader. A steadily growing value means the follower has stopped receiving news.
Failed Reads and Writes
Counts of failed read requests from the leader and failed writes on the follower, with the most recent error message.
Operations Per Second
How many operations the follower is applying. Compare it with the leader’s write rate to see whether the follower can keep up.
Followers Versus Leaders
The count of leader indices that should be followed against those that actually are. A gap means an auto-follow pattern is missing something.
Remote Cluster Health
Whether the follower cluster can currently connect to the leader cluster, and how many connections are alive.
Where to Find Them
Elasticsearch offers statistics for individual follower indices and for the whole cluster, showing checkpoints, read and write counts, and recent exceptions. Kibana displays cross-cluster replication status in its management screens, and stack monitoring can collect metrics over time so you can chart them. Export the key numbers to the same dashboards and alerting tools that your team already uses, so CCR alerts arrive in the same place as every other alert.
Alerts That Help
- Lag above your RPO fraction. If the RPO is 60 seconds, alert at 30, so you can act before the goal is broken.
- Time since last read is climbing. This catches a stuck follower even when lag looks small because the leader is quiet.
- Any fatal error on a follow task. A fatal error means the follower has stopped and will not recover on its own.
- Missing followers. Compare the list of leader indices that match your patterns to the list of followers.
- Remote connection down. This gives the earliest warning of link or credential problems.
Lag measured in operations can look tiny during a quiet night, even when the follower is stuck, because there is nothing to copy. Always watch the freshness of the follower’s last successful read together with the lag, so a stalled link cannot hide behind a quiet leader.
Logs and Audit Trails
Keep cluster logs for both sides, and record who created, paused, promoted or removed followers. During an incident review, these records answer questions like “who promoted the follower, and when?” Good records also help detect mistakes, such as a follower being paused for maintenance and forgotten for weeks.
23Advantages, Disadvantages and Trade-offs
CCR solves real problems, and it also brings real costs. A fair look at both sides helps you decide when it is worth using.
Advantages
- Keeps a live copy in another cluster, protecting against loss of a whole site
- Recovery is fast, because the copy is already loaded and searchable
- Brings data close to users in other regions, so reads are quicker
- Lets many small clusters feed one central reporting cluster
- Built into Elasticsearch, with no extra pipeline software to run
- Pull-based design protects the leader and copes with link problems
- Copies mappings and many settings automatically, so structures stay in sync
Disadvantages
- Asynchronous, so a sudden disaster can lose the last moments of data
- Follower is read-only, so it cannot accept writes until it is promoted
- Copies mistakes and deletions too, so it is not a backup
- Needs a paid Elastic subscription level
- Doubles storage and adds network and cross-region data transfer costs
- Follower shard count must match the leader’s, which limits later changes
- Failback after a disaster needs a careful rebuild
Key Trade-offs to Decide
Freshness versus cost. A faster link, more follower capacity and tighter monitoring keep lag lower, but each adds cost. Decide how fresh the copy must be and buy only that much.
Simplicity versus flexibility. One leader with one follower is easy to run and to explain. Hub-and-spoke and active-active patterns unlock more uses but need stronger discipline in naming, aliases and runbooks.
Speed of recovery versus risk of a wrong switch. Automated failover reduces the RTO, but it raises the risk of a mistaken failover or a split-brain. Manual failover is slower, but gives people a chance to confirm. Many teams choose semi-automated steps with a human decision at the start.
Local reads versus fresh reads. Serving users from a nearby follower gives speed, but with slightly old data. Serving everyone from the leader gives fresh data, but with more distance. Use each where it fits.
24Design Patterns and Anti-Patterns
Teams that have run CCR in production repeat the same good habits, and the same avoidable mistakes.
Patterns That Work
Warm Standby
A follower cluster in another region receives everything and stays ready. Applications use it only after a failover, or for read-only reporting in the meantime.
Regional Read Replicas
Followers in each major region serve local searches, while all writes go to one leader. Sensitive reads that need full freshness go to the leader.
Hub and Spoke Reporting
Many source clusters feed one central cluster through prefixed follower names, giving analysts one place to look.
CCR Plus Snapshots
A follower protects against site loss, and regular snapshots protect against mistakes and corruption. Together they cover both dangers.
Anti-Patterns to Avoid
What people do
They treat CCR as their only backup.
Why it hurts
An accidental deletion, a bad script or corrupted data on the leader is copied to the follower within seconds. When the mistake is discovered, both copies are ruined.
Better approach
Use CCR for availability and locality, and keep regular snapshots stored separately for recovery from mistakes.
What people do
They stretch one cluster across distant regions instead of using two clusters with CCR.
Why it hurts
High latency and network wobbles hurt cluster stability, and a broken link can leave the cluster unable to elect a leader node or protect its data safely.
Better approach
Keep each cluster inside a low-latency location and connect clusters with CCR.
What people do
They set up CCR once and never test a failover.
Why it hurts
On the day of the disaster, the team finds missing permissions, unknown steps, forgotten indices or applications that cannot switch. The RTO turns out to be hours, not minutes.
Better approach
Rehearse failover and failback regularly, and keep the runbook current with what you learn.
What people do
They allow writes to both the old leader and the promoted follower after a failover.
Why it hurts
The two copies diverge and cannot be merged automatically. Some data will be lost or need manual reconstruction.
Better approach
Make redirection complete, stop old writers and always rebuild the old leader as a fresh follower of the new primary.
What people do
They never monitor lag and only check that “the follower exists”.
Why it hurts
A follower can exist and still be days behind because of a stuck task or expired credentials. The RPO is broken without anyone knowing.
Better approach
Alert on lag, on freshness of the last read and on follow task errors, not just on the existence of follower indices.
25Best Practices and Common Mistakes
A short list of habits that separate smooth CCR deployments from painful ones.
Write down the goals first
Set the RPO, the RTO and the purpose of each follower before choosing settings. Every later decision should trace back to those goals.
Decide the shard count carefully
Follower shards must match leader shards. Choose a shard layout with growth in mind, because changing it later means rebuilding.
Size for peak, not average
Link bandwidth, follower disks and processing power should handle the busiest hour with room to spare.
Automate followers with patterns
Use auto-follow patterns for rolling indices and data streams, and review the list of followers regularly.
Secure both sides equally
Encrypt the link, use least-privilege credentials, rotate secrets and apply the same access rules on the follower cluster as on the leader.
Rehearse and document
Practice failover and failback, time the steps, and keep a runbook that a tired person can follow.
Common Mistakes
- Forgetting the license. Discovering late that the required subscription level is missing can stall an entire project.
- Underestimating storage. The follower holds a full copy plus its own replicas, and grows as the leader grows.
- Long pauses. Leaving a follower paused past the retention period forces a full rebuild.
- Name collisions. Copying similarly named indices from several leaders into one follower cluster without prefixes.
- Testing only the happy path. Trying CCR when everything is healthy, but never during a broken link or expired credential.
- Ignoring application changes. Failing to plan how applications will find the promoted cluster, for example through DNS or gateway settings.
26Day-Two Operations: Living With CCR
Setting CCR up is one day of work. Running it for years is the rest. These are the routine tasks that keep a setup healthy.
Pausing and Resuming
Sometimes you need to stop following on purpose, for example during maintenance on the follower cluster or the network. Pausing a follower stops its read tasks but keeps its data. Resuming makes it continue from its last checkpoint. The important rule is to keep pauses short compared with the leader’s retention period. A pause of a few hours is usually harmless. A pause of several days may force a full rebuild.
Changing Mappings and Settings
When a new field is added on the leader, the follower learns about the new field automatically as part of following. You should not change the structure of a follower by hand while it is following, because the leader is the single source of truth. If you need a different structure, change it on the leader, or create a separate index on the follower side from the copied data.
Big structural changes, such as splitting an index into more shards, cannot be applied to a follower in place. The usual approach is to create a new leader index with the new layout, copy data into it and create a new follower for it. Plan these moves during quiet times, and keep the old pair running until the new pair is caught up and verified.
Rotating Credentials
Secrets should not live forever. When you rotate the credentials that the follower uses to reach the leader, update the remote cluster settings on the follower before the old credentials stop working. Check the connection afterward and confirm that lag stays low. Doing this on a schedule turns a risky emergency into a boring routine.
Changing the lock on a shared storage room works best when you give the new key to everyone before you swap the lock. If you change the lock first, the couriers arrive to a door that will not open. Rotating credentials follows the same order: new key first, then retire the old one.
Upgrading
Cluster upgrades need special care because leader and follower must remain compatible with each other during the process. Read the upgrade notes for your versions, test in a copy of your setup, and upgrade in the order that Elastic’s documentation recommends. After each step, verify that following continues and that lag is normal. Keep the runbook for failover handy in case an upgrade goes wrong on one side.
Cleaning Up
Followers that nobody needs any more waste disk and network. Review the list of followers regularly. Remove those for retired indices, and unfollow those that are finished. Also review auto-follow patterns: an old pattern that still matches a busy naming scheme can fill the follower cluster with data that no longer has a purpose.
Keep a short calendar of recurring CCR chores: credential rotation, failover rehearsal, follower review and upgrade planning. Tasks that are written down and scheduled actually get done, while tasks that live only in someone’s memory tend to be forgotten until they cause an incident.
27A Simple Troubleshooting Guide
When something looks wrong, a calm, ordered check finds the cause faster than guessing. Work from the outside in.
Is the link alive?
Check whether the follower cluster reports that its remote cluster connection to the leader is connected. If not, look at firewalls, routes, certificates and expired credentials before anything else.
Are the follow tasks running?
Look at the follower statistics. Each shard should have an active task. A task with a fatal error has stopped for good and needs a person to look at the message.
Is the lag growing or stable?
Compare the leader’s and follower’s checkpoints over a few minutes. A growing gap means the follower cannot keep up. A stable gap that is not shrinking may mean the follower is stuck.
Is the follower healthy on its own?
Check disk space, memory pressure, cluster health and shard allocation on the follower. A follower that cannot write because a disk is full will fall behind no matter how good the link is.
Is the leader still keeping history?
Confirm that soft deletes and retention leases are working for the leader index. If history was lost, the follower must be rebuilt.
Did anything change recently?
Recent upgrades, credential rotations, network changes, new auto-follow patterns or mapping changes are the most common triggers. Ask what changed before the trouble began.
Common Symptoms and First Guesses
| Symptom | First Thing To Check |
|---|---|
| Follower document count is lower than the leader’s | Whether following is still running and whether lag is growing |
| Follower creation fails at the start | Whether soft deletes are on for the leader index and whether the follower name already exists |
| Auto-follow did not create a follower | Whether the name matches the pattern and whether the pattern is paused or has recent errors |
| Authorization errors on the follower tasks | Expired or rotated credentials and missing permissions |
| Initial copy is very slow | The recovery speed limit, link throughput and load on the leader |
| Searches on the follower miss recent data | Current lag and the refresh interval on the follower |
Write the troubleshooting order into your runbook, and add every new lesson to it after each incident. Over time, this short document becomes the most valuable page in the whole CCR setup.
28Deployment and Cloud Considerations
Where the clusters run, and how they are connected, shapes cost, speed and risk.
Choosing Where Clusters Live
Clusters can run in your own data centers, on virtual machines in a public cloud, in containers managed by Kubernetes, or on a managed service such as Elastic Cloud. A common design places the leader in one cloud region or data center and the follower in another, far enough apart that one event cannot damage both. Choose locations with different power, network and natural risks, not just different rooms in the same building.
Connecting the Clusters
The network design has three choices. The clusters can talk over the public internet with strong encryption, over private links offered by the cloud or a telecom provider, or over a virtual private network. Private links give steadier speed and lower risk, but they cost more and take time to arrange. Whatever you choose, test the actual throughput and delay between the two sites before designing around them.
Counting the Cost
CCR costs money in four places. First, the follower cluster needs its own servers and disks. Second, the network link may have a fixed fee. Third, many cloud providers charge for data moved between regions, and a busy index can move a lot of data over a month. Fourth, people need time to build, test and maintain the setup. Estimate all four before deciding, and compare them with the cost of an outage for your business.
Environments and Releases
Practice in a test environment that mirrors production: two small clusters, the same security settings and realistic sample data. Use it to test upgrades, credential rotation and failover. Elasticsearch versions between leader and follower must be compatible, so read the compatibility rules in Elastic’s documentation before planning upgrades. Usually, you upgrade in an order that keeps the two clusters compatible at every step, rather than upgrading both at once.
Example: A Quiet Upgrade Weekend
A team plans to upgrade both clusters to a newer version. They first read the compatibility rules, upgrade the follower cluster in the test environment, and confirm that following continues to work. On the production weekend, they upgrade in the tested order, check lag after each step, and finish with a failover rehearsal on the follower. Because the steps were practiced, the upgrade happens without any customer-visible impact.
Cross-cluster replication is available with Elastic’s paid subscription levels, and the exact tier and version requirements have changed over time. Confirm both in Elastic’s current documentation before you commit to a design.
29Real-World Scenarios
The stories below are illustrative composites that show how organizations of different kinds use CCR. They are not accounts of specific companies.
A Global Online Retailer
A retailer with customers on three continents keeps its product catalog in one leader cluster. Follower clusters in each region serve local search, so a shopper in Sydney sees results as quickly as one in Amsterdam. Cart and stock checks go to the leader, which keeps them exact. During a regional outage, traffic can be moved to another region’s follower with only a short interruption.
A Bank Meeting Resilience Rules
A bank uses Elasticsearch for searching transaction histories and fraud alerts. Regulators ask it to prove that it can keep services running after the loss of a data center. It runs a follower cluster in a second data center, rehearses failover twice a year, and keeps signed records of the results. Snapshots stored separately protect it against mistakes and tampering.
An Energy Company With Field Sites
A company operating wind farms places a small cluster at each site to collect sensor readings, even when the connection to head office is poor. Each site is a leader. A central hub cluster follows all of them, giving engineers one dashboard for the whole fleet. When a site loses connectivity, its local cluster keeps working and the hub catches up once the link returns.
A Software Provider With Data Residency Rules
A software company serves customers in several regions, some of whom require that their data stays in their own region. It keeps separate leader indices per region and uses followers only for data that its customers have allowed to be replicated. This gives faster access for approved data while respecting local rules for everything else.
A Media Publisher During Big News
A news site sees a huge spike in searches during major events. Followers in several regions absorb most of the read traffic, keeping the leader free to accept new articles and edits. If the main region has trouble, editors continue from a promoted follower.
30Common Myths and Misunderstandings
Half-true ideas about CCR cause wasted effort and bad surprises. Here are the ones that appear most often.
“CCR is a backup.”
It is a live mirror. Deletions and mistakes are copied too. Use snapshots for backups.
“CCR means zero data loss.”
Replication is asynchronous, so the follower can be a few seconds behind. Plan for a small, measured loss window.
“I can write to the follower.”
Not while it is following. It is read-only until it is unfollowed and promoted.
“Both clusters can write to the same index.”
An index has one leader. Two-way designs use separate indices for each side, followed by the other.
“CCR and cross-cluster search are the same.”
CCR copies data. Cross-cluster search queries remote data without copying it. They solve different problems.
“Once set up, it needs no attention.”
Links break, credentials expire, indices are added. Monitoring and rehearsals are part of the deal.
“Failover is just flipping a switch.”
It involves several steps and a decision, and applications must be redirected. Failback needs a rebuild.
“Followers can have any shard layout.”
The number of primary shards must match the leader. Only replica counts and placement are free.
Treat every disaster-recovery claim as unproven until you have tested it. A design is only as good as the last rehearsal that showed it working.
31Glossary of Terms
A quick reference for the important words in this guide, in plain language.
| Term | Plain Meaning |
|---|---|
| Asynchronous replication | Copying data after confirming the write locally, so the copy may lag slightly |
| Auto-follow pattern | A rule that automatically creates followers for new leader indices whose names match |
| Bootstrapping | The first bulk copy of a leader index that starts a new follower |
| CCR | Cross-Cluster Replication, the feature that copies indices between clusters continuously |
| Cluster | A group of Elasticsearch nodes working together as one system |
| Cross-cluster search | A feature that searches several clusters at once without copying their data |
| Data stream | A structure for time-based data that manages hidden backing indices behind one name |
| Eventual consistency | The promise that copies may differ briefly but will match once changes arrive |
| Failback | Returning to the original site after a failover, usually by rebuilding it as a follower |
| Failover | Switching from a failed primary system to a standby copy |
| Follower index | A read-only copy of a leader index in another cluster |
| Global checkpoint | The highest sequence number safely stored on all in-sync copies of a shard |
| ILM | Index Lifecycle Management, which moves indices through stages as they age |
| Leader index | The original, writable index that followers copy |
| Least privilege | Giving people and programs only the access their job needs |
| Remote cluster | A saved connection to another cluster, with a nickname |
| Replication lag | How far the follower is behind the leader |
| Retention lease | A follower’s claim asking the leader to keep history from a certain point onward |
| RPO | Recovery point objective, the amount of recent data you can afford to lose |
| RTO | Recovery time objective, the time you can afford to be without service |
| Sequence number | A counter attached to each change on a shard that records the order of events |
| Shard | One piece of an index, stored and searched independently |
| Soft deletes | A setting that keeps recent history, including deletions, so followers can read it later |
| Split-brain | A situation in which two systems both believe they are the primary and accept conflicting writes |
| Unfollow | Cutting the link between a follower and its leader so the follower becomes an ordinary index |
32Frequently Asked Questions
Quick answers to the questions people ask most often when they first meet CCR.
33Summary and Key Takeaways
Key Takeaways
- CCR keeps a live copy of an index in another cluster. A read-only follower pulls every change from a leader, so the copy stays close to current.
- It is asynchronous and pull-based. Writes stay fast, links can fail without breaking the leader, and the follower runs a few seconds behind, so a sudden disaster can lose the last moments.
- Sequence numbers, soft deletes and retention leases make it work. Followers ask for exactly the changes they are missing, and the leader keeps that history for a limited time.
- Three uses dominate. Disaster recovery through failover, nearby reads for faster searches, and hub-and-spoke reporting from many small clusters.
- CCR is not a backup. It copies mistakes and deletions, so pair it with snapshots, and use cross-cluster search when you only need to query remote data without copying it.
- Plan for peak, secure both sides and monitor lag. Size links and follower capacity for the busiest hour, encrypt the connection, use least-privilege credentials, and alert on lag and stalled reads.
- Rehearse failover and failback. Promotion is a few clear steps, failback needs a rebuild, and only regular practice turns a plan into a real recovery time.