Start here. By the end of this chapter you will know what a disaster is in this context, what “multi-region” means, and which two numbers rule every design decision.

A disaster, in this context, is any event that makes a whole site unavailable or destroys data that you cannot easily rebuild. A power failure across a data center, a fiber cut, a cloud region outage, a flood, a faulty software release that corrupts indices, a mistaken command that deletes them, or an attacker who encrypts your storage: all count. Disaster recovery, often shortened to DR, is the set of designs and procedures that lets you carry on, or quickly restart, after such an event.

A region is a geographic area, for example “Frankfurt” or “Virginia,” made of one or more data centers that share a location and usually share some infrastructure. A multi-region architecture places copies of your Elasticsearch data, and the ability to run on it, in at least two regions that are far enough apart that a single event cannot hurt both.

What is special about Elasticsearch?

Elasticsearch is a distributed system, so people sometimes assume it “just handles” failure by itself. It handles some kinds of failure very well: a machine dying, a disk breaking, even a whole zone going offline inside one region. It was not designed to stretch one single cluster across regions that are hundreds or thousands of kilometers apart, because the constant chatter between nodes needs fast, reliable links. So for region-level disasters, Elasticsearch gives you separate clusters plus tools that keep them in step: snapshots, cross-cluster replication, and cross-cluster search. A multi-region DR architecture is the careful combination of those tools with your ingest paths, your traffic routing and your procedures.

The two numbers that rule everything

Every DR design is judged by two numbers. The Recovery Point Objective, or RPO, is how much recent data you can afford to lose, measured in time. “RPO of five minutes” means that after a disaster you may lose at most the last five minutes of data. The Recovery Time Objective, or RTO, is how long you can afford to be down. “RTO of thirty minutes” means service must be back within half an hour. Smaller numbers cost more money and more effort. The whole art of the subject is choosing numbers that match the real value of your data, and then designing for them.

A short history

Elasticsearch has offered snapshot and restore for a long time, since the early days of the 1.x series, and for years it was the main DR tool: copy the data to cheap storage, and rebuild elsewhere if needed. Later versions added scheduled snapshot management, index lifecycle management, and searchable snapshots that can serve queries straight from stored snapshots. In the 6.x series, cross-cluster replication arrived, first as a beta and then as a fully supported feature. It allowed one cluster to continuously copy indices from another, which for the first time gave Elasticsearch a built-in, near-real-time way to keep a second region warm. Cross-cluster search, improved security keys for cross-cluster links, and hosted deployment features that make linking clusters across regions easier followed. Today a multi-region design can mix all of these building blocks according to the RPO and RTO you need.

A well-built cluster already survives many failures. To see why it still is not enough, list the ways things go wrong and match each to a layer of protection.

Failures come in sizes

It helps to sort failures by how much they take away. A node failure is one machine going down, which replica shards handle automatically. A zone failure is one data center or availability zone inside a region going down, which a cluster spread over three zones with shard allocation awareness can survive. A region failure takes out every zone in a region at once, for example a regional network fault, a control-plane outage at a cloud provider, or a large natural event. And a logical failure is one where the machines are perfectly healthy but the data is wrong: someone deleted an index, a script overwrote documents, a bug wrote garbage, or ransomware encrypted the storage.

Why replicas are not disaster recovery

New users often say, “We already have replicas, so we are safe.” Replicas protect against the loss of machines. They live in the same cluster, in the same region, under the same administrators, and they faithfully copy every change, including harmful ones. If a region goes dark, all the replicas go with it. If someone deletes an index, every replica is deleted in the same instant. Replicas are like keeping two copies of a document in the same drawer of the same office: fine if one copy gets coffee on it, useless if the building burns down.

Why not stretch one cluster across regions?

The tempting idea is to put some nodes in each region and let one big cluster span them. It rarely works well. The nodes must agree on the state of the cluster, and that agreement depends on quick and reliable communication. Between distant regions, delays are long and links sometimes break. Writes must wait for copies on the far side, so every indexing operation slows down. And if the link between regions fails, the cluster has to decide which side keeps going, which risks pausing everything or confusing the system. Elastic’s own guidance is to keep a cluster inside one low-latency location, spread over zones, and use separate clusters for separate regions.

The business side

Regions really do fail, and so does everything that depends on a single one. Beyond raw outages, many industries have rules. Financial regulators in several parts of the world require firms to show that critical systems can recover within stated times. Contracts with customers include availability promises. Internal policies say that the loss of one location must never mean the loss of the business. A design that survives only zone-level failures may not satisfy any of these.

None of this means every system needs the most expensive design. A small internal tool might be perfectly fine with a nightly snapshot and a day to rebuild. The next chapters show a ladder of options so that you can pick the rung that fits.

A dozen words unlock the whole subject. Each one is explained with what it is, why it exists, where you meet it, and a picture from everyday life.

Region and availability zone

A region is a large geographic area. Inside it are several availability zones, physically separate data centers with independent power and networking, connected by fast links. Zones protect you against local failures. Regions protect you against wide ones. Zones are like separate buildings on one campus; regions are separate campuses in different cities.

Replica shard

Elasticsearch splits each index into shards, and can keep extra copies of each shard called replicas. Replicas give safety against node loss and extra read capacity, but they stay inside one cluster. They are the first line of defense, not the last.

Snapshot

A snapshot is a saved copy of indices, and optionally of cluster settings, written to an external repository such as object storage. Snapshots are incremental: after the first one, only the new pieces are stored. A snapshot is a photograph of your data at one moment, kept in a safe place away from the cluster.

Cross-cluster replication

Cross-cluster replication, or CCR, makes a follower index in one cluster continuously copy a leader index in another. The follower pulls changes from the leader, so it stays only seconds behind. It is like a scribe in a second office who copies every new page written in the first office as soon as it appears.

Cross-cluster search

Cross-cluster search lets a cluster query indices in other clusters without copying them. It does not create a backup, but it helps DR designs by letting users search several regions with one request and by letting a DR region read data that lives elsewhere.

Failover and failback

Failover is the act of moving service from the failed primary to the DR region. Failback is the planned return to the primary after it has recovered. Failover is usually urgent and simple in aim. Failback is usually calm but surprisingly tricky in detail, because the two sides may have drifted apart.

Promotion

A follower index is read-only. To accept writes in the DR region it must be promoted, that is, converted into a regular index that no longer follows a leader. Promotion is the technical heart of failover. It is like a deputy becoming the chief when the chief is gone.

Fencing

Fencing means making sure the old primary can no longer accept writes once the DR region has taken over. Without it, both sides may accept different writes and the data diverges, an unpleasant situation often called split-brain. Fencing is like taking the old chief’s keys so that two people are not giving orders at once.

Active-passive and active-active

In an active-passive design, one region serves all traffic and the other waits. In an active-active design, both regions serve traffic at the same time, each usually responsible for its own share of the writes. Active-passive is simpler and cheaper. Active-active offers faster recovery and use of the second region’s capacity, but demands careful design to avoid conflicting writes.

Runbook

A runbook is a written, step-by-step procedure for a specific situation, such as “region failover.” A good runbook can be followed by a tired person at 2 a.m. It names who decides, who acts, which commands to run, and how to check that each step worked.

Now let us zoom out and look at a complete, typical multi-region design: which parts exist, where each one lives, and how they connect.

A multi-region DR design is not one feature. It is a small system made of several layers that cooperate. The diagram below shows a common warm-standby arrangement: a primary region that normally serves everything, a DR region kept in step, and a traffic manager in front that decides where users go.

flowchart LR
    APP["Applications and users"] --> DNS["Global DNS or traffic manager"]
    DNS -->|"normal traffic"| PLB["Primary load balancer"]
    DNS -.->|"after failover"| DLB["DR load balancer"]
    subgraph PRIMARY["Primary region"]
        PLB --> PC["Primary cluster across three zones"]
        PK["Kibana primary"] --> PC
    end
    subgraph DRREGION["DR region"]
        DLB --> DC["DR cluster across three zones"]
        DK["Kibana DR"] --> DC
    end
    PC ==>|"cross-cluster replication"| DC
    PC --> SNAP["Snapshot repository in object storage"]
    SNAP -->|"second copy kept in DR region"| SNAP2["Snapshot repository in DR region"]
    SNAP2 -.-> DC

Walking through the diagram

At the top left, applications and users send their searches to a name, not to a machine. A global DNS service or traffic manager answers with the address of the load balancer in the region that is currently in charge. Under normal conditions that is the primary. After a failover, the answer changes to the DR region’s load balancer. Inside each region, the load balancer forwards requests to the Elasticsearch cluster, which is itself spread over three zones. Each region also has its own Kibana.

The thick arrow between the clusters is cross-cluster replication: the DR cluster continuously pulls changes from the primary and applies them, so its copy stays seconds behind. The lower arrows show the snapshot path: the primary writes snapshots to object storage, and a second copy of that snapshot data exists in the DR region, from which the DR cluster could restore in an emergency. Replication gives you speed. Snapshots give you a safe historical copy that survives even mistakes that replication would faithfully copy.

The components in detail

Configuration is part of the architecture

Replication copies data. It does not automatically copy everything that makes the DR cluster work like the primary. Index templates, lifecycle policies, ingest pipelines, users and roles, saved dashboards in Kibana, alerting rules, and machine learning jobs are all configuration. If you forget them, the DR cluster may hold every document yet be unusable, because the dashboards are missing and nobody can log in. The best practice is to keep configuration as code in a repository and apply it to both regions with the same automation.

This is the engine room. You do not need to be a distributed systems expert, because every idea comes with a picture.

How cross-cluster replication works

Every write to a leader shard is given a growing sequence number, like a page number in a ledger. The leader keeps a record of recent operations, using a mechanism that retains deleted and updated document history for a while, called soft deletes. The follower shard keeps track of its own progress, a value called the global checkpoint, which says “I have applied everything up to number 48,210.” Then it simply asks the leader, over the remote link, “please send me the operations after 48,210.” The leader sends a batch, the follower applies it, moves its checkpoint forward, and asks again. Because the follower does the asking, the leader does not need to know about the follower’s health, and a slow follower does not slow the leader’s writes.

sequenceDiagram
    participant W as Writer application
    participant L as Leader shard in primary region
    participant F as Follower shard in DR region
    W->>L: Index a document
    L->>L: Record it and assign a sequence number
    F->>L: Ask for operations after my checkpoint
    L-->>F: Send the new operations in a batch
    F->>F: Apply them and advance my checkpoint
    F->>L: Ask again for the next operations

Catching up after an outage

What if the link between regions breaks for a while? The leader holds on to the history that the follower still needs, using a mechanism called a retention lease. It is like a librarian who keeps a book on hold for someone. But the hold does not last forever: by default it expires after roughly twelve hours, to prevent a forgotten follower from making the leader hoard history endlessly. If the follower comes back within the lease period, it catches up quickly by replaying the missing operations. If it comes back too late, the history is gone and the follower has to be rebuilt from scratch by copying the leader’s files again, a much slower job. This is a very practical reason to monitor replication and to repair a broken link within hours, not days.

The first copy

When a follower index is first created, it has no history to replay. It performs an initial bootstrap, copying the leader’s shard files across the network, and only afterward switches to the sequence-number method. Bootstrapping a very large index takes time and bandwidth, and the copy speed is deliberately limited by default so that it does not overwhelm the leader or the link. Plan for it when you first set up DR, and when you add a large new index.

What replication does and does not copy

Replication copies documents and the mapping changes that go with them, together with certain index settings. It does not copy cluster-wide items such as templates, lifecycle policies, pipelines, security roles or saved dashboards, and some index-level details, for example aliases, may need to be set up separately. Also, because replication is faithful, it copies mistakes: a deleted document, or a wrong update, is replicated to the follower within seconds. This is exactly why replication alone is not a backup.

How snapshots work

Elasticsearch stores shard data in immutable files called segments. A snapshot copies these files to the repository, and, importantly, only copies files that the repository does not already hold. So the first snapshot is large, but later ones contain only the new segments. Deleting an old snapshot removes only the files that no other snapshot needs. This makes it practical to take snapshots often, for example every fifteen minutes or every hour, and to keep many of them.

A snapshot of many shards is not one instant frozen across the whole cluster. Each shard is captured at about the time the snapshot reaches it. Documents written while the snapshot is running may or may not be included. For most uses, that is fine, but it is worth remembering when you compare document counts after a restore.

Restoring from a snapshot

A restore copies the segment files back into a cluster and brings the shards online. The time it takes depends mostly on data size and on how fast the storage and network can deliver. A quick calculation shows why this matters for RTO: restoring 20 terabytes at 500 megabytes per second takes about 40,000 seconds, or a little over eleven hours. Newer features can shorten this by mounting a snapshot as a searchable snapshot, so queries can begin before all the data has been copied, though that feature depends on your license level.

A DR setup lives through several phases: calm operation, trouble, failover, life in the DR region, and a careful return. Knowing each one prevents most surprises.

stateDiagram-v2
    [*] --> Normal
    Normal --> Degraded : region impaired
    Degraded --> Normal : recovers quickly
    Degraded --> FailoverInProgress : decision to fail over
    FailoverInProgress --> RunningInDR : DR cluster promoted
    RunningInDR --> Resync : old primary returns
    Resync --> FailbackInProgress : old primary caught up
    FailbackInProgress --> Normal : traffic moved back

Phase one: normal operation

Most of the time nothing is wrong, and this is when the real work happens. Data flows into the primary. Replication copies it to the DR region within seconds. Snapshots run on schedule. Monitoring watches the lag and the health of every link. Configuration changes are applied to both regions. Regular drills prove that the machinery still works. The quality of your disaster response is decided here, long before any disaster.

Phase two: degraded

Then something goes wrong. Maybe the region is slow, or partly down, or nobody is sure. This is the hardest moment, because failing over is a big decision with a cost of its own, and failing over unnecessarily is also a failure. Good teams define in advance the signals that mean “declare a disaster,” who has the authority to decide, and how long they will wait for the region to recover on its own.

Phase three: failover

Once the decision is made, the team follows the runbook: stop writes to the old primary if possible, promote the follower indices, redirect ingestion, switch traffic, and verify. The data loss window equals the replication lag at the moment of the outage, and it is recorded so that the business knows exactly what may be missing.

Phase four: running in the DR region

The DR cluster is now the primary in practice. It serves users and takes writes. It may also need to grow, because a standby sized for cost may be smaller than what production needs, and many teams add nodes at this stage. Meanwhile, snapshots should start from the new primary as soon as possible, so that the safety net is rebuilt.

Phase five: resynchronization and failback

When the original region returns, its data is stale, and it may even contain writes that never reached the DR region before the disaster. It cannot simply be switched back on. Its indices are rebuilt as followers of the now-active cluster, they catch up, and only when they are fully in step does the team plan a calm, scheduled switch back. This can be done during a quiet hour with the same discipline as the failover.

Before going deeper, here is the route you follow to build a first warm-standby design. Knowing the road makes the later chapters easier to follow.

The work is mostly about preparation and decisions. The technical commands are short. The hard parts are choosing numbers, agreeing on scope, and rehearsing. Here is the journey for a beginner building a primary and one DR region.

Before you begin

Check four things. First, the license: cross-cluster replication is a paid feature and generally requires a higher subscription tier on the cluster that follows, so confirm what your license includes. Second, the versions: keep both clusters on compatible versions, and read the compatibility notes before upgrades. Third, the network: the DR cluster must be able to reach the primary through a remote cluster connection, with the right ports and an encrypted, authenticated link. Fourth, the scope: decide which data truly needs protection. Not every index deserves the same treatment.

There is no single “DR design.” There is a ladder of designs, from cheap and slow to expensive and fast. Pick the rung that fits the value of your data.

flowchart TD
    Q["How much data can you afford to lose and how long can you be down?"] --> A{"Hours of loss and downtime acceptable?"}
    A -->|"Yes"| B["Backup and restore"]
    A -->|"No"| C{"Minutes acceptable and cost matters most?"}
    C -->|"Yes"| D["Pilot light"]
    C -->|"No"| E{"Must both regions accept writes?"}
    E -->|"No"| F["Warm standby"]
    E -->|"Yes"| G["Active-active"]

Rung one: backup and restore

The cheapest approach. You take regular snapshots to object storage, keep a copy in another region, and write down how to build a new cluster and restore into it. No second cluster runs in normal times. When disaster strikes, you create a cluster in the DR region, restore the data, and switch traffic. The RPO equals the time since the last usable snapshot, often minutes to hours. The RTO equals the time to build the cluster plus the time to restore the data, often hours or more. This suits internal tools, archives and anything that can be down for a while. It is like keeping insurance papers and a spare key in a bank vault: cheap, but rebuilding takes days.

Rung two: pilot light

A small, always-on core in the DR region, with the rest ready to be created quickly. In Elasticsearch terms, you run a small DR cluster that continuously replicates only the most critical indices, while the bulk of the data is covered by snapshots. When disaster strikes, you promote the critical indices, scale the cluster up, and restore the rest in the background. The name comes from the small flame in a gas heater that lets the full burner light instantly. It gives a much better RTO for the critical part at modest cost.

Rung three: warm standby

A fully working DR cluster runs all the time, continuously following the primary through replication. It is usually smaller or leaner than the primary but complete in function, and it may even serve read-only dashboards or reports to relieve the primary. When disaster strikes, promotion and a traffic switch bring it into service in minutes. The RPO is the replication lag, usually seconds. The RTO is minutes. The cost is a second cluster running around the clock. This is the most common choice for important production search.

Rung four: active-active

Both regions serve users and accept writes at all times. Because replication in Elasticsearch flows in one direction per index, active-active designs achieve two-way sharing by giving each region its own indices, or its own data streams, for the data it writes, and having each region follow the other’s. Alternatively, an ingestion queue sends every event to both regions. Users read from the nearest region, and a disaster simply removes one of the two. The RTO can be close to zero, since the survivor is already serving. The price is complexity: you must prevent conflicting writes, remove duplicates, and manage two-way flows.

These ranges are only rough guides. Your real numbers depend on data size, link speed, cluster tuning and how well you have rehearsed.

A good multi-region design stands on a good single-region design. If each region is fragile, you will be failing over far too often.

Regions are big, and most incidents are much smaller than a whole region. If each of your clusters can survive the loss of a machine or a whole zone without human help, you reserve the expensive regional failover for real regional disasters. So the first layer of any DR architecture is a cluster that is well spread inside its region.

flowchart LR
    subgraph ZA["Zone A"]
        M1["Master-eligible node"]
        H1["Data node with primary shards"]
    end
    subgraph ZB["Zone B"]
        M2["Master-eligible node"]
        H2["Data node with replica shards"]
    end
    subgraph ZC["Zone C"]
        M3["Master-eligible node"]
        H3["Data node with replica shards"]
    end
    M1 --- M2
    M2 --- M3
    M1 --- M3

Three zones, not two

Elasticsearch decides who is in charge by majority vote among its master-eligible nodes. With three such nodes in three separate zones, the loss of any one zone still leaves two of the three, a majority, and the cluster carries on. With only two zones, the loss of one zone might leave exactly half the voters, which is not a majority, and the cluster may stop. If you truly have only two zones, you need a small tiebreaker node in a third location. The simplest rule is to use three zones wherever possible.

Shard allocation awareness

Nodes can be labeled with the zone they belong to, and Elasticsearch can be told to be zone aware. It then makes sure that a primary shard and its replica never sit in the same zone. With one replica and three zones, losing a zone still leaves a full copy of every shard elsewhere. A related option, called forced awareness, prevents the cluster from piling all the copies into the surviving zones during an outage, which could overload them.

Enough replicas and enough headroom

At least one replica for important indices is the usual rule. You must also leave capacity headroom: if one of three zones disappears, the remaining two must carry the whole load. That means each zone should be sized to hold roughly half of the total load, not a third of it. It is like packing three lifeboats so that any two can carry all the passengers.

Dedicated master nodes and tiers

In larger clusters, run three dedicated master-eligible nodes so that heavy searches cannot starve the coordination work. Use data tiers, hot for recent data, warm and cold for older data, and, where your license allows, frozen for the oldest, so that the expensive fast storage holds only what needs it. Fewer, larger, well-tiered shards also mean faster recoveries after failures, both inside a region and across regions.

Snapshots are the oldest, simplest and most forgiving DR tool. They are your last line of defense, especially against mistakes that replication copies too well.

Repositories

A snapshot repository is a place outside the cluster where snapshots are stored, usually cloud object storage such as Amazon S3, Google Cloud Storage or Azure Blob Storage, or, in private data centers, a shared file system. A repository is registered with the cluster by name, together with the credentials needed to use it. Because it lives outside the cluster, it survives the loss of every node.

Scheduling with snapshot lifecycle management

You do not take snapshots by hand. Snapshot lifecycle management, or SLM, lets you define policies: which indices, how often, and how long to keep the snapshots. A typical policy might take a snapshot every fifteen minutes for critical data and keep the last forty-eight hours, then keep one per day for thirty days. Because snapshots are incremental, frequent snapshots cost little. Your RPO from snapshots is roughly the interval between them.

Putting a copy in another region

A snapshot repository in the same region as the cluster does not protect against a regional disaster. There are two common ways to get a copy elsewhere. The first, and the one Elastic’s guidance favors, is to register two repositories, one in each region, and let policies write to both. The DR region then reads from a repository that sits in its own region. The second is to use the storage provider’s own bucket replication to copy the repository’s files to a bucket in another region. This is convenient, but you must remember that the provider copies files without understanding Elasticsearch’s internal bookkeeping, and that Elastic’s documentation warns against modifying or copying repository contents behind its back. A replica of a bucket taken in the middle of a snapshot may be incomplete. If you use it, test restores from the copied repository regularly, and prefer the two-repository approach when in doubt.

Protection against deliberate harm

Ransomware and malicious insiders like to destroy backups too. Protect the repository with the storage provider’s own tools: versioning, object lock or write-once retention where available, separate accounts and credentials from the ones used by day-to-day administrators, and strict permission policies. An attacker who can delete your snapshots along with your cluster has defeated the whole plan.

What to include

Snapshots can include indices and data streams, and they can also include feature states, which capture internal system data such as security configuration and Kibana saved objects, plus cluster-wide settings. Decide deliberately what each policy includes. For DR, keeping a separate policy for configuration data, taken frequently and kept for a long time, is a good habit, since losing dashboards and users can hurt as much as losing documents.

Restoring, and how fast you can do it

The main limit is the speed of storage and network. Restoring in parallel across many nodes helps, and so does having many moderately sized shards rather than a few giant ones. The earlier example of about eleven hours for 20 terabytes shows why large data sets should not rely on snapshots alone if the RTO is short.

When minutes of downtime and seconds of data loss are the goal, replication is the workhorse. This chapter shows how to use it well.

Leader and follower

Cross-cluster replication works index by index. An index in the primary cluster is the leader. You create a follower index in the DR cluster that copies it. The follower is read-only: you cannot write to it directly, because it is defined entirely by the leader. This one-way rule avoids conflicts, and it is the reason the failover step, promotion, exists.

Auto-follow patterns

Creating a follower for every index by hand is not practical, especially with time-based indices that appear every day. An auto-follow pattern is a rule on the DR cluster that says “any new index on the remote cluster whose name matches this pattern should be followed automatically.” With data streams, which manage a series of hidden backing indices, the pattern covers the stream, and new backing indices are followed as they appear. Auto-follow is what makes replication low-maintenance.

Promotion: the failover step

When the DR region must take over, each follower index must be converted into a normal, writable index. For ordinary indices the steps are: pause the following, close the index, tell it to stop following (unfollow), and open it again. For replicated data streams, a dedicated promotion operation converts the stream so that it accepts new writes. Promotion is quick, since no data is copied, and it is usually scripted so that it can be run across hundreds of indices reliably.

Two-way designs without conflicts

Because one index follows one leader, active-active designs avoid conflicts by splitting who writes what. The European region writes to indices named for Europe, the American region writes to indices named for America, and each region has follower copies of the other’s indices. Searches that use a wildcard over both names see the whole picture in either region. No two regions ever write to the same index, so there is nothing to merge.

Limits and cautions

A few facts deserve attention. Replication needs a suitable license. It copies data and mappings, not templates, policies, pipelines, roles or dashboards. It faithfully copies mistakes. If a follower is disconnected for longer than the leader’s retention lease, it must be rebuilt from scratch. If a leader index is deleted, the follower stops following, and its data stays, which is helpful, but the deletion of documents inside a leader is copied. Lifecycle policies also deserve care: the rollover of a time-based index happens on the leader side, and the follower side needs matching policies that treat followers correctly, using the unfollow step where needed before they act on the data.

A perfect copy of your data is useless if nothing can write to it or read from it. This chapter covers the layers in front of and beside Elasticsearch.

The ingest layer is part of DR

Data reaches Elasticsearch through agents, collectors, pipelines and often a message queue. If all of these live in the primary region, a regional outage stops the flow of new data, however healthy the DR cluster is. A complete design therefore repeats the ingest layer in the DR region, and decides how events reach it. The popular pattern is a message queue in each region with the primary’s queue mirrored to the DR region. Producers write to the local queue; workers read from it and index into Elasticsearch. If the primary region fails, the mirrored queue already holds recent events and the DR workers take over.

flowchart LR
    SRC["Log and event sources"] --> Q1["Message queue in primary region"]
    Q1 --> W1["Ingest workers in primary region"]
    W1 --> ESP["Primary Elasticsearch cluster"]
    Q1 -->|"mirrored"| Q2["Message queue in DR region"]
    Q2 --> W2["Ingest workers in DR region - paused or active"]
    W2 --> ESD["DR Elasticsearch cluster"]
    ESP ==>|"cross-cluster replication"| ESD

Two ways to feed a standby

There are two basic designs. In the first, replication does the copying: only the primary receives writes, the DR cluster follows it, and the DR ingest workers stay paused until failover. It is simple and avoids duplicates. In the second, the ingest layer does the copying: every event is delivered to both regions and both clusters index it independently. This gives two truly independent copies, which is nice for active-active use, but it requires care, because the two copies are not guaranteed to be identical if delivery fails in one place.

Avoiding duplicates and gaps

Whenever events may be delivered twice, for example after a queue replays them during failover, duplicates can appear. The cure is to give every event a stable identifier, either one that is already in the data or one computed from its content, and to use that as the document identifier in Elasticsearch, so that a second delivery simply overwrites the first. Gaps are handled by keeping events in the queue long enough for workers to catch up. Think of numbering every letter that goes into a mailbox so that a re-sent letter is recognized rather than counted twice.

Routing readers: DNS and load balancers

Users and applications should reach Elasticsearch through names, never through fixed machine addresses. A global DNS service or a global load balancer answers with the region that is currently active. Health checks watch the primary and can be configured to change the answer, though for a whole-region switch many teams prefer a human decision, since the cost of a wrong switch is high. Keep the lifetime of DNS answers short, typically a minute or so, so that clients pick up the change quickly. Test that your client libraries respect short lifetimes: some cache the address for much longer.

Clients that can help themselves

Client applications can be configured with a list of endpoints, so that if the first is unreachable they try the next. This is useful as a second line of defense, but be careful with automatic fallback to the DR cluster while it is still read-only: writes would fail. A safer design lets clients read from both regions but write only where the runbook says writing is allowed.

Searching across both regions

Cross-cluster search lets a user in either region query both clusters at once. In an active-active design, a single search using a name pattern like “logs-*” on the local side and on the remote side returns the combined answer. In a disaster, if the primary is unreachable, the search can skip that cluster and return what the surviving region holds, marked as partial. This is a good way to keep dashboards useful during the crisis.

The neighbors of Elasticsearch

Elasticsearch rarely stands alone. Identity providers, secret stores, deployment pipelines, alerting channels and monitoring systems all sit around it. A DR plan that fails over the cluster but leaves the login system in the dead region cannot be used by anyone. List every dependency and give each one its own answer for “what happens if its region is gone?”

This is the moment of truth. Here is the sequence of steps, why each step exists, and how to return safely afterward.

flowchart TD
    S1["Detect - alarms and health checks fire"] --> S2["Declare - incident commander decides to fail over"]
    S2 --> S3["Fence - stop writes to the old primary"]
    S3 --> S4["Record replication position and the data loss window"]
    S4 --> S5["Promote followers to regular indices"]
    S5 --> S6["Start ingest workers in the DR region"]
    S6 --> S7["Switch traffic with DNS or the load balancer"]
    S7 --> S8["Verify searches, dashboards and alerts"]
    S8 --> S9["Communicate and start failback planning"]

Step by step

Human decision or automatic?

Automation is wonderful for the steps of a failover: promoting a hundred indices by script is faster and more reliable than by hand. But most teams keep the decision to fail over in human hands. Automatic region-level failover can trigger on false alarms, such as a short network blip between monitors and the region, and switching back and forth is costly. A good compromise is to automate everything except the decision, and to make the decision as easy as pressing one clearly labeled button.

An example timeline

Failback: the careful return

When the original region is healthy again, resist the urge to switch back at once. First, decide what to do with any data the old primary holds that never reached the DR region. Then rebuild the old primary’s indices as followers of the now-active cluster, either by deleting the stale indices and creating fresh followers, or by restoring from a snapshot and following from there. Let them catch up fully and verify that counts match. Only then, in a planned and quiet window, repeat the same runbook in the opposite direction: fence, promote, switch, verify. Some teams decide never to fail back at all, and simply treat the former DR region as the new primary and rebuild a new DR region elsewhere, which can be simpler and safer.

A second region costs money and adds load. This chapter shows where the costs come from and how to keep them sensible.

The bandwidth question

Replication sends every change across regions, so the link must carry your write volume. A simple calculation gives a feel. If your cluster receives 500 gigabytes of new data per day, that is 500,000 megabytes spread over 86,400 seconds, or about 5.8 megabytes per second on average. Real traffic is bursty, so plan for several times that at peak. Compression on the link can help. Cloud providers often charge for data leaving a region, so this steady flow becomes a line on the bill, and it is worth estimating before you decide what to replicate.

The size of the DR cluster

The most important cost decision is how big the DR cluster should be. A standby that matches production is safe and expensive. A leaner standby is cheaper but must grow after failover, which takes time and requires that capacity be available in the DR region exactly when everybody else is also trying to get it. Common compromises include the following.

Effects on the primary

Replication adds work to the leader: it must serve the reads that followers request, and it must hold on to history for them. For most systems this extra load is small, but on very busy clusters it is worth watching. Keep shard counts reasonable, since thousands of tiny shards make replication slower to set up and slower to recover. Snapshots also read from the primary’s storage. Schedule heavy snapshot windows away from your busiest hours, and use fast object storage in the same region for the primary’s repository.

Capacity planning in plain steps

First, decide the target: at failover, the DR region must handle X percent of production traffic. Second, translate X into nodes, storage and memory using your normal sizing rules. Third, decide how much of that must exist before the disaster and how much can be added afterward, and multiply the scale-up time by your RTO budget to see if it fits. Fourth, add up costs for compute, storage, transfer and snapshots, and compare with the cost of downtime. Repeat the exercise every time your data grows by half.

Every DR design is a bundle of trade-offs. Knowing them helps you defend your choices when the budget is discussed, and when the pager rings.

Trade-off one: loss versus cost

Because replication is asynchronous, the DR copy is always a little behind. Making it synchronous, so that no write is acknowledged until both regions have it, would remove data loss but would slow every write by the delay between regions and would make the primary depend on the DR region being available. Elasticsearch chooses to be fast and available, and accepts a small loss window. If your data cannot tolerate even seconds of loss, you must solve it at the source, for example by keeping events in a durable, replicated queue until both regions have confirmed them.

Trade-off two: speed of recovery versus complexity

Each rung up the ladder recovers faster and is harder to run. Active-active gives the fastest recovery but demands the most from your engineers. Warm standby is a sweet spot for many: minutes of downtime, with a manageable design. Choose the simplest design that meets your numbers, because a simple plan is one that people will actually follow under stress.

Trade-off three: automation versus control

Fully automatic failover is fast but risks acting on false alarms. Fully manual failover is safe but slow. Automating the steps while keeping the decision human tends to give the best balance.

Trade-off four: consistency versus availability

In distributed systems, when the network between sites fails, you cannot have both perfect agreement and full availability. Elasticsearch’s multi-region tools favor availability and eventual agreement. That is why your design must plan for what happens to the small set of writes that had not yet crossed the link.

Comparing the tools

These tools work best together. Most mature designs use all four in different roles, with each covering the weakness of another.

A second region doubles the number of places where your data lives, and therefore the number of places to protect. Security must be designed into DR from the start.

Protect the link between regions

Replication and snapshot traffic carries your complete data. The connection between clusters must be encrypted and authenticated. Elasticsearch supports certificate-based trust between clusters, and, in newer versions, a narrower model based on cross-cluster API keys that can be limited to the replication or search access you actually need. Prefer the narrow model where your version and license allow it, keep the keys in the protected secure settings store, rotate them on a schedule, and set expiry dates. Prefer private connections between regions, such as private links or peered networks, over routes across the public internet.

Encryption at rest and in the repository

Data sits on disks in both regions and in the snapshot repositories. Use encrypted volumes for the cluster’s disks, and the storage provider’s encryption with keys that you control for the repositories. Remember that encryption keys are part of DR too: if the DR region cannot use the key that protects the snapshots, the snapshots are unreadable. Keep keys available in both regions, and test their use during drills.

Separate the powers

Give the DR machinery its own identity, with the narrow rights it needs: reading from the primary, writing to the DR cluster, managing snapshots. Keep the credentials for snapshot deletion apart from the credentials used for daily work, and place the backup repository in a separate cloud account or project where possible, so that a stolen everyday administrator login cannot erase the safety net. For emergencies, prepare a break-glass account: a highly privileged login kept sealed, monitored and used only when necessary, so that nobody is locked out at the worst moment.

People and permissions must exist in both regions

Users, roles and role mappings that live inside the primary cluster are not automatically replicated. Keep them as code, or connect both clusters to the same external identity provider, and make sure that identity provider itself survives a regional failure. Nothing is more frustrating than a fully restored cluster that nobody can log in to.

Compliance and where data may live

Replicating data creates a copy in another place, and some laws and contracts care about where copies live. If your data must stay within a country or a legal zone, choose a DR region inside that boundary, or replicate only the data that is allowed to travel. Document which data goes where, and keep evidence of your DR drills, since auditors often ask to see them. Retention rules and legal holds also matter: snapshots that are kept too long can violate deletion promises, while snapshots that are deleted too soon can violate preservation duties.

A DR system that has silently stopped working is worse than none. This chapter shows what to watch, and how to prove the whole plan with drills.

What to watch

Elasticsearch gives you statistics for replication that show, for each follower, how many operations it is behind and how long ago it last read from the leader. That lag is your live RPO, so it is the single most valuable number. Snapshot lifecycle policies record the time and result of their last run and of the last failure. Repositories can be verified to check that the cluster can still read and write them. The remote cluster connection reports whether it is connected. All of these can be checked by scripts and alerts.

Watch from outside

Monitoring that runs inside the primary region dies with the primary region. Put at least the key checks, and the alert delivery itself, in a third place: the DR region, another cloud, or an independent monitoring service. A smoke detector that shares a fate with the house it protects only reports the fire once it is too late.

Logs and evidence

Keep the logs of every drill and every real event: who did what, when, and what the results were. They help you improve the runbook, satisfy auditors and train new engineers. Cluster logs, audit logs and the notes from the incident channel together tell the complete story.

Testing: the only proof

A plan is proven only by trying it. There are several levels of drill, and a healthy program uses all of them.

Netflix has publicly described practicing the evacuation of an entire region as a routine exercise, which is a good model of the attitude: if you do it often, it stops being frightening. You do not need that scale to benefit. Even a small team can hold a tabletop exercise every quarter and a real failover of one service twice a year.

Patterns are recipes that experienced teams keep repeating because they work. Anti-patterns are the traps those same teams learned to avoid.

Patterns worth copying

Traps to avoid

A short, practical checklist drawn from careful teams, followed by the mistakes that trip up nearly everyone once.

Common mistakes and their cures

Here is how the same ideas adapt to very different industries. Named companies below are used as familiar illustrations of the kind of scale and problem involved, not as claims about their internal systems.

A complete walk-through: the product search index

To make everything concrete, here is how a team might build DR for a product search index, from a blank page to a proven design.

After the launch, the story from the start of this article changes. At 2:14 a.m. the region goes dark, the on-call engineer opens a runbook she has practiced, and a little over half an hour later customers are searching again.

Short answers to the questions that beginners ask most often.

The whole article in one place. If you remember only this box, you still know the essentials.