Start here. By the end of this chapter you will know what a shard copy is, what the setting controls, and why it is often misunderstood.

Elasticsearch splits every index into pieces called shards. To survive the loss of a machine, each shard exists in several copies. One copy is the primary, and the others are replicas. Every write goes first to the primary and is then passed on to the replicas, so that all the copies hold the same data.

Write consistency tuning is the practice of choosing how many of those copies must be active, meaning up and ready, before Elasticsearch begins a write. The setting that does this is called wait_for_active_shards, and this article writes it simply as wait_for_active_shards. You can give it a number, such as 2, or the word “all.” By default the number is 1, which means that only the primary copy has to be active. If you raise it to 2, at least one replica must also be active. If you set it to “all,” every copy that the index is supposed to have must be active.

A precondition, not a promise

The most important thing to understand about this setting is what kind of rule it is. It is a precondition: a check made before the write starts. It says, “I will not begin unless enough copies are ready.” It is not a guarantee that the write reached that many copies by the time it finished. Once a write is underway, a replica can still fail, and the write can still succeed on the copies that remain. Later chapters explain how Elasticsearch handles that case, and why the response to every write tells you how many copies really received it.

Where you meet it

You will meet the setting in three places. You can put it on an individual write request, such as indexing a document, sending a bulk request, updating or deleting. You can set it as a default for a whole index, so that every write to that index follows it unless a request says otherwise. And you will meet a related use when you create, open, roll over or resize an index, where the same idea means “do not return until enough shard copies have started.”

A short history

Older versions of Elasticsearch, before the 5.0 release in 2016, had a similar setting named consistency. It accepted the words “one,” “quorum” and “all,” and it was borrowed from the language of distributed databases. Over time the Elasticsearch team found the name misleading, because it suggested a stronger guarantee than the system actually gave, and they replaced it with the more honest wait_for_active_shards in the 5.0 series. In the same period, and in later versions, the replication machinery itself was strengthened with sequence numbers and checkpoints, which let the system repair copies much more reliably after failures. Today the setting is a small, well-defined control on top of a robust replication design.

A healthy cluster has all its copies ready. Real clusters are not always healthy. To see why this setting exists, look at what happens to writes when copies go missing.

Copies go missing all the time

A shard copy can be unavailable for many ordinary reasons. A server is restarted for patching. A cloud provider replaces a machine. A network cable is unplugged, or a switch has a bad hour. A disk fills up and Elasticsearch stops using it. A node runs out of memory and is restarted by the operating system. A whole availability zone loses power. In each case, some shard copies disappear from the cluster for seconds, minutes or hours, and the cluster starts rebuilding them elsewhere.

What the default does in those moments

With the default setting of 1, a write needs only the primary copy. If a replica is missing, the write goes ahead on the primary alone and succeeds. The client sees a normal success. The cluster status turns yellow, which means “all primaries are fine but some replicas are missing,” but the application usually never looks at the cluster status. During this window the newest data has no backup.

This is a sensible default for many workloads, because it keeps writes flowing. A logging system that receives a million events per minute would rather keep accepting them than stop during a patch. But for some data, a single-copy period is a risk worth avoiding: financial records, orders, medical events, anything expensive or impossible to recreate.

The choice: refuse or accept

The heart of the matter is a choice between two values. Availability means that the system keeps working. Safety means that the system does not accept data unless it can protect it properly. When copies are missing, you cannot have both fully. The setting lets you decide, index by index or request by request, where to stand.

Why the confusion?

Many people assume that a setting with the word “consistency” in its history must guarantee that every write reaches a quorum, as some databases do. Elasticsearch does not work that way. It does not vote on each write. Its design is that the primary applies the write, sends it to all the copies it currently knows to be in step, and only then replies. Which copies count as “in step” is tracked by the cluster’s master node. The setting wait_for_active_shards does not change that design. It only adds a gate at the start. This distinction is the source of most misunderstandings, and it is why this article spends time on the internal working in Chapter 5.

The rest of this article gives you the tools to decide, for each of your indices, which side of that line is right.

A dozen words unlock the whole subject. Each one is explained with what it is, why it exists, where you meet it, and a picture from everyday life.

Shard and shard copy

An index is divided into shards, and each shard is a self-contained slice of the data that can be stored on one node. To protect against failure, each shard exists in more than one copy. A copy is a full, independent version of that slice. Think of a shard as a chapter of a book, and copies as photocopies of that chapter kept in different rooms.

Primary and replica

Among the copies of a shard, exactly one is the primary. It is the one that receives every write first. The others are replicas. They receive the writes from the primary and serve searches too. If the primary is lost, one of the replicas is promoted to become the new primary. The primary is the editor-in-chief of the chapter; replicas are the copy editors who keep identical versions.

Number of replicas

The index setting number_of_replicas decides how many replica copies each shard should have. With 1 replica, each shard has 2 copies in total. With 2 replicas, 3 copies. This number is the target, the number the cluster tries to reach. The number of copies actually alive at a given moment may be lower.

Active copy

A copy is active when it is assigned to a node and has finished starting, so it is ready for work. A copy that is unassigned, or that is still copying its data from another node in a process called recovery, is not active yet. The setting wait_for_active_shards counts active copies.

The in-sync set

The master node of the cluster keeps a list, for each shard, of the copies that are known to be fully up to date. This list is called the in-sync set. When the primary applies a write, it sends the operation to every copy in the in-sync set and waits for their answers. A copy that fails to keep up is removed from the set, and stops counting until it is repaired. The in-sync set is like the list of committee members who have read all the latest minutes.

Sequence numbers and checkpoints

Every write to a shard receives a growing sequence number, like a numbered page in a ledger. Each copy tracks how far it has caught up, and the cluster tracks the point up to which all in-sync copies have everything, known as the global checkpoint. These numbers let a lagging copy catch up by replaying only what it missed, and let the cluster decide safely which copy can become primary after a failure.

Acknowledgment

When a write is finished, the client receives an acknowledgment, the response that says “done.” In Elasticsearch, this response arrives only after the primary and all in-sync replicas have applied the write, and, with the default durability rule, have safely recorded it to disk. So an acknowledgment is a genuine statement about safety, but it counts the copies that were in sync at that moment, which may be fewer than the target.

The timeout

If the required number of active copies is not available, the write does not fail at once. It waits, hoping that copies will become active, for a period set by the timeout parameter, which is one minute by default. If the copies are still not ready when time runs out, the write fails with an error saying that there are not enough active shard copies. It is like a shop assistant who says “please wait a minute while I find the second key” and, if none appears, says “sorry, I cannot help.”

The three kinds of values

The setting accepts three kinds of values. A number from 1 up to the total number of copies (the primary plus the replicas) means “at least this many copies must be active.” The word all means “every copy the index should have.” The default of 1 means “just the primary.” A number larger than the total number of copies is rejected as impossible.

Now let us zoom out and look at the machinery: which parts take part in a write, where each one lives, and where the “enough active copies” check fits in.

The setting wait_for_active_shards is not a separate service. It is a check built into the normal path that every write follows. The diagram below shows that path, with the gate that the setting controls drawn in as a decision.

flowchart LR
    CL["Client"] --> CO["Coordinating node"]
    CO --> GATE{"Enough active shard copies?"}
    GATE -->|"Yes"| P["Primary shard on node A"]
    GATE -->|"No - wait up to the timeout"| ERR["Error - not enough active copies"]
    P --> R1["Replica shard on node B"]
    P --> R2["Replica shard on node C"]
    R1 -->|"acknowledges"| P
    R2 -->|"acknowledges"| P
    P --> RESP["Response with successful and total copies"]
    M["Master node - keeps routing table and in-sync lists"] -.-> GATE
    M -.-> P

Walking through the diagram

A client, such as an application or a log shipper, sends a write to any node of the cluster. That node becomes the coordinating node for the request: it works out which shard the document belongs to and finds where that shard’s primary lives. Before the write is applied, the cluster looks at the gate: are enough copies of that shard active? The answer comes from the cluster’s routing table, a map of every shard copy and its state, which the master node maintains and shares with all nodes. If enough copies are active, the write goes to the primary shard. If not, the write waits, and if the copies do not appear before the timeout, it ends with an error.

After passing the gate, the primary applies the write and sends it to the replicas on other nodes. Each replica applies it and answers. When all the copies in the in-sync set have answered, the primary replies, and the response travels back to the client, carrying counts of how many copies took part. The master node does not handle the write itself. It only maintains the map and the in-sync lists that the other parts rely on.

The components in detail

Where copies should live

The setting counts copies, but copies are only useful if they are on different machines. Elasticsearch never puts a replica on the same node as its primary. For stronger safety you can also tell it about your zones or racks, using allocation awareness, so that copies of the same shard are spread across them. Requiring two active copies is much more meaningful when the two copies cannot be lost by a single power failure.

This is the engine room. You do not need to be an expert in distributed systems, because every idea comes with a picture.

sequenceDiagram
    participant C as Client
    participant N as Coordinating node
    participant P as Primary shard
    participant R as Replica shard
    C->>N: Write with wait_for_active_shards
    Note over N,P: Before the write starts, the required number of active copies is checked
    N->>P: Forward the write to the primary
    P->>P: Validate, apply and number the operation
    P->>R: Send the operation to each in-sync replica
    R-->>P: Apply and acknowledge
    P-->>N: Result with counts of copies
    N-->>C: Response

Step one: the check

When the write arrives, the cluster compares two numbers: how many copies of the target shard are active, and how many your setting requires. If the active count is high enough, the write proceeds at once. If not, the write does not fail immediately. Instead it waits, and each time the cluster’s picture of the shard copies changes, for example because a replica finished starting, the check is repeated. This continues until the requirement is met or the timeout expires. In the second case the write fails with an error, and importantly, nothing has been written for that shard.

Step two: the primary applies the write

The primary validates the operation, applies it to its own copy of the data, and gives it the next sequence number. It also records the operation in its transaction log, a file that lets the shard recover its recent changes after a crash. Because every operation passes through the primary, the primary is the single place where the order of writes is decided, which is what keeps all copies consistent with each other.

Step three: forward to the in-sync copies

The primary sends the operation, in parallel, to every copy in the in-sync set. This is the crucial point, and the source of the most common misunderstanding. The replication does not stop at the number you gave to wait_for_active_shards. If three replicas are in sync, the write goes to all three, even if you asked for only two active copies. The setting is a gate at the start, not a limit on the copying.

Step four: wait for the answers

Each replica applies the operation, records it in its own transaction log, and answers. With the default durability setting, each copy flushes the log to disk before answering, so an acknowledged write has survived on every copy that answered. The primary waits for all in-sync copies.

Step five: handle a copy that fails

What if a replica does not answer, or reports an error? The primary does not fail the write. Instead it tells the master node that this copy has fallen out of step. The master removes the copy from the in-sync set, and the copy is marked as failed and scheduled for repair. Only after that does the primary acknowledge the write to the client. The response then reports that one copy failed. This is how a write can succeed even though a replica failed during it, and it is exactly the case that the pre-write check cannot prevent.

Step six: the response

The response to a successful write includes a small section describing the shards. It has three numbers: the total number of copies that the shard is supposed to have, the number of successful copies that applied the write, and the number of failed copies. A response with a total of 3, three successful and none failed means all is well. A response with total 3, successful 2, failed 1 means that one replica errored during the write and is being repaired. A response with total 3, successful 1 and failed 0 means that two copies were not active at all when the write ran, so they were never even tried. Reading these numbers is the only way to see how well protected a particular write was.

Fencing off old primaries

What if the primary itself is lost, and a replica is promoted? Each time a new primary is chosen, a counter called the primary term goes up by one, and every operation carries it. Copies refuse operations that come with an older term. So if a slow or disconnected old primary wakes up and tries to send operations, the copies reject them. This fencing prevents two primaries from both accepting writes for the same shard and creating conflicting histories.

The guarantee, in plain words

Putting it together: when you receive a success response, the write is safely recorded on the primary and on every copy that was in sync at that moment. If the primary is lost afterward, the cluster promotes a copy that is in the in-sync set, so it has the write. The write can only be lost if all the copies that held it are lost together, for example, a primary with no active replicas at the time of the write. And that is exactly the exposure that wait_for_active_shards lets you shrink: by requiring at least two active copies, you make sure that when the primary applied the write, at least one other copy was ready to receive it.

Every write passes through a short life: it arrives, sometimes waits, is applied, replicated and acknowledged, or times out. Knowing each stage helps you read errors and slow responses.

stateDiagram-v2
    [*] --> Received
    Received --> Waiting : not enough active copies
    Received --> Applying : enough active copies
    Waiting --> Applying : copies become active
    Waiting --> TimedOut : timeout reached
    Applying --> Replicating : primary applied the operation
    Replicating --> Acknowledged : in-sync copies confirmed
    TimedOut --> [*]
    Acknowledged --> [*]

Stage one: received

The coordinating node accepts the request, works out the target shard and looks at the requirement: either the value given in the request, or, if none was given, the default stored in the index settings. Then it checks the routing table.

Stage two: waiting, if needed

If the requirement is not yet met, the write is parked. It costs almost nothing while it waits, since it holds no locks on data, but the client is waiting too. The write leaves this state in one of two ways: the missing copies become active, and the write moves on, or the timeout ends, and the write fails.

Stage three: applying and replicating

Once through the gate, the primary applies the write and forwards it to the in-sync copies. From this point the write is committed to happening. The setting has no further role.

Stage four: acknowledged

The response returns with the shard counts. In the normal case, all copies succeeded. In a degraded case, some copies are missing or failed, and the counts tell you.

The timeout ending

When the timeout expires, the write fails with an error that says there are not enough active copies, and states how many were needed and how many were found. The status code of the response is the kind used for “service unavailable,” which tells your client that a retry may succeed later. Because the check happens before the write starts, a failure of this kind means the write did not begin on that shard, so retrying it is safe from the point of view of duplicates. Timeouts on the client side are different: if your own client gives up waiting before the cluster answers, you cannot know whether the write happened, and you should make retries safe by using explicit document identifiers.

Bulk requests and many shards

A bulk request contains many writes, aimed at many different shards. The requirement is checked per shard. If the shards of the target index have enough active copies for some documents and not for others, the documents for the healthy shards succeed and those for the unhealthy shards fail individually, each with its own error inside the bulk response. This is why you must always read the per-item results of a bulk response, and not just the overall status.

Before going deeper, here is the route you follow when you decide to tune this setting for the first time. Knowing the road makes the later chapters easier to follow.

The setting itself is tiny. The thinking around it is what matters. Here is the journey for a beginner who wants to protect an index of customer orders.

Before you begin

Check four things. First, how many replicas does the index have? If it has none, then there is nothing to wait for, and the first task is to add a replica. Second, how many data nodes and zones does the cluster have? Copies must live on different nodes, so you need at least as many nodes as copies. Third, what is the current health of the cluster? If it is already yellow because replicas cannot be placed, a stricter setting will simply block writes. Fourth, what happens in your application when a write fails? If it just drops the data, a stricter setting can cause loss instead of preventing it.

The setting accepts only a few kinds of values. This chapter shows exactly what each one means, how to calculate the right number, and what happens when the number is impossible.

Start with the arithmetic

Every shard has one primary and some number of replicas, so the total number of copies is always one plus the number of replicas. With 0 replicas, a shard has 1 copy. With 1 replica, 2 copies. With 2 replicas, 3 copies, and so on. The value you give to wait_for_active_shards must be between 1 and that total. Anything higher is impossible to satisfy and is refused.

The default: 1

The default requires only the primary. It is the most available choice: as long as the primary exists, writes go ahead, however many replicas are missing. It matches the way most search and log workloads are used, and it is the right choice for the great majority of indices.

A number: at least this many active

Choosing a number such as 2 means “the primary and at least one replica must be active.” This is the most common way to tighten the rule. Its value depends on how many replicas you have. With exactly one replica, 2 is the same as “all,” and any missing replica blocks writes. With two replicas, 2 lets you lose one replica and keep working, while still guaranteeing that every write finds at least one partner copy ready.

The word “all”

The value all means that the primary and every replica must be active. It gives the strongest protection at the start of a write, and it is also the most fragile: the loss of any single copy, even for a routine restart, blocks writes on that shard until the copy is back. Use it only when you truly need every copy present, and be ready to see write errors during each maintenance event.

The majority pattern

Many distributed systems use a “majority” or “quorum,” meaning more than half of the copies. You can imitate this pattern by choosing a number equal to the total divided by two, rounded down, plus one. With three copies, that is 2. With five, 3. This gives a good balance: it tolerates the loss of a minority of copies, and guarantees that each write starts with a majority ready. Remember, though, that Elasticsearch does not vote on writes. The pattern only sets the entry rule, and the replication itself still goes to all in-sync copies.

Impossible values

If you ask for a number larger than the total number of copies, for example 3 on an index with one replica, Elasticsearch rejects the request with a validation error right away, rather than waiting for the timeout. The message says that the number of active copies to wait for cannot be greater than the number of copies of the shard. Negative numbers are rejected too. A related trap is asking for “all” on an index whose replicas can never be placed, for example one replica on a cluster with a single data node. Then the replica is permanently unassigned and every write will wait until the timeout and fail.

Interaction with changing replica counts

The value “all” is not a fixed number. It follows the index’s configured number of replicas at the moment of the write. If someone changes the replica count, what “all” requires changes with it. Some indices use automatic expansion of replicas, which grows or shrinks the count with the number of nodes, and there “all” moves too. If you want a fixed rule, use a number.

Zero for index-level operations

For some operations that create or open indices, and depending on the version, a value of zero may be accepted, meaning “do not wait for any shard at all.” This is rarely wise, because the response would return before anything is ready to use. Check your version’s documentation before using it.

The same rule can be given in several places. Knowing which one wins, and which one is best, saves many surprises.

On the individual request

Every write operation accepts the parameter. You can put it on a single document write, an update, a delete, or a bulk request, along with the timeout. When you do, it applies to that request alone. This is the most flexible option, and it suits special cases: a critical write that needs a stricter rule than usual, or a big backfill that should not be blocked by a single missing replica. The drawback is that every writer must remember to send it, which is easy to forget when many tools and teams write to the same index.

As an index default

The second place is the index itself. Each index has a setting, named index.write.wait_for_active_shards, that acts as the default for all writes to that index. It is a dynamic setting, so you can change it on a live index and it takes effect immediately. If a request does not carry its own value, the index default applies. If the index does not have one either, the global default of 1 applies. This is the best place for a rule that should always hold, since it works even for tools that cannot send the parameter, such as many log shippers and agents.

In index templates

Time-based data creates new indices constantly: one per day, or a new backing index whenever a data stream rolls over. Setting the rule on each new index by hand would be impossible. Instead, put the setting into the index template that matches those names. Every new index then arrives with the rule already in place. For data streams, the template of the stream carries the setting to each backing index. Do not forget to update the template when you change your policy, since the older indices keep the value they were born with until you change them.

For tools that cannot send parameters

Many ingestion tools send their writes without any option for this parameter. That is not a problem if you use the index default, because the rule lives in the cluster, not in the tool. This is a strong argument for setting the rule on the index or the template, and using the request parameter only for exceptions.

Which operations honour it

The timeout goes with it

Each of these operations has a companion timeout, the time it is willing to wait for the copies. The default is one minute. Shorten it if your users cannot wait, and lengthen it if you are willing to wait through a restart. A stricter requirement with a very short timeout produces many quick failures during any disturbance. A stricter requirement with a longer timeout produces slow writes but fewer failures. Choose the pair together.

The word “consistency” is used for several different ideas. Mixing them up is the most common reason people tune the wrong knob.

When someone says “I want my writes to be consistent,” they might mean any of four quite different things. Each is controlled by a different setting. Separating them will save you a lot of confusion, and it will show exactly where wait_for_active_shards fits.

Meaning one: enough copies ready

This is the topic of this article. It asks: how many copies must be alive before I accept a write? It controls the protection at the moment the write begins. The knob is wait_for_active_shards, together with the number of replicas.

Meaning two: is it safely on disk?

This asks: when the response says “done,” has the data been saved on disk, so it survives a crash of the machine? Elasticsearch records each operation in a transaction log on every copy. By default, the log is flushed to disk before each request is acknowledged, which is called request durability. You can relax this so that the log is flushed on a timer, by default every five seconds. That makes writes faster but means that a crash might lose the last few seconds of acknowledged writes on that copy. The knob here is the transaction log durability setting and its sync interval. Even with the relaxed setting, other copies still hold the data, which is another reason for replicas.

Meaning three: can a search see it yet?

This asks: after my write succeeds, will a search find the document immediately? Elasticsearch is called near real-time: a new document becomes searchable only after the next refresh, which by default happens about once per second. Fetching a document by its identifier is immediate, but searching is not. The knobs are the refresh interval of the index and the refresh option on a write, which can force a refresh or wait for the next one. This is a question about visibility and has nothing to do with the number of copies.

Meaning four: what if two writers collide?

This asks: if two clients update the same document at nearly the same time, what happens? Elasticsearch uses optimistic concurrency control: each document carries a sequence number and a primary term, and an update can say “apply only if the document is still at this sequence number.” If someone else changed it first, the update is refused, and the client can re-read and try again. The knobs are the sequence number and primary term conditions, or version checks. It prevents lost updates between clients, and is unrelated to how many shard copies are alive.

Why cluster health is another thing again

The cluster health interface has its own option, wait_for_status, which makes the call wait until the whole cluster reaches a given colour: green, yellow or red. It looks similar but works at a different level. It waits on the whole cluster, and is used by scripts that want to hold until the cluster has settled. The write setting works per shard, per operation.

Theory becomes real when things break. This chapter walks through the common failures and shows how each value of the setting behaves.

flowchart TD
    F["Node holding a replica leaves the cluster"] --> D["Replica becomes unassigned"]
    D --> W{"Required active copies for writes"}
    W -->|"1"| A["Writes continue on the primary alone - single copy risk"]
    W -->|"2 with one replica"| B["Writes wait, then fail after the timeout"]
    W -->|"2 with two replicas"| C["Writes continue - primary plus one replica still active"]
    A --> R["Replica rebuilt on a node - redundancy restored"]
    B --> R
    C --> R

Scenario one: a replica’s node restarts

A routine restart takes a node offline for about three minutes. The replicas on that node disappear. When a node leaves, Elasticsearch does not rush to rebuild its shards elsewhere, because most nodes come back quickly and rebuilding is expensive. It waits for a delay, one minute by default, before reallocating. During this time the affected replicas are unassigned. With the default rule, writes continue on the primaries alone. With a requirement of 2 and one replica, writes to the affected shards wait, and fail if the timeout of one minute is reached before the node returns. With a requirement of 2 and two replicas, the second replica keeps things going.

Scenario two: a node is lost for good

A machine dies and will not return. If it held replicas, they are rebuilt on other nodes after the delay. The rebuild takes time proportional to the amount of data, and until it finishes, the same trade-off applies. If it held primaries, a replica on another node is promoted to primary, within moments, and the shard continues, though with fewer copies until the rebuild completes. This is exactly the case where having required at least two copies pays off: the promoted primary is guaranteed to hold every acknowledged write.

Scenario three: a whole zone disappears

If the cluster is spread over three zones with zone-aware placement, losing one zone removes about a third of the copies of every shard. With two replicas, each shard still has at least two copies, and a requirement of 2 is still satisfied. With one replica and a requirement of 2, half the shards that had a copy in that zone now lack their second copy, and writes to those shards will wait. This is a clear example of why the requirement and the number of replicas should be chosen together.

Scenario four: a network partition

A network fault splits the cluster in two. The side that keeps a majority of master-eligible nodes continues; the other side cannot elect a master and stops accepting writes for its shards. Shard copies on the minority side are treated as lost by the majority side, and the majority promotes replacements. The fencing described earlier prevents old primaries on the minority side from causing damage when the partition heals. The requirement for active copies is judged by the side that has a master, using its own view.

Scenario five: the primary is lost

When a primary fails, writes to that shard cannot proceed until a replica has been promoted. Writes wait, at the gate or in the queue, and resume as soon as the promotion is complete, usually within seconds. If there is no in-sync replica to promote, the shard stays red and writes fail once the timeout passes, whatever value of the setting you use. No setting can create a primary out of nothing.

Scenario six: a rolling upgrade

During an upgrade, nodes are restarted one by one. At each restart, the replicas on that node are briefly missing. With a strict requirement and few replicas, each restart will cause a short period of write errors. With enough replicas, the requirement can stay satisfied throughout. Many teams temporarily relax the rule during upgrades, or add a replica before the upgrade begins.

There is no single right value. There is a right value for each kind of data. This chapter gives a simple method to find it.

flowchart TD
    Q["How valuable is each write?"] --> A{"Can you afford to lose a recent write if a node is lost?"}
    A -->|"Yes"| B["Keep the default of 1"]
    A -->|"No"| C{"Does the index have at least two replicas?"}
    C -->|"Yes"| D["Use 2 - primary plus one replica, and one replica may be down"]
    C -->|"No"| E{"Can writes pause during restarts and failures?"}
    E -->|"Yes"| F["Use 2 with one replica - same as all"]
    E -->|"No"| G["Add a replica first, then use 2"]

Step one: decide what a lost write costs

Ask a plain question: if the newest few minutes of writes vanished in a rare double failure, what would happen? For debug logs, nothing. For product page views, a small dent in a chart. For customer orders or payment records, angry customers, refunds and possibly legal trouble. The answer sets how much inconvenience you should accept in order to reduce the chance.

Step two: decide what a refused write costs

Now ask the opposite question: if writes were refused for a few minutes during a restart, what would happen? If a durable queue sits in front of Elasticsearch and can hold the events until the cluster is ready, a pause costs almost nothing. If users are typing data into a form and each refused write shows an error page, a pause is expensive. This second question is often forgotten, and it is just as important as the first.

Step three: fix the number of replicas first

The setting is limited by the number of replicas, so decide that first. With zero replicas, there is no redundancy, and nothing to wait for. With one replica, you get single-failure protection, but a strict rule makes every replica outage block writes. With two replicas, you can be strict and still tolerate one replica being unavailable. For truly valuable data, two replicas plus a requirement of 2 is a well-proven combination. It costs an extra copy of storage, and it buys resilience through restarts, upgrades and single failures.

Step four: pick the value

Step five: pair it with a timeout and a plan

A stricter rule needs a partner: a timeout that fits the kind of disturbance you expect, and a client that knows what to do when a write is refused. For batch pipelines, a longer timeout and a queue are natural. For interactive applications, a short timeout and a clear message for the user may be right.

Step six: test and revisit

Try the choice in a rehearsal, as described earlier, and revisit it whenever the cluster changes size, when you add zones, or when the business value of the data changes. A rule chosen when the cluster had three nodes may not fit a cluster of thirty.

The same setting also appears on operations that create or restore indices. Here the question is not “may this write begin?” but “when should I say the index is ready?”

Creating an index

When you create an index, Elasticsearch adds it to the cluster state and then starts allocating its shards to nodes. The create call has to decide when to return. By default it waits until the primary of each shard has started, then reports success. If you give a larger requirement, it waits until that many copies of each shard have started. When the timeout ends before that has happened, the call returns an answer that says the index was created but not yet ready. Two flags appear in the response. The first, acknowledged, says that the master accepted the creation. The second, shards_acknowledged, says whether the required number of shard copies started in time. A response with the first true and the second false is not an error: the index exists and is being prepared, it just was not ready in time.

flowchart LR
    CREATE["Create index request"] --> MASTER["Master adds the index to the cluster state"]
    MASTER --> ALLOC["Shard copies are allocated and started"]
    ALLOC --> CHK{"Required copies started before the timeout?"}
    CHK -->|"Yes"| OK["acknowledged true and shards_acknowledged true"]
    CHK -->|"No"| PART["acknowledged true and shards_acknowledged false"]

Why this matters to automation

Scripts often create an index and immediately start writing. With the default of 1, when the create call returns, every primary is started, so the first writes will succeed. If your script asks for a stricter number on creation and the cluster cannot start that many copies in time, the script will see the “not ready” flag, and should decide whether to wait, retry or stop. Ignoring that flag is a common source of confusing failures right after a deployment.

Opening a closed index

A closed index has no active shards. Opening it makes the cluster start them again, and the same requirement decides when the call returns. Opening a big index with many shards can take time, so the timeout and the requirement are worth considering, especially in scripted maintenance.

Rollover, shrink, split and clone

Operations that create a new index from an old one, such as rollover for time-based data, or resizing operations such as shrink, split and clone, accept the same parameter. It controls how long they wait for the new index’s shard copies to start before returning. For rollover in particular, this matters for data streams: after a rollover, new writes go to the new backing index, and the first writes will meet the requirement of that new index. Make sure the template of the stream gives the new index the same rule you gave the old one.

Templates keep the rule alive

Because time-based data creates new indices all the time, only a template can guarantee that every new index carries the right default. Check that your lifecycle policies, rollover actions and data stream templates all produce indices with the setting you expect. A quick review of the settings of the newest index of each family, after a rollover, is a cheap way to be sure.

A stricter rule has costs in speed, capacity and operations. This chapter shows where they appear and how to keep them small.

The check itself is cheap

In a healthy cluster, the gate takes almost no time: it is a comparison of two numbers using information every node already holds. Raising the requirement from 1 to 2 does not make healthy writes slower. This surprises many people, and it follows from the design: the primary already sends every write to all in-sync replicas and waits for them, whatever the requirement.

What does slow writes down

Write latency is driven by the number of in-sync copies and the speed of the slowest one. A primary waits for every in-sync replica, so a slow node, a busy disk or a distant zone in the copy set slows the write. Adding a replica adds one more copy to wait for, and therefore adds a little latency and a lot of extra network, processor and disk work. So the cost of durability is paid mostly through the number of replicas, not through the setting. Choose the number of replicas with that in mind, and use zones that are close to each other in network terms.

What the setting does change: behaviour during trouble

The setting shapes what happens when copies are missing. A strict rule turns a period of missing replicas into a period of waiting and errors. The waiting time is bounded by the timeout. During that time, requests pile up at the client or in the queue, so a strict rule requires a plan for backlog.

Timeouts that fit together

Three timeouts interact. The cluster-side timeout says how long the write waits for copies. The client-side timeout says how long your application waits for any answer. And the timeout of anything in between, such as a proxy or load balancer, says how long the connection may stay open. If the client gives up before the cluster does, the client does not learn the outcome, and you may see duplicates on retry. The safe pattern is: cluster-side timeout shorter than the client’s, and the client’s shorter than the proxy’s.

Retrying well

When a write fails because too few copies were active, the right response is to retry later, with a growing pause between attempts and a little randomness to avoid many clients retrying in step. Give up after a sensible limit, and put the write somewhere safe, such as a dead-letter queue, for later handling. Use explicit document identifiers so that a retry of a write that did in fact succeed replaces the same document and does not create a duplicate.

Do not confuse it with backpressure

Elasticsearch can also refuse writes when it is overloaded, not when copies are missing. In that case the write queues are full, and the response is a “too many requests” error. That is a different problem, with a different cure: slow down, use smaller bulk requests, add capacity. The error for missing copies says the service is unavailable. Learn to tell them apart in your logs.

Capacity: leave room to rebuild

The strictness of your rule only helps if the cluster can rebuild lost copies quickly. That needs spare nodes and disk space. If a node with 2 terabytes of data is lost and the remaining nodes are nearly full, the replicas cannot be rebuilt anywhere and stay missing. A simple planning rule is to be able to lose one node, or one zone, and still have room to place all its copies elsewhere.

Speed of rebuilding matters too. The time during which writes are blocked, if your rule is strict and you have few replicas, is about the time to rebuild the affected shard. A shard of 50 gigabytes rebuilt at 100 megabytes per second takes 500 seconds, a little over eight minutes. A shard of 500 gigabytes takes about eighty-three minutes. This is one more reason to keep shards moderately sized, and it is why very large shards and very strict rules do not go well together.

Reliability: the layers

Think of reliability as layers: replicas protect against machine loss, the requirement stops writes from being accepted with no backup, queues protect against pauses, snapshots protect against mistakes and larger disasters, and cross-region replication protects against the loss of a whole region. The setting in this article is one useful layer among these, and it works best in combination with the others.

No setting is free. Knowing what a stricter requirement gives and takes helps you defend your choice when things go wrong.

Trade-off one: availability versus durability

This is the central trade-off. The default favours availability: writes always go, even at the cost of a temporary single-copy state. A stricter value favours durability: writes wait until a backup is ready. In distributed systems language, you are choosing which of the two to give up a little of when a failure occurs. You cannot have both completely, and the setting lets you choose per index.

Trade-off two: strictness versus operations

The stricter the rule, the more your applications feel every maintenance event. If the operations team restarts a node every night for patching, a strict rule with one replica means nightly write errors. You can soften this by adding a replica, by pausing writes through a queue, or by choosing a slightly less strict value.

Trade-off three: cost versus safety

Each additional replica multiplies the storage and the write work. Two replicas cost roughly half again as much as one, and three cost twice as much. Reserve the expensive combination for data that deserves it.

Trade-off four: simplicity versus fine control

One global rule is simple to understand and audit. Per-index rules match the data better but must be managed, documented and kept in templates. Most teams settle on a small number of classes, such as “standard” and “critical,” and assign each index to one of them.

Comparing the options

These choices are not rivals of other protections. Snapshots, cross-region replication and upstream queues all cover different failures, and a mature design uses several of them together.