Three plain ideas, put together: a cluster, a state, and agreement. Once each is clear, the rest follows naturally.

Elasticsearch stores and searches documents, such as products, articles, or log lines. To handle a lot of data, it does not run on one machine. It runs on many machines called nodes, and a group of nodes that work together is called a cluster. Each index is cut into pieces called shards, and those shards are spread across the nodes, with extra copies called replicas kept for safety.

Now think about what the nodes need to know. Which nodes are in the cluster right now? Which indexes exist, and what fields do they have? Which node holds the main copy of shard 3 of the orders index? Which copy is the freshest? Is any part of the cluster temporarily locked? Every node must give the same answers to these questions, or requests would be sent to the wrong places and data could be lost.

The cluster state is the single, shared record that answers all of these questions. It is a data structure that describes the cluster as a whole. It has a version number, it is changed by exactly one node at a time, and a copy of the latest version is held in the memory of every node in the cluster.

What it is not

A very common mistake is to think that the cluster state contains the documents. It does not. If you index a million documents, the cluster state does not grow by a million entries. The documents live inside shards, on the disks of the data nodes. The cluster state only records that the index exists, what its fields look like, and which nodes hold which shards. This is why the state can stay small even when the data is enormous, and also why it can become large for a different reason: too many indexes, too many shards, or too many fields.

The one-sentence version

If you remember one line from this chapter, make it this: the cluster state is the versioned, cluster-wide description of nodes, indexes, settings, and shard locations, written only by the elected master, agreed by a majority, and cached in memory on every node.

Every design has a problem behind it. The problem here is agreement.

When only one machine is involved, everything is easy. It knows what it has. When many machines cooperate, three difficulties appear at once.

• Disagreement. If node A believes that shard 2 lives on node B, while node C believes it lives on node D, then a write may go to one place and a read to another. The user sees missing or old data.
• Simultaneous changes. If two administrators create an index with the same name at the same moment on two different nodes, someone must decide which one wins. Without a single decision maker, both might succeed and produce a broken cluster.
• Failure. Machines crash, networks break, and messages get lost. The cluster must keep a consistent picture even while parts of it disappear and come back.

The cluster state solves these problems together. It gives the cluster one shared picture. It routes all changes through one elected writer, the master, so there is never a fight over who decides. And it uses a voting rule so that, even during failures, the cluster never ends up with two different maps both claiming to be the real one.

A short history

The exact release for each step is worth double-checking in the release notes of your version. The direction of travel is clear: agreement became safer and more automatic, while the operator’s main job moved from “avoid split brain” to “avoid an oversized state.”

Why not keep the map in a separate database?

Some systems store their cluster description in an outside service. Elasticsearch chose to keep it inside the cluster itself. This makes installation simple, because there is nothing else to run, and it keeps the map close to the machines that need it. The price is that Elasticsearch has to solve the agreement problem on its own, which is why this part of the system is so carefully built.

A dozen words explain nearly everything about cluster state. Learn them here, and later chapters will read like common sense.

Roles: who does what

A node can carry several roles. A small development cluster may have one node doing everything. A large production cluster usually separates them. Master-eligible nodes handle elections and cluster-wide decisions. Data nodes hold shards and do the searching and indexing. Coordinating nodes receive requests, split them up, and gather the results. Every one of these nodes, regardless of role, holds the cluster state in memory, because they all need the map to do their jobs.

Metadata versus routing

The state has two big halves that are worth learning to separate. The metadata half is about what exists and how it is defined. It changes when someone creates an index, adds a field, or updates a setting. The routing half is about where things physically are. It changes when a shard moves, a replica is added, or a node leaves. The metadata half is usually the larger one. The routing half is the one that changes most often during failures.

Versions and immutability

Every published cluster state has a version number that only goes up. A state is treated as read-only once it is created. To change anything, the master builds a brand-new state, with the change applied and a higher version, and publishes that. This makes it easy to reason about: at any moment, a node has one complete and consistent picture, and never a half-changed one.

Quorum in one story

Imagine three judges who must decide a case by majority. Two of them agreeing is enough to have a decision, and the third cannot contradict it. If a storm cuts off one judge, the other two can still decide. If the storm cuts off two judges, the remaining one cannot decide alone, because she cannot know whether the other two reached a different decision. Elasticsearch works in the same way: with three master-eligible nodes, any two form a quorum, and a single isolated node stops making cluster-wide decisions instead of guessing.

Quick glossary

Who holds the state, who writes it, and how does a change reach every machine?

Every node in the cluster holds a full copy of the latest committed cluster state in memory. That is what lets any node, even a coordinating node with no data, route a request correctly. Only one node, the elected master, is allowed to create new versions. The master-eligible nodes also save the metadata part of the state to their disks, and they are the ones who vote on changes.

flowchart TB
    Admin["Admin or application: create index, change settings"] --> Coord["Any node receives the API request"]
    Coord --> M["Elected master node: the only writer of cluster state"]
    subgraph Masters["Master-eligible nodes: vote and save the state on disk"]
        M
        ME1["Master-eligible node 2"]
        ME2["Master-eligible node 3"]
    end
    subgraph Others["All other nodes: hold a full in-memory copy"]
        D1["Data node A"]
        D2["Data node B"]
        C1["Coordinating node"]
    end
    M -->|publishes new state| ME1
    M -->|publishes new state| ME2
    M -->|publishes new state| D1
    M -->|publishes new state| D2
    M -->|publishes new state| C1
    

Read the picture from top to bottom. A request that needs a cluster-wide change, such as creating an index, can arrive at any node. That node forwards the request to the elected master. The master builds the next version of the state and sends it to every node in the cluster, including the other master-eligible nodes, the data nodes, and the coordinating nodes. Nothing in this picture requires the master to be a big machine. Its job is careful bookkeeping, not heavy data work.

The main components

Walking one change through the machine

Now we open the folder and look at the pages. Knowing what is inside tells you what can make it large or slow.

flowchart LR
    CS["Cluster State"] --> N["Nodes: who is in the cluster and their roles"]
    CS --> MD["Metadata"]
    CS --> RT["Routing table: where every shard copy lives"]
    CS --> BL["Blocks: read-only or no-master restrictions"]
    CS --> ID["Identity: cluster id, version, elected master"]
    MD --> S["Persistent settings"]
    MD --> IX["Index settings, mappings, aliases"]
    MD --> TP["Templates and data streams"]
    MD --> OT["Pipelines, lifecycle policies, repositories, stored scripts"]
    

Identity and membership

At the top sits the identity of the state itself: a unique cluster identifier, the current version number, a unique identifier for this exact version, and the name of the elected master. Next comes the list of nodes: for each member, its identifier, its name, its address, its roles, and its attributes, such as a zone label. When a node joins or leaves, this list changes.

Metadata: the long-lived facts

The routing table

The routing table is the part that answers “where is everything?” For each index, it lists every shard, and for each shard, every copy: whether it is the primary or a replica, which node holds it, and which condition it is in. The main conditions are unassigned (no node yet), initializing (being set up on a node), started (ready for use), and relocating (moving to another node). Chapter 8 explores how this table drives cluster health and how the master fills it.

Blocks

Blocks are restrictions recorded in the state. A block at the cluster level might say that there is no master yet, so no cluster-wide changes are possible. A block at the index level might say that the index is read-only, for instance because a disk ran very low on space. Because blocks live in the state, they are seen by every node at the same moment.

Custom sections

Features such as index lifecycle management, snapshots, and data streams add their own sections to the state. These are called custom sections. They are the reason the state grows when you use many features at once, and one of the reasons the exact content differs between versions.

What is not inside

Why the contents matter

Every item above is copied to every node and re-sent whenever it changes. A thousand templates, ten thousand indexes, and hundreds of thousands of fields all sit in this one structure. That is why the discipline of keeping the state lean, discussed in Chapter 13, is one of the most important habits in running a large cluster.

This is the heart of the subject. Here is exactly how one change becomes the new truth on every machine.

flowchart TD
    T["Task arrives in the master queue: for example create index"] --> B["Master batches tasks of the same kind"]
    B --> N["Master computes the next cluster state: version plus one"]
    N --> P["Publish phase: send the state or a diff to every node"]
    P --> Q{"Did a majority of voting nodes accept and save it?"}
    Q -- No --> F["Update fails: the master may step down"]
    Q -- Yes --> C["Commit phase: tell nodes the state is official"]
    C --> A["Every node applies the new state"]
    A --> K["Master acknowledges the request to the caller"]
    

Step one: the queue

Every request that changes the cluster becomes a small task placed in a queue on the master. Creating an index, deleting one, updating a mapping, changing a setting, reporting that a shard has started, and reacting to a lost node are all tasks of this kind. Each task has a priority. Reactions to failures and shard-started reports are urgent, because delays would leave the cluster in a weakened condition. Routine administrative changes have normal priority, and background housekeeping has low priority.

Step two: batching and computing

A single worker thread on the master takes tasks from the queue. To save effort, it groups tasks of the same kind and processes the whole group at once, producing one new state instead of many. For example, a hundred reports of the form “shard started” can be turned into a single new state that marks all hundred shards as started. This batching is a large part of why a master can survive bursts of activity.

The worker never edits the current state. It builds a new, separate state with the changes applied and a version number one higher. Because states are never edited in place, every node can keep using the old picture until the new one is officially accepted.

Step three: publishing

The master sends the new state to every node in the cluster. To save network traffic, it usually sends only a diff, meaning a description of what changed since the previous version. A node that has the previous version can apply the diff. A node that does not, perhaps because it just joined or missed an update, cannot, and the master falls back to sending the complete state. States are also compressed before they travel.

Step four: the vote

When a node receives a new state, it does not use it right away. First it checks that it comes from the master it currently follows, in the current term. Master-eligible nodes also write the new state to their disks. Only then do they answer “accepted.” The master counts the answers. When a majority of the voting nodes have accepted, the state is safe: even if the master crashes at that exact moment, a majority holds the new state, and any future master will find it.

Step five: the commit and apply

Once a majority has accepted, the master sends a commit message. It says, in effect, “this state is now official.” Each node then applies the state: it swaps its in-memory copy for the new one and lets the rest of the node react. A data node might create a shard, delete an index it no longer needs, or start a recovery. Component after component is notified in a fixed order.

Step six: the answer

The master answers the original request. Many administrative operations wait until enough nodes have applied the change, within a time limit, before saying “acknowledged.” If the time limit passes first, the operation may report that it was not acknowledged, even though the change did happen. That is why an unacknowledged answer does not always mean failure, and why you should look at the state, not only at the response.

The values above are documented defaults for recent versions and can be changed, so check the documentation of the version you run.

One update at a time

The master publishes one new state at a time. It does not start the next update until the current one has been handled. This is what keeps versions in a simple, strict order, and it is also the reason that a very slow update, or a very slow node, can delay everything behind it. The queue of waiting tasks then grows, which is one of the clearest warning signs you can watch for.

A shared map is only as trustworthy as the process that chooses its writer. Here is how the cluster chooses and how it stays sure.

Forming a cluster

When a brand-new cluster starts for the first time, the master-eligible nodes must agree on who is in the first voting group. A setting that lists the initial master-eligible nodes is used once, only on the very first start, to bootstrap the cluster. After that, the cluster remembers its voting group on disk and adjusts it automatically as master-eligible nodes come and go. The setting must be removed after the first start, because leaving it in place could allow an accidental new cluster to form if a node is ever restarted in an unexpected way.

Electing a master

Master-eligible nodes watch for a master. If a node cannot find one, or the current master fails its checks, the node starts a pre-vote. This is a polite question to the others: “would you vote for me if I stood for election?” Only if enough nodes say yes does the node actually start an election, which raises the term number. This trick prevents a node with a flaky network connection from repeatedly disrupting a healthy cluster with pointless elections. In an election, each node votes for at most one candidate per term. A candidate that receives votes from a majority of the voting group becomes the master.

The voting configuration and quorum

The voting configuration is the set of master-eligible nodes whose votes count. A quorum is more than half of that set. With three voters, a quorum is two. With five voters, a quorum is three. Both elections and cluster state updates require a quorum. The cluster keeps the voting configuration as an odd number where possible, by leaving out one node from an even-sized group, because an even group tolerates no more failures than the next smaller odd group.

Why split brain cannot happen now

Suppose a network problem cuts a cluster of three master-eligible nodes into a group of two and a group of one. The group of two still holds a majority, so it can elect a master, accept updates, and continue. The lone node cannot reach a majority, so it cannot elect itself, and it cannot commit any state. It steps aside and waits. When the network heals, it rejoins and catches up. At no point are there two masters both making valid decisions, because two groups cannot both hold a majority of the same voting set.

flowchart LR
    subgraph Left["Majority side: 2 of 3 voters"]
        A["Master-eligible node 1"]
        B["Master-eligible node 2"]
    end
    subgraph Right["Minority side: 1 of 3 voters"]
        C["Master-eligible node 3"]
    end
    A --- B
    Left -. "network partition" .- Right
    B --> W["Continues: elects or keeps a master and commits updates"]
    C --> X["Stops: cannot reach a majority, so no cluster changes"]
    

Fault detection: how nodes know something is wrong

Two kinds of regular checks keep the cluster honest. The master pings each follower to make sure it is alive and still following it. Each follower pings the master to make sure it is still there. These pings are frequent, about once per second by default, and a node is declared lost only after several failed attempts in a row, so that a single dropped message does not cause a false alarm. When the master decides that a node is gone, it removes it from the state, which triggers the reactions described in Chapter 8.

The lag detector

A node that is alive but very slow can hold everything up, because it may not apply new states in time. A lag detector watches for nodes that stay behind for too long, by default ninety seconds after a state has been committed. Such nodes are removed from the cluster, and they may rejoin once they are ready. This protects the cluster from a single struggling machine.

Dedicated master nodes

In a large cluster, master-eligible nodes should be dedicated to that job. A master that is also a busy data node can be slowed by heavy searches or long garbage collection pauses, and a slow master means slow cluster changes and, in the worst case, needless elections. Dedicated masters can be modest machines, because their work is bookkeeping, but they need reliable memory, disk, and network. A special kind of node can be given only a vote and never become master, which is useful as a low-cost tie-breaker in a third location.

The part of the state that changes most in real life is the seating chart of shards. Learn how it is decided and how it turns into the health colors you see.

How a shard copy moves through conditions

Health colors come from the routing table

flowchart TD
    A["Read the routing table"] --> B{"Is every primary shard assigned and started?"}
    B -- No --> R["RED: some data is unavailable"]
    B -- Yes --> C{"Is every replica shard assigned and started?"}
    C -- No --> Y["YELLOW: data available but not fully protected"]
    C -- Yes --> G["GREEN: all copies are ready"]
    

How the master decides where shards go

Whenever the routing table changes in a way that leaves unassigned shards, or when balance is poor, the master’s allocation service runs. It considers each unassigned shard copy and asks a panel of rule checkers, called deciders, whether a given node is allowed to hold it. The deciders look at many things.

Trusting the right copy: in-sync tracking

Suppose a primary shard fails. The cluster must promote a replica. But which replica is safe? One that missed recent writes would silently lose data. To prevent this, the metadata records, for every shard, the set of copies that are known to be fully up to date. This is called the in-sync set. Only a member of the in-sync set may be promoted to primary. A stale copy that comes back after a long absence must first catch up, and it is never trusted blindly. This small piece of bookkeeping inside the state is what protects data during failures.

When a node leaves

If the master decides that a node is gone, it removes the node from the state, and shard copies that lived there become unassigned. For lost primaries, an in-sync replica is promoted. For lost replicas, the master could immediately start rebuilding them elsewhere. But a node that has merely restarted is likely to come back within minutes, and rebuilding many gigabytes of data would be wasted effort. Therefore, by default, the master waits about one minute before starting to rebuild, a delay controlled by a setting on each index. If the node returns within that time, its existing data is reused, and recovery is quick.

Rebalancing

Over time, shards may end up unevenly spread. The master occasionally moves shards to even out both the number of shards per node and the disk use, always with limits on how many moves happen at once. Balancing is polite work that gives way to more urgent tasks, and it can be paused by a setting during delicate operations such as rolling upgrades.

Follow the state through a few ordinary events, and its behavior becomes easy to predict.

Event one: an index is born

flowchart LR
    S1["Create index request reaches the master"] --> S2["State v1: index metadata added, shards unassigned"]
    S2 --> S3["Allocation: master picks nodes for each shard copy"]
    S3 --> S4["State v2: shards marked initializing"]
    S4 --> S5["Data nodes create shards and report started"]
    S5 --> S6["State v3: shards marked started"]
    S6 --> S7["Cluster health turns green"]
    

The picture shows an important truth: even a simple action like creating an index produces a chain of state versions. First the master adds the index’s definition and its shard entries, all unassigned. Then allocation chooses nodes and produces another version with the shards initializing. Data nodes then do the physical work of creating the shards and report back. Each of those reports becomes a task, and the master batches them into a further version that marks the shards as started. Only then does the health turn green.

Event two: a new field appears

When a document arrives with a field the index has never seen, and dynamic mapping is on, Elasticsearch must add that field to the mapping. The mapping lives in the cluster state, so the data node that saw the new field sends a mapping update request to the master. The master builds a new state with the extended mapping and publishes it. Indexing of that document waits until the update is accepted. If a stream of documents keeps introducing new field names, the master is asked again and again, and each request costs a full round of publish and commit.

Event three: a node joins or leaves

A new node contacts the master, passes some safety checks, such as having the same cluster identifier and a compatible version, and is added to the node list in a new state. Shards may then be assigned to it. When a node leaves, whether by choice or by failure, the master removes it from the list and the consequences of Chapter 8 follow: promotion of replicas, delayed rebuilding, and rebalancing.

Event four: rollover and lifecycle actions

Lifecycle policies and data stream rollovers create and delete indexes on a schedule. Each of these actions is a state change: a new index appears, an alias or data stream definition is updated, an old index is deleted. A busy system with many data streams may therefore perform a steady stream of small updates all day. That is normal, and it is one reason the master should never be starved of resources.

Event five: an index is deleted

Deleting an index removes its metadata and its routing entries from the state. Data nodes that hold its shards see the new state and delete the files from their disks. The cluster also records a small tombstone entry, a note that the index was deleted, so that leftover copies on a node that was absent during the deletion are not mistaken for a live index when that node returns.

Event six: settings change

Changing an index setting, adding an alias, or updating a cluster setting is one state update each. Because the change is atomic, every node switches to the new behavior at the moment it applies the new state, which is why nodes can agree about rules such as which disk thresholds are in force.

Summary of what triggers updates

A map kept only in memory would vanish at the first power cut. Here is how the cluster remembers itself.

What is saved to disk, and where

The state is saved on disk, but not by every node in the same way. Master-eligible nodes save the cluster-wide metadata, including the metadata of every index. Data nodes save the metadata of the indexes for which they hold shards, so that they know what their own files mean. Nodes that are neither master-eligible nor data nodes save nothing about it. The routing table is not saved, because it can be rebuilt from what nodes report when they start.

A full cluster restart

When every node stops and starts again, the cluster must rebuild its state from disk. The master-eligible nodes elect a master. Until this has happened and the saved metadata has been recovered, a cluster-wide block is in place, and requests fail with a message that the state has not yet been recovered. Then nodes report the shards they hold, the master rebuilds the routing table, and shards are started.

A few gateway settings let you tell the cluster how many data nodes to wait for before it starts assigning shards. If the master begins assigning shards as soon as the first two data nodes show up, and the rest arrive minutes later, it will waste effort copying data that would have been found in place. Waiting for most of the expected nodes, for a limited time, avoids this.

Losing the master

If the master fails, and a majority of voters remain, a new master is elected within seconds. It loads the last committed state it knows, which is guaranteed to be the newest because a majority had accepted it. During those seconds, changes to the cluster are not possible, but reading and writing documents on existing shards keeps working, because that traffic does not need the master.

Losing most of the masters

If more than half of the master-eligible nodes are lost for good, the cluster cannot form a majority. It cannot elect a master, and it cannot change its state. The right response is to bring the lost nodes back, if at all possible. Only if they can never return, there is a last-resort command-line tool that ships with Elasticsearch which can reset the voting group so that the surviving nodes can form a new cluster. It is dangerous, because it may lose recent changes to the state, and it must be used only with the documentation open and with backups in hand.

Dangling indexes

Sometimes a data node holds files for an index that the cluster state does not list. This can happen if the node was away while the index was deleted, or after certain disasters. Such an index is called dangling. Elasticsearch does not add it back on its own, because that could bring back deleted data. An operator may inspect the dangling indexes and choose to import or delete each one, and must explicitly accept the risk of data loss when importing.

Snapshots and the global state

A snapshot can optionally include the global cluster state: persistent settings, templates, lifecycle policies, and similar metadata, in addition to the indexes’ own data. Restoring this global state into a new cluster can rebuild a working environment, but it should be done thoughtfully, since it can overwrite settings and templates that already exist. Think of snapshots as the real insurance policy for the state, because a single cluster’s disks are not enough to protect against every disaster.

A disaster recovery checklist

You cannot manage what you cannot see. Elasticsearch gives you several windows onto the state, each showing a different slice.

The state dump

The full state dump is the most direct window. You can ask for the whole state or only for parts of it, such as the metadata or the routing table, and you can limit it to certain indexes. On a large cluster, the full dump can be enormous, tens or hundreds of megabytes, and requesting it repeatedly puts load on the node that serves it. Always ask for the smallest slice that answers your question.

How to read a dump

Comparing what nodes believe

Because every node holds its own copy of the state, you can, in a troubleshooting session, ask several nodes for their version of the state and compare the version numbers and the master. In a healthy cluster, they match, apart from tiny delays right after an update. A node that is stuck on an old version is a strong lead: it is probably slow, overloaded, or partly disconnected.

Health with patience

The health endpoint can also wait. You can ask it to hold the answer until the cluster reaches a certain color, or until a number of shards are moving, or until there are no more tasks waiting, for up to a time limit. Scripts that perform maintenance use this to pause safely between steps, instead of guessing how long to sleep.

Some of the rules that shape the cluster are kept inside the state itself. This makes them easy to change, and easy to forget.

Where settings can live

Elasticsearch settings come from several places. Some are written in configuration files on each node and can only be changed by restarting that node; these are called static settings. Others can be changed while the cluster runs; these are dynamic settings, and they are stored in the cluster state so that all nodes see them at the same moment. Dynamic cluster settings can be marked persistent, meaning they survive a full restart, or transient, meaning they were meant to disappear after one.

Recent versions discourage transient settings, because their behavior during restarts and failures has surprised many operators, and recommend persistent settings or configuration files instead. Check the documentation for your version.

Examples of settings that shape the state

Changing and removing a setting

A dynamic setting can be updated with the settings API. To remove a setting and return to its default, you set it to nothing, which makes the cluster forget it. Each change is one state update, applied atomically on every node. Because these changes are stored in the state, they are also part of what a snapshot with global state would carry, so a restored cluster may inherit them, sometimes to the surprise of the person who restores it.

Keep settings in a record

A cluster with many hand-changed persistent settings becomes a puzzle: nobody remembers why a rule was added. A better habit is to keep a written list or a configuration file in source control, listing every non-default setting, the reason, the person who set it, and the date. Review this list every quarter and remove what is no longer needed. Maintenance switches, such as disabling allocation during an upgrade, should have an entry in the runbook that also says how to switch them back.

The cluster state is small when a cluster is small. At large scale, it becomes the thing that limits how far the cluster can grow.

Documents scale beautifully: add nodes, spread shards, and capacity grows. The cluster state does not scale in the same way. Every node must hold all of it, and every update must reach every node. So its cost grows with the number of nodes, and with the size of the state, at the same time. The cluster state is therefore the part of Elasticsearch where “too many small things” causes the most trouble.

What makes a state large

A numbers walkthrough

Let us use invented numbers to feel the cost. Suppose the cluster state is 50 megabytes and the cluster has 100 nodes. If every update sent the whole state, one update would move 5 gigabytes across the network, and a hundred updates per minute would need hundreds of gigabytes of traffic per minute just for bookkeeping. Diffs make this survivable, because a typical update changes only a few kilobytes, so the cost is small. But a node that misses an update, or joins the cluster, needs a full copy, and a burst of restarts can cause many full transfers at once.

Now consider the memory cost. Every node keeps the whole state in its heap, and sometimes several versions at once, while it applies a new version. A 50-megabyte state might take several times that in memory once it is expanded into objects. On a small node with a modest heap, that is a noticeable share of the memory, taken before a single search runs.

The master as a bottleneck

All state changes pass through one thread on one machine. Whatever the size of the cluster, the master can only process so many updates per second, and each one takes longer when the state is larger, because building and serializing the new state costs time in proportion to its size. When updates arrive faster than the master can finish them, the queue of pending tasks grows, and the effects become visible: index creation takes longer, shards take longer to start, and recovery after failures slows down.

Guard rails Elasticsearch gives you

• Shard limits per node. A default cap on the number of shards each data node may hold protects nodes and the state from unbounded growth. Do not raise it without good evidence.
• Field limits per index. A cap on the number of fields in an index stops runaway mappings. Raising it is a warning sign, not a solution.
• Slow-task warnings. The master writes a warning to its log when a state update takes unusually long, which points to the cause.
• Diffs and compression. Built-in efficiency features that keep ordinary updates small.

How to keep the state lean

The many-clusters alternative

When a single cluster truly needs to hold an enormous number of indexes, splitting the workload across several smaller clusters is often wiser than growing one giant state. Cross-cluster search can still query them together when needed. Each cluster then has its own smaller state, its own master, and its own limits, and a problem in one does not spread to the others.

Every design choice carries a price. Here is the honest balance sheet of having a single, shared cluster state.

How strict should you be?

A cluster is only as reliable as its ability to keep the state healthy through failures. Here is how it behaves, and how to prepare it.

What happens in each failure

Designing for resilience

• Three or more dedicated master-eligible nodes. An odd number, on reliable machines, separate from the busy data nodes.
• Spread across failure zones. Put master-eligible nodes in three different zones, if possible, so that losing one zone never costs a majority. A small voting-only node in a third location can serve as a tie-breaker.
• Zone awareness for shards. Tell the allocation rules about zones, so copies of a shard live in different zones.
• Replicas everywhere important. At least one replica for indexes you cannot afford to lose, so that a failed node causes no data loss.
• Reliable networking. Low latency and stable links between master-eligible nodes, because the vote depends on them.
• Snapshots. Regular, tested snapshots as the last line of defense.

Rolling restarts and upgrades

To restart nodes one at a time without disruption, the usual approach is to limit shard allocation to primaries before you begin, so that the cluster does not waste effort rebuilding replicas for a node that will be back in minutes. You then restart each node, wait for it to rejoin, and restore normal allocation, waiting for health to return to green before touching the next node. Master-eligible nodes should be restarted with care: never more than one at a time, and only when the cluster is healthy.

Slow is the enemy of reliable

Most instability in the state layer does not come from a clean crash. It comes from slowness: a master paused by long garbage collection, a network that drops packets, a disk that stalls while saving the state. Slow masters can lose their followers’ confidence, provoke elections, and cause the cluster to flap between masters. Keep heap sizes sane, watch garbage collection on masters, use fast disks, and avoid running heavy workloads on the same machines as masters.

The cluster state is the control panel of the cluster. Whoever can change it can change almost everything. That deserves careful protection.

Who may look, and who may change

Elasticsearch security separates the right to look at cluster information from the right to change it. Read-only monitoring privileges allow someone to view health and statistics, and often the state dump. Management privileges allow changes: to settings, templates, lifecycle policies, snapshots, and the shape of indexes. Give people and programs only the level they need. A dashboard needs to read. It does not need to change. A deployment tool might need to manage templates, but not the whole cluster.

What the state reveals

The state contains the names of indexes and aliases, the structure of mappings, the addresses of nodes, the layout of shards, and more. This is metadata, not documents, but it can still be sensitive. In a system where index names contain customer names, for instance, the state reveals who the customers are. Limit who can read the full state, and be careful before pasting a state dump into a support ticket or a public forum. Remove names and addresses first.

Keep strangers out of the cluster

A node that joins the cluster receives the whole state. If an attacker could start a node that joins your cluster, they would learn its layout and could receive shard copies, and therefore data. Prevent this by enabling encryption and mutual authentication between nodes, so that only nodes with valid certificates can join, and by restricting network access to the ports that nodes use to talk to each other.

Dangerous power tools

The command-line tools that can rewrite the voting group or reset metadata are extremely powerful, and they work directly on the files of a stopped node. Anyone with shell access to a master-eligible machine could, in principle, misuse them. Restrict shell access, log administrative sessions, and treat the machines that hold the state as sensitive systems.

Bootstrapping safely

The setting used to form a new cluster for the first time should only be present during that first start. Remove it afterward, so that a mistaken restart on a new machine cannot silently create a second cluster with the same name. A good habit is to give each cluster a unique name, which also prevents nodes from accidentally joining the wrong one.

Changes as a control point

Because settings, templates, and policies are all changes to the state, they make a natural place to apply review and approval. Keep these definitions in source control, apply them through a pipeline, and grant direct human access only for emergencies. That way, every important change has a record, a reviewer, and a way back.

The cluster state rarely fails suddenly. It usually gives warnings first. These are the signals worth watching.

Reading the patterns

Logs that help

The master’s log tells the story of the cluster. It records elections, nodes joining and leaving, and warnings when a state update or a task takes too long. Warnings about lagging nodes name the node that fell behind, and how far. Read the master’s log first when the cluster behaves oddly, and keep logs long enough to investigate problems that appeared days ago. Ship them to a central place, so that they survive the loss of a machine.

A troubleshooting playbook

Good habits keep the state small and calm. Bad habits are behind most cluster-wide slowdowns.

Patterns that work

Anti-patterns to avoid

A short list you could hand to a new teammate on their first day.

Best practices

Common mistakes

• Thinking the state contains the data. It contains the map, not the documents. Deleting or corrupting it is a metadata problem with serious consequences, but a small state can sit on top of a huge dataset.
• Ignoring shard counts. Adding shards “for safety” or “for future growth” makes the state larger and every node slower.
• Raising limits to silence warnings. Increasing the field limit or the shard limit hides the symptom and keeps the cause.
• Treating an unacknowledged response as failure. The change may have happened. Look at the state.
• Restarting masters casually. Restart them one at a time, and only when the cluster is healthy.
• Skipping tests of recovery. A snapshot that has never been restored is only a hope.
• Forgetting to remove the bootstrap setting. It should not stay in the configuration after the cluster has formed.
• Assuming transient settings are safe. They can vanish at restarts and cause confusing behavior.

Myths versus facts

Where does the cluster state quietly decide success or trouble?

Many well-known organizations have said publicly that Elasticsearch is part of their systems, including Wikipedia’s search, Netflix, Uber, and Stack Overflow. They rarely publish the details of how they manage cluster state, so the scenarios below are typical patterns for such workloads, not descriptions of anyone’s private configuration.

Short answers to the questions that come up most often.

If you skim only one part of this guide, make it this one.

The cluster state is the shared map of an Elasticsearch cluster. It says who is in the cluster, what indexes exist, how they are defined, and where every shard lives. It is written by one elected master, agreed by a majority, and carried in the memory of every node. It is small and quiet when the cluster is well shaped, and it is the first thing to strain when a cluster has too many tiny indexes, shards, or fields.