The shortest possible answer: master election is the process by which the master-eligible nodes of an Elasticsearch cluster choose exactly one of themselves to be the leader that manages cluster-wide decisions.

Let us build the definition slowly. Elasticsearch is a search and analytics engine. Companies use it to search products, explore application logs, count events, and draw charts. It usually runs as a cluster, which is a team of servers called nodes that share the work and the data. Data is stored in indices, which are like big labeled cabinets of documents, and each index is cut into pieces called shards so that many nodes can hold and search them together.

A team like this needs someone in charge of the things that affect the whole team: which indices exist, what fields they contain, which node holds which shard, and which nodes are currently part of the cluster. In Elasticsearch, that someone is the elected master node, often just called the master. At any moment, a healthy cluster has exactly one.

Master election is the voting procedure that picks that one node. It runs when a new cluster starts for the first time, and it runs again whenever the current master is lost, stops responding, or steps down. Only nodes that are allowed to lead, called master-eligible nodes, take part in the vote.

Three things that make an election safe

Elasticsearch’s election is built around three simple promises. First, at most one master at a time: the cluster must never have two leaders working in parallel. Second, majority rule: a node becomes master only when more than half of the voting members agree. Third, no lost decisions: a new master must know everything the old master had already decided, so that agreed changes are never forgotten.

What master election is not

Master election does not decide where your documents live, and it does not sit in the path of every search or write. Searches and document writes flow directly between clients and the data nodes. The master handles the rules of the cluster, not the traffic. It is also different from the choice of a primary shard, which is the main copy of a slice of data. Chapter 16 explains this common confusion in detail.

Where you will meet it

You will meet master election when you start a brand-new cluster and must tell the nodes who the first voters are, when a node crashes and the cluster health flickers for a few seconds, when a network problem separates two groups of nodes, when you plan a rolling restart, and when you see log lines such as “master not discovered yet.” In each case, understanding the vote turns a mysterious message into a clear picture.

Getting many computers to agree is one of the hardest problems in computing. Master election is Elasticsearch’s answer: agree once on a leader, and let the leader make the rest of the decisions.

Imagine two nodes in the same cluster receive two requests at the same moment. One asks to create an index with a certain structure, and the other asks to create an index with the same name but a different structure. If both nodes act independently, the cluster now contains two versions of the truth. Searches will give different answers depending on which node they reach, and repairing the mess later can be painful or impossible.

One pen, one notebook

The simplest fix is to allow only one node to change the cluster’s structure. All requests that change the structure are sent to that node. It decides the order, applies each change, and tells everyone else the result. Because there is only one decision maker, conflicts cannot happen. This is the job of the elected master.

But a single leader creates a new problem. What if the leader dies? A cluster that depends on one fixed node is only as reliable as that node. So Elasticsearch lets several nodes be master-eligible, and it uses an election to choose a replacement automatically. The design goal is simple to say and difficult to achieve: there must always be at most one leader, and there must be a new one soon after the old one is gone.

The two hard truths of networks

Election is hard because computers and networks misbehave in two ways. First, messages can be lost or delayed. A node that does not answer might be dead, or it might be alive and simply slow, or the network cable between you and it might be cut. From the outside, these three cases look identical. Second, nodes can crash and restart at any time, forgetting whatever they kept only in memory. A good election method must stay correct under both problems.

The danger called split brain

The worst thing that can happen is split brain: a network problem cuts the cluster into two groups, and each group elects its own master. Now two leaders accept changes independently. When the network heals, the two histories cannot be merged, and some acknowledged changes must be thrown away. Older versions of Elasticsearch were exposed to this risk when administrators set the voting rules incorrectly. Modern versions prevent it by design, using majority voting. Chapter 8 tells the full story.

An illustrative story

A short history

Older Elasticsearch versions, up to the 6.x line, used a homegrown method called Zen Discovery. It worked, but it demanded that administrators set a number by hand, the minimum number of master nodes, and mistakes in that number were a common cause of data problems. Version 7.0 introduced a new cluster coordination system, designed with a formal model and inspired by well-known agreement algorithms such as Raft. It manages the voting rules automatically, made elections faster and safer, and removed the old manual setting. This guide describes that modern system.

To follow the rest of this guide you need eight simple ideas: node, cluster, role, cluster state, quorum, vote, term, and the three election states.

Consensus in plain words

The word consensus means that a group of computers agrees on a value, even when some of them fail and messages go missing. Master election is a consensus problem: the value to agree on is “who is the leader right now.” Publishing a change to the cluster state is another consensus problem: the value to agree on is “what is the new state.” Elasticsearch solves both with the same idea, that a decision is valid only when a majority accepts it.

Why majorities work

Any two majorities of the same group must overlap in at least one member. If five nodes vote, any group of three and any other group of three share at least one node. That shared member cannot vote for two different leaders in the same round, so two competing leaders can never both win. This overlap property is the mathematical heart of safe elections, and it explains why odd numbers such as three or five master-eligible nodes are recommended.

A cluster in one picture

The diagram shows a small cluster. Three master-eligible nodes take part in elections, and one of them is currently the elected master. Data nodes hold the shards. Clients reach the cluster through a load balancer or an API gateway that spreads requests over the nodes.

flowchart TD
    A["Client application"] --> B["Load balancer or API gateway"]
    B --> D1["Data node 1"]
    B --> D2["Data node 2"]
    B --> D3["Data node 3"]
    M1["Master-eligible node 1: elected master"] --- M2["Master-eligible node 2: follower"]
    M2 --- M3["Master-eligible node 3: follower"]
    M1 --- M3
    M1 -.->|Publishes cluster state| D1
    M1 -.->|Publishes cluster state| D2
    M1 -.->|Publishes cluster state| D3

Look closely at Figure 1. Client requests do not go through the master. The dotted lines show that the master sends the cluster state, the shared notebook, to all nodes. This split between the control path and the data path is one of the most important ideas in Elasticsearch operations.

Not every node is allowed to lead. Roles decide who can vote, who can be elected, and who only does data work.

Every Elasticsearch node has a list of roles, set in its configuration. The roles tell the cluster what the node is allowed to do. Some roles are about data, such as holding shards or processing incoming documents. One role, master, is about leadership. A node with the master role is a master-eligible node: it takes part in elections and may be chosen as the leader.

The main roles for elections

Dedicated master nodes

A dedicated master node has only the master role. It holds no shards and does no searching or indexing. Its whole job is to keep the cluster state and take part in elections. Because it is not busy with heavy work, it responds quickly and is less likely to be pushed into long pauses. Large production clusters almost always use dedicated masters, often three of them. Small clusters may skip them and let data nodes also be master-eligible, at the cost of mixing heavy work with leadership work.

Voting-only nodes

A voting-only node is a master-eligible node that has been given an extra role called voting_only. It votes like any other master-eligible node, but it can never become the leader. This is useful as a cheap tiebreaker. If you have two rooms with one master-eligible node each, a third small voting-only node in a third location lets the cluster keep a majority when one room fails, without needing a full-sized leader there.

Data nodes and coordinating nodes

A data node stores shards and runs searches and indexing on them. A coordinating node has no special roles and simply receives client requests, sends them to the right shards, and merges the answers. Neither of them votes or leads. They follow the cluster state that the elected master publishes. In a well-designed cluster, these roles are separated so that heavy queries cannot disturb leadership.

flowchart LR
    subgraph Voters["Master-eligible nodes: can vote"]
        A["Master-eligible node A: may lead"]
        B["Master-eligible node B: may lead"]
        C["Voting-only node C: votes, never leads"]
    end
    subgraph Workers["Nodes without the master role: no vote"]
        D["Data node"]
        E["Data node"]
        F["Coordinating node"]
    end
    A -.-> D
    A -.-> E
    A -.-> F

Setting roles

Roles are listed in the node’s configuration file using the node.roles setting. If the setting is missing, a node receives a default set of many roles, including master and data. This default is convenient for learning, but for production you should choose roles on purpose, because a node that quietly holds the master role can be elected even if you never planned it.

The master is the cluster’s manager, not its worker. Knowing the exact scope of its job helps you size it, protect it, and predict what breaks when it is missing.

The elected master is responsible for cluster-wide decisions. These are the decisions that must be made once and shared by everyone. Because it is the only node that changes the cluster state, it sees every structural change before anyone else does.

The master’s duties

• Tracking membership. It knows which nodes are part of the cluster, adds nodes that join, and removes nodes that stop responding.
• Creating and deleting indices. When a client asks for a new index, the master decides its settings and records it in the cluster state.
• Managing mappings. A mapping describes the fields of an index and their types. When new fields must be added, the master approves and records the change.
• Deciding shard allocation. It decides which node holds which shard copy, moves shards to balance load, and promotes a replica to primary when a primary is lost.
• Handling settings and templates. Changes to cluster settings, index settings, and templates flow through it.
• Running some management jobs. Several scheduled housekeeping features, such as index lifecycle management, are driven from the elected master.

What the master does not do

The master does not receive your search requests, and it does not receive your document writes. A write goes from the client to a node, then to the node holding the primary shard, and then to the replicas. A search goes to the nodes holding the relevant shards. The master is not on that path. This is why a cluster can keep answering searches for existing data even in the short moment when there is no master, though it cannot change its structure until a new master appears.

A busy master can slow the whole cluster

Although the master is not on the data path, every structural change waits in a queue at the master. If the master is overloaded, creating an index or applying a mapping may take a long time, and shard recovery after a failure may be delayed. Clusters with thousands of indices or hundreds of thousands of shards give the master a heavy workload. Later chapters show how to spot and prevent that.

A good mental model

Think of the cluster as having two parts. The control plane decides how the cluster is organized, and the master lives there. The data plane stores and searches the data, and data nodes live there. A healthy design keeps the control plane light and stable, so that decisions are quick, and lets the data plane grow with your traffic.

Everything the master decides ends up in one structure. Understanding it explains why elections must be careful and why a huge cluster state slows everything down.

The cluster state is a compact record of how the cluster is organized right now. It is the shared notebook of the whole team. Every node holds a recent copy, and the elected master is the only one allowed to write in it.

What is inside

Notice what is not in the cluster state: your documents. The state holds the structure, not the contents. Even so, it can grow large in clusters with many indices, many fields, or many shards, and its size is a key factor in how fast the master can work.

Versions and terms on every state

Each published cluster state carries two numbers: the term of the leader that produced it, and a version that increases with every change. Together they tell any node which of two states is newer. A state from a higher term is always newer, and within the same term, a higher version is newer. These two numbers play a central role in elections, because a voter compares them before deciding whether a candidate is up to date.

Stored on disk for safety

Master-eligible nodes save the current term and the latest accepted cluster state on their own disks before they acknowledge a change. Because of this, a restarted master-eligible node remembers what it agreed to. Without this memory, a node could forget that it had already voted, and might vote twice in the same term, which would break the safety of the election. This is why the data directory of a master-eligible node must never be wiped casually, and why a fast, reliable disk matters even for a dedicated master that holds no shards.

Full states and differences

When the master publishes a change, it does not always send the whole notebook. If a node already has the previous version, the master sends only the difference, which is small and quick. If a node is new, or has fallen too far behind, the master sends the full state. Big clusters benefit a lot from this saving, but a very large state still costs time whenever a full copy has to travel.

Why size matters

Every structural change means computing a new state, sending it to all nodes, waiting for acknowledgments, and applying it. The larger the state, the longer each step takes. Common causes of a bloated state include huge numbers of tiny indices, indices with thousands of fields, and unbounded creation of new fields from unpredictable data. A clean design keeps the state small, which keeps the master fast and elections calm.

A quorum is the number of votes that must agree before a decision counts. Elasticsearch keeps track of who is allowed to vote and adjusts that list automatically.

The set of master-eligible nodes whose votes count is called the voting configuration. A quorum is a majority of that set: more than half of its members. For a voting configuration of three nodes, a quorum is two. For five, it is three. The formula is simply half the number of voters, rounded down, plus one.

How many failures can you survive?

The pattern is that adding a node to make an even number gives no extra protection, because the quorum grows along with the group. That is why guides insist on odd numbers. Elasticsearch enforces this automatically: if you have an even number of master-eligible nodes, it leaves one of them out of the voting configuration, so the effective number of voters stays odd.

Automatic adjustment

The voting configuration is not fixed. When master-eligible nodes join or leave, the elected master changes the configuration for you, as long as the change is safe. This adjustment is what removed the need for the old manual “minimum master nodes” setting. You can grow a cluster from three to five master-eligible nodes, and the voting set follows. You can also shrink it, as long as you follow the safety rule below.

The safety rule for removing nodes

The rule is easy to remember: never stop half or more of the voting nodes at the same time. If you stop fewer than half, the remaining nodes still hold a quorum, and the master can update the voting configuration to match. Wait for that adjustment to finish before stopping the next node. If you must remove half or more of the master-eligible nodes, use the voting configuration exclusions feature, which tells the cluster to move the vote away from those nodes first, so that shutting them down does not cost you the quorum.

Split brain is the classic disaster of leader-based systems. Modern Elasticsearch prevents it by making it mathematically impossible for two leaders to win.

Suppose a cluster of five nodes is running happily, and a network fault cuts it into two groups: three nodes on one side and two on the other. Each side can still talk internally, but not across the cut. From the point of view of each group, the other side has vanished.

What would go wrong without majority voting

If the two-node group decided that it must elect its own master because the old one is out of reach, it would do so. Both groups would then accept changes independently. After the network is repaired, the two histories conflict: two sets of indices, two shard layouts, and possibly two different versions of the same document. Data written to one side must be discarded. That outcome is called split brain, and it is a data loss event.

How the quorum rule stops it

With majority voting, only the group that holds a majority of the voting configuration can elect or keep a master. In our example, the group of three has a quorum of three out of five, so it continues normally. The group of two has only two out of five, which is not a majority, so it cannot elect a master. It stops changing the cluster and waits. When the network heals, the two nodes rejoin the larger group and catch up. No decision was made in two places, so nothing needs to be thrown away.

flowchart TD
    subgraph Before["Before the network fault: 5 voters, one cluster"]
        A1["Node 1"] --- A2["Node 2"] --- A3["Node 3"] --- A4["Node 4"] --- A5["Node 5"]
    end
    Before --> Cut["Network fault splits the cluster"]
    Cut --> Side1["Side with 3 nodes: has a majority, elects or keeps a master"]
    Cut --> Side2["Side with 2 nodes: no majority, cannot elect a master, waits"]
    Side2 --> Heal["Network heals: the 2 nodes rejoin and catch up"]

Old and new compared

What the minority side experiences

Nodes on the minority side cannot see a master, so they apply a protective block. By default, this block rejects write operations, while reads continue against the last known cluster state, which may be a little stale. If you prefer the strictest behavior, the cluster.no_master_block setting can be changed to reject all operations. Clients that are connected to the minority side will see errors such as “master not discovered” or “blocked by no master,” until the cluster heals or the client reaches the majority side.

Why three zones matter

The majority rule has an implication for where you place master-eligible nodes. If all three are in one data center room, a failure of that room takes out all voters. If they are spread over three separate zones, the loss of any one zone leaves two voters, which is still a majority. This is why many production designs use three zones, or two zones plus a tiebreaker in a third. Chapter 18 shows those designs.

Before any election can happen, nodes must find each other. And a brand-new cluster needs a one-time founding step to decide who the first voters are.

Discovery: finding your peers

Discovery is the process by which a node finds other nodes that might be part of its cluster. A node cannot search the whole network, so you give it a list of addresses to try, called seed hosts, in the discovery.seed_hosts setting. The list does not need to include every node, only enough well-known ones. The node contacts these addresses, and they tell it about the other nodes they know. This chain of introductions spreads knowledge of the whole group.

In cloud environments, you can use a seed provider instead of a fixed list. A provider asks the cloud platform or a file for the current addresses, which is convenient when machines come and go. Containers and Kubernetes setups often use a service name that resolves to the current members.

Bootstrapping: the founding meeting

A brand-new cluster has a special problem. There is no existing voting configuration, so nobody knows how many votes are needed. Elasticsearch solves this with a one-time bootstrap. You provide the setting cluster.initial_master_nodes, a list of the names of the master-eligible nodes that will form the first voting configuration. Each of these nodes waits until it has found a majority of the listed nodes, and then the first election can take place.

flowchart TD
    A["Master-eligible nodes start for the first time"] --> B["Each node probes its seed hosts"]
    B --> C["Nodes find each other and compare the initial master node list"]
    C --> D{"Found a majority of the listed initial nodes?"}
    D -->|No| E["Keep probing and log that the master is not discovered yet"]
    E --> B
    D -->|Yes| F["First election runs and one node becomes master"]
    F --> G["Voting configuration is stored in the cluster state"]
    G --> H["The initial master nodes setting is no longer needed"]

Rules of bootstrapping

• Use it only for a brand-new cluster. After the cluster has formed, the initial master nodes setting has no purpose and should be removed from the configuration.
• Never use it on a node that joins an existing cluster. A new node that is told to bootstrap may create its own separate cluster instead of joining the real one.
• List the same nodes everywhere. Every master-eligible node in the first group should have an identical list of names, using each node’s configured node name.
• Do not include non-master nodes. The list names only the founding voters.
• Do not bootstrap two clusters with the same seed hosts. This confuses which nodes belong together.

What happens on a laptop

If you start Elasticsearch on a single machine for learning, without special network settings, nodes on that machine can find each other and form a cluster automatically. This convenience applies only in development mode, when the node listens on the local machine. In production mode, where the node listens on a real network address, Elasticsearch runs strict startup checks and refuses to start unless discovery is configured with seed hosts, a provider, or the initial master list. This prevents accidental clusters that never form.

The single-node case

A cluster of one node is a valid cluster, and it elects itself. For this case, Elasticsearch offers a single-node discovery mode that skips the search for peers. It is useful for development and tests. It is not a way to run a fault-tolerant service, since the loss of that one node loses the cluster.

Cluster identity

Once a cluster is created, it receives a unique identifier that is stored in its cluster state and on the disks of its nodes. A node that carries the identity of one cluster refuses to join a different cluster, even if the names look the same. This safety catch prevents data from two unrelated clusters from being mixed, and it explains an error you may meet if a node was accidentally bootstrapped on its own.

Here is the heart of the topic: how a group of candidates turns into one leader in a fraction of a second, without any central coordinator.

An election begins when a master-eligible node finds itself without a leader. This is true when a node starts up, when it loses contact with the leader, or when the leader tells the world that it is stepping down. The node becomes a candidate and goes through the following steps.

sequenceDiagram
    participant A as Candidate A
    participant B as Node B
    participant C as Node C
    A->>B: Pre-vote: would you vote for me?
    A->>C: Pre-vote: would you vote for me?
    B-->>A: Yes
    C-->>A: Yes
    A->>A: Choose a higher term
    A->>B: Request a vote for the new term
    A->>C: Request a vote for the new term
    B-->>A: Vote granted
    C-->>A: Vote granted
    A->>A: Quorum reached, become leader
    A->>B: Publish cluster state naming A as master
    A->>C: Publish cluster state naming A as master
    B-->>A: Accepted
    C-->>A: Accepted
    A->>B: Commit
    A->>C: Commit

Why pre-voting matters

Without pre-voting, a node that was briefly cut off from the network would return with a higher term number and force the healthy leader to step down, even though nothing was wrong with the cluster. That disruption is like a student who was absent for a week coming back and demanding a new election. Pre-voting stops it: nodes that are content with their leader answer no, so the returning node cannot gather a quorum and simply rejoins as a follower.

The up-to-date rule

The check in step five protects agreed decisions. A voter remembers the newest cluster state it has accepted, identified by term and version. It will not vote for a candidate whose newest state is older. Because a change is committed only after a quorum has accepted it, and any two quorums overlap, at least one voter in the winning quorum holds every committed change. Therefore, a winning candidate has always seen everything that was committed, and no agreed change is ever forgotten.

When two candidates compete

Sometimes two nodes start at almost the same time, and each gets some votes but neither gets a majority. This is called a split vote. Nothing bad happens: both attempts simply fail, and the candidates wait different random delays before trying again. Randomness makes it very unlikely that they collide twice in a row. Typically, one of them wins within a fraction of a second.

Every master-eligible node is always in one of three states. Watching how nodes move between them makes leadership changes easy to understand.

A master-eligible node is at any moment a candidate, a leader, or a follower. These three states form a simple cycle, and every transition has a clear cause.

stateDiagram-v2
    [*] --> Candidate
    Candidate --> Leader: Wins an election with a quorum
    Candidate --> Follower: Accepts a leader that already exists
    Follower --> Candidate: Leader checks fail
    Leader --> Candidate: Loses quorum or sees a higher term
    Follower --> Follower: Applies published cluster states

What a term really means

A term is a numbered period of leadership. When an election succeeds, the winner leads for that term. If the leader is lost and a new election starts, the number goes up by at least one. Terms make it possible to compare claims: when two nodes both think they are leader, the one with the higher term is right, and the other must step down. Terms are stored on disk together with the cluster state so that they survive restarts.

The rules that keep the states safe

• One vote per term. A node grants at most one vote in any term, and it remembers this even after a restart, because the term and its vote are saved on disk.
• Higher term wins. A node that receives any message with a higher term immediately updates its own term and gives up any leadership claim.
• Only a quorum makes a leader. A candidate does not become leader on the strength of a few votes, and it does not become leader by declaring itself so.
• Leaders must keep proving themselves. A leader that cannot get a quorum to accept its published states steps down.

Leader stability in practice

In a healthy cluster, the leader stays the same for days or weeks, and the term number hardly changes. If you see the term increase often, it means that leadership is unstable, and something in the environment is causing nodes to lose contact: long pauses, a flaky network, overloaded machines, or too tight timeouts. A stable term number is a quiet sign of a healthy cluster.

Nodes that are not candidates

Data nodes and other nodes without the master role never go through these states. They simply find the elected master, join it, and follow its cluster states. If the master changes, they detect the change and connect to the new one. They do not vote, and they cannot become leaders.

Elections rely on small random delays to avoid collisions. A handful of settings control those delays, and the defaults are usually right.

If every master-eligible node started an election the instant it lost the leader, all of them would collide, votes would be split, and nobody would win. Elasticsearch solves this with randomized waiting. Each candidate waits for a random time before it tries, so one of them almost always goes first and collects the votes before the others even start.

How the back-off works

The first attempt uses a wait between zero and the initial timeout. If the attempt fails, the upper limit grows by the back-off time. After the second failure, it grows again, and so on, until it reaches the maximum. This growing range means that when the cluster is having real trouble, for example because many nodes are competing or the network is slow, candidates spread out further and further apart, which makes a clean win more likely.

When to change them

Almost never. The defaults were chosen for typical networks. If your nodes are on a network with high latency, for example spread across distant places, votes take longer to travel, and the election duration may need to be raised so that candidates do not give up before the replies arrive. Lowering the values too much can cause repeated collisions. Raising them too much makes recovery slower than necessary. Treat any change as an experiment to be tested in a non-production cluster first.

Elections are quick when the cluster is healthy

Because the delays are short and the vote is a single round of messages, an election in a healthy cluster with a good network takes a fraction of a second. Most of the total time of a failover is not the election itself, but the time to notice that the old leader is gone, which is the topic of Chapter 14.

After the election, the leader’s daily work is to publish changes. Each change is announced, confirmed by a majority, and only then made official.

Every structural change, whether it is a new index, a moved shard, or a new node, becomes a new cluster state. The leader must be sure that the change is safely recorded by a majority before anyone treats it as real. It uses a two-step procedure known as a two-phase commit. In the first phase, the change is proposed and accepted. In the second phase, it is committed and applied.

sequenceDiagram
    participant Cl as Client
    participant M as Elected master
    participant N1 as Node 1
    participant N2 as Node 2
    Cl->>M: Create an index
    M->>M: Compute the new cluster state
    M->>N1: Publish new state
    M->>N2: Publish new state
    N1-->>M: Accepted and saved to disk
    N2-->>M: Accepted and saved to disk
    M->>N1: Commit
    M->>N2: Commit
    N1-->>M: Applied
    N2-->>M: Applied
    M-->>Cl: Index created

Why two phases

The first phase lets the leader find out whether a majority can store the change. If it cannot reach a majority, it does not commit, and no node ever treats the change as final. This protects against the case where the leader crashes halfway through: a change that was never committed can be forgotten, while a change that was committed is guaranteed to exist on a majority and will be found by the next leader.

The queue of changes

The master handles changes one at a time, in order of priority. Urgent tasks, such as removing a lost node or promoting a replica, come before routine ones, such as cleaning up. Tasks of the same kind are batched together when possible to save work. The list of waiting tasks can be inspected with the pending tasks operation. A long queue is a sign that the master is overloaded or that its changes are slow to be accepted.

Timeouts of publication

A publication must finish within the cluster.publish.timeout setting, which is 30 seconds by default. If a majority does not commit in that time, the master concludes that it cannot lead and steps down, which triggers a new election. A separate setting, cluster.publish.info_timeout, controls after how long an informational log line is written to warn that a publication is taking suspiciously long. Nodes that are slow to apply a committed state are watched by a lag detector, and a node that does not catch up within the follower lag timeout, by default 90 seconds, is removed from the cluster so that it cannot hold everything back.

A failed publication

If the master cannot get a quorum to accept a state, for example because of a network split, it does not commit and it steps down. Followers that never received a commit keep the older, committed state. This is how the cluster avoids exposing half-finished changes, and why the newest committed state is always safe to rely on.

An election begins only when someone notices that the leader is gone. Regular health checks in both directions make that happen.

Once a cluster is formed, its nodes keep checking on each other with a steady heartbeat. There are two kinds of check, and they run in opposite directions.

The settings behind the checks

These are controlled by the settings cluster.fault_detection.leader_check.* and cluster.fault_detection.follower_check.*, each with interval, timeout, and retry count. With the defaults, a node that stops answering entirely is declared failed after about half a minute in the worst case.

flowchart TD
    A["Follower sends a leader check every second"] --> B{"Reply received in time?"}
    B -->|Yes| A
    B -->|No| C["Count a failure"]
    C --> D{"Three failures in a row?"}
    D -->|No| A
    D -->|Yes| E["Follower stops trusting the leader"]
    E --> F["Follower becomes a candidate and starts discovery"]
    F --> G["Pre-vote and election as in the earlier chapters"]

Fast detection when a node shuts down cleanly

If a node is stopped in a normal way, or the process crashes and the operating system closes its network connections, the other nodes notice the closed connection immediately. They do not have to wait for the timeouts. This is why an ordinary restart of a master usually leads to a new leader within a few seconds. The slow case is a machine that freezes or a network that silently drops packets. There, nothing closes, and the other nodes must wait for the timeouts to run out.

A slow node looks like a dead node

The checks cannot tell the difference between a dead node and a node that is alive but too busy to answer within the time limit. A long garbage collection pause, a heavily loaded machine, a virtual machine that was paused during maintenance, or a swapped-out process can all make a healthy node miss enough checks to be removed. This explains why leaders that are also busy data nodes are riskier, and why dedicated master nodes with modest loads are more stable.

What the leader does with a lost node

When the master’s follower checks fail, the master publishes a new cluster state without that node. Shards that lived on the lost node are then handled: replicas on other nodes are promoted to primaries where needed, and missing replica copies are rebuilt elsewhere, though only after a delay controlled by the delayed allocation setting, which is one minute by default, so that a brief restart does not trigger a big data movement.

Do not make the checks too aggressive

Lowering timeouts and retry counts makes failures detected faster, but it also makes the cluster more likely to react to short hiccups. In a cluster where nodes pause occasionally, aggressive settings cause a cycle: a node is removed, rejoins, is removed again, and every step triggers cluster state changes and shard movements. The defaults strike a good balance. Change them only with clear evidence and testing.

Let us follow one failure from the first missed heartbeat to the moment the cluster is fully back to normal.

Imagine a cluster with three dedicated master-eligible nodes, called A, B, and C, and several data nodes. Node A is the elected master. Suddenly, node A’s machine loses power.

sequenceDiagram
    participant A as Node A: old master
    participant B as Node B
    participant C as Node C
    participant D as Data nodes
    A--xB: Power lost, no more replies
    B->>B: Leader check fails, become candidate
    C->>C: Leader check fails, become candidate
    B->>C: Pre-vote request
    C-->>B: Yes
    B->>C: Vote request for a new term
    C-->>B: Vote granted
    B->>C: Publish state naming B as master
    B->>D: Publish state naming B as master
    D-->>B: Joined new master
    A->>B: Later: A restarts and joins as follower

What clients see

During the gap, applications may see errors that mention a missing master, such as “master not discovered” or a cluster block named “no master.” Searches on existing indices often keep working, because they are answered by data nodes using the last known routing information. Writes are rejected by default, since a cluster that cannot change its structure must not accept writes that might depend on it. Good clients retry with a short delay, and the problem disappears within seconds once a new master is elected.

How long does it take?

The no-master block

While a cluster has no elected master, a block prevents operations that could cause harm. The setting cluster.no_master_block defaults to write, which rejects writes while allowing reads based on the last known state. Setting it to all also rejects reads and other operations. The default suits most cases, because it preserves availability for searching while protecting data changes.

Two different kinds of leadership exist in Elasticsearch. Mixing them up leads to wrong conclusions during incidents.

Master election chooses the leader of the cluster. There is a second, separate idea: for every shard, one copy is the primary, and the others are replicas. All new writes to a shard go first to its primary, which then copies them to the replicas. When a node holding a primary fails, one of its replicas is promoted to become the new primary. This is called primary promotion, and it is not an election in the sense of this guide.

Connection between the two

The two ideas are linked through the master. Only an elected master can decide to promote a replica. If both a master and a data node fail at the same time, first a new master is elected, and then that master promotes replicas. In-sync replicas are tracked carefully so that a promoted copy contains every acknowledged write. This is why keeping at least one replica of important indices is a wise design, and why a healthy master is a precondition for repairing the data layer.

Cluster status colors

You may have seen the cluster health colors. Green means all primary and replica shards are assigned. Yellow means all primaries are assigned but some replicas are not. Red means at least one primary is unassigned. These colors describe the state of shards, not the state of the election. A cluster with a perfectly healthy master can still be red if data nodes are missing, and a cluster with no master cannot report health at all until a master is found.

Why the difference matters during incidents

When people say “the master failed” while they mean “a node holding primaries failed,” they start looking in the wrong place. Ask two questions first. Is there an elected master right now? That is a question about elections. Are the shards assigned? That is a question about allocation and promotion. The answers lead to different actions.

Most master-related outages during normal life are caused by human changes, not failures. A few simple habits avoid all of them.

Adding a master-eligible node

To add a master-eligible node to a running cluster, start it with the same cluster name, the seed hosts pointing to existing nodes, and the master role in its roles. Do not set the initial master nodes list on it, because the cluster is already formed. When the node joins, the elected master will include it in the voting configuration if that keeps an odd number of voters. Watch the logs of the new node to see that it joined and check that the cluster state shows it.

Removing a master-eligible node

Follow the safety rule from Chapter 7: stop fewer than half of the voters at once, wait for the automatic adjustment between removals, and use exclusions when removing half or more. Removing nodes carelessly is the number one way to lose a quorum by accident.

Rolling restarts

A rolling restart means restarting nodes one at a time, for example to change a setting or to update software, while the cluster stays online. The steps for master-eligible nodes follow a rhythm.

Order for version upgrades

For upgrades to a newer version, Elastic publishes a specific order in its documentation, and it generally places master-eligible nodes in the sequence that keeps a newer node from leading an older group. Always follow the documented order for your versions, and take a snapshot first.

Never clone a node with its data

Every node stores a unique identity on its data path. Copying a node’s entire data directory to create a second node produces two nodes with the same identity, which confuses membership and can corrupt votes. Create new nodes with empty data directories and let them join normally. If you must move a node to new hardware, move its data path and retire the old machine completely so that the identity exists only once.

Do not wipe the data path of a master-eligible node lightly

A master-eligible node stores its term, its latest accepted state, and its voting memory on disk. If you delete that directory and start the node again, it comes back as a stranger with no memory. In a cluster with a healthy majority, it can rejoin and receive the state. But wiping the data paths of several voters at once can destroy the cluster’s history and leave you with tools of last resort, as described in Chapter 25.

Good master election starts with good architecture. The number, size, and location of master-eligible nodes decide how well the cluster survives trouble.

Common layouts

Spread across failure zones

A failure zone is a group of machines that can fail together, such as a rack, a data center room, or a cloud availability zone. To survive the loss of a whole zone, put the master-eligible nodes in at least three zones, one or more in each, so that the loss of any one zone still leaves a majority.

flowchart LR
    subgraph Z1["Zone 1"]
        M1["Master-eligible node 1"]
        D1["Data nodes"]
    end
    subgraph Z2["Zone 2"]
        M2["Master-eligible node 2"]
        D2["Data nodes"]
    end
    subgraph Z3["Zone 3"]
        M3["Master-eligible node 3"]
        D3["Data nodes"]
    end
    LB["Load balancer or API gateway"] --> D1
    LB --> D2
    LB --> D3
    M1 --- M2
    M2 --- M3
    M1 --- M3

Hardware for master nodes

• Modest CPU. Dedicated masters do little computation, though large clusters need more when the state is big and changes are frequent.
• Enough memory for the state. Heap must hold the cluster state and its copies during updates. Bigger clusters with many shards need more. Use monitoring of heap and garbage collection to size it.
• Fast, reliable disk. The master saves accepted states to disk before it answers, so slow disks slow every change. Small but fast local storage is best.
• Stable network. Low, steady latency between master-eligible nodes matters more than high bandwidth.
• No swapping. A node that swaps pauses unpredictably and can miss heartbeats. Lock memory or disable swap on master nodes.
• Isolation from noisy neighbors. Avoid sharing the machine or the virtual host with heavy workloads.

How many master-eligible nodes?

Three is the usual answer. Five gives more tolerance but a little more coordination work. More than five is almost never useful. Adding data nodes does not mean adding master-eligible nodes: the number of voters should stay small even when the cluster grows to hundreds of nodes.

Separate roles in bigger clusters

In a large cluster, separating roles pays off. Dedicated masters keep leadership calm, dedicated coordinating nodes take the burden of merging big search results, and data nodes focus on shards. Ingest and machine learning roles can live on their own nodes as well. Every heavy activity that leaves the master alone makes elections less likely.

The election itself is cheap. What decides whether the master stays calm is the size of the cluster state and the rate of changes it has to process.

Because there is only one elected master and it handles every structural change one after another, it is a natural bottleneck for control-plane work. This is not a flaw: the design trades some speed for safety. But it means that you must keep the master’s workload reasonable as the cluster grows.

What makes the master busy

Shard count is the biggest lever

Every shard is an entry in the routing table, and every shard change is a state update. A cluster with an enormous number of small shards forces the master to track and update a lot of information for very little data. Elasticsearch has a default guard of one thousand shards per data node for most data nodes, set by cluster.max_shards_per_node. Staying comfortably below the guard, and using fewer, larger shards, is one of the most effective ways to keep the master healthy.

Keep mappings under control

Each index carries its mapping in the cluster state. Indices with thousands of fields multiply that size, and uncontrolled dynamic fields make the state grow with incoming data. Protective measures include strict or limited dynamic mapping, a cap on the number of fields per index through index.mapping.total_fields.limit, and careful design so that data values are never used as field names.

Handle index creation at a healthy pace

Systems that create a new index for every small group of documents, such as one index per customer per day, can create thousands of new indices and shards each day. Using data streams with sensible rollover rules and shared indices for small tenants reduces the number of structural changes the master must handle.

Batching and priorities

The master groups similar tasks and orders tasks by priority. This helps, but it cannot make a huge queue vanish. If tasks arrive faster than they can be processed, the queue grows, and structural changes start to wait. The pending tasks list and the maximum waiting time in the health output show how deep the queue is.

Latency between master-eligible nodes

Each publication needs a round trip to a majority of voters. If master-eligible nodes are far apart, each change takes longer, and the timeouts of fault detection may be reached during ordinary network delays. For this reason, Elastic advises against stretching a single cluster across distant regions. For multi-region designs, use separate clusters connected by cross-cluster features, so that each region has its own quick elections.

Scaling the number of voters

More voters mean more messages per election and per publication, and a slightly bigger chance that one of them is slow. That is why three or five is recommended. The cluster can grow to many hundreds of nodes with the same three voters, as long as the state stays manageable.

Whoever controls the master controls the structure of the cluster. Elections must therefore be protected from strangers, from mistakes, and from misuse.

The election runs over the transport layer, the private channel that nodes use to talk to each other. If that channel is open and unauthenticated, anybody who can reach it may pretend to be a node. A stranger with the master role could join, cast votes, and possibly influence the cluster. Securing elections means securing this channel and the people who may operate it.

Network design

Place master-eligible nodes on a private network segment. Allow the transport port only between nodes of the cluster. Do not expose master nodes to client traffic at all. Clients reach the cluster through a load balancer or an API gateway that points at data or coordinating nodes. This keeps user requests away from the leader and reduces the attack surface.

Operators and privileges

Some operations can affect leadership directly. Voting configuration exclusions, cluster settings that change fault detection, and shard allocation settings all belong to the cluster management privilege. Give this privilege to a small, trusted group, protect their accounts with strong authentication, and record their actions in an audit trail. A mistaken setting by one operator can have the same effect as an attack.

Protecting stored state

The cluster state saved on master-eligible nodes describes the structure of your data: names of indices, fields, and settings. This is often sensitive in itself. Protect it with disk encryption, restrict access to the data directory to the Elasticsearch service account, and treat backups of these directories with the same care as backups of data.

Certificates and elections

Because election traffic depends on certificates, an expired certificate can stop nodes from talking and lead to a leaderless cluster. Monitor certificate expiry on all master-eligible nodes as part of your normal health checks. Where possible, automate renewals.

Audit and change control

Treat changes to master nodes, roles, seed hosts, and voting settings as controlled changes: request, review, apply, and record. A written history helps you find the cause when leadership behaves oddly, and it helps auditors see that the cluster is managed with care.

A stable master is a quiet master. A handful of signals tells you when it stops being quiet, often long before users notice.

Questions the tools answer

Important log lines

Metrics worth charting

• Number of master changes over time. The simplest and best stability measure.
• Term number. A rising term signals repeated elections.
• Pending tasks and maximum wait. Show how deep the master’s queue is.
• Cluster state size and publication time. Show growth of the control-plane workload.
• Node count. A sudden drop or repeated changes point to membership problems.
• Heap and garbage collection on master-eligible nodes. Long pauses are the top cause of unwanted elections.
• Unassigned shards and recovery activity. Show the effect of leadership problems on data.

Alerts that help

Create a small set of alerts. First, an alert when the cluster has no elected master for more than a short period, such as thirty seconds. Second, an alert when the master changes more than a few times in an hour. Third, an alert when pending tasks wait longer than a set limit for several minutes. Fourth, an alert when a master-eligible node reports long garbage collection pauses. Fifth, an alert for the number of nodes falling below the expected count. Each alert should link to a short runbook.

Watching a single node’s view

Some issues appear only from the perspective of one node, for example a node that is cut off from the others and keeps logging “master not discovered.” Central log collection lets you compare what each node saw at the same moment, which usually tells the whole story of an incident in minutes.

No design is perfect. Here is an honest scorecard for the majority-vote election so that you can decide where to invest effort.

The central trade-off: consistency against availability

When a network split happens, a distributed system must choose between staying available on both sides and staying consistent. Elasticsearch’s election chooses consistency for the cluster’s structure. The side with a majority carries on, and the side without one stops changing the cluster. This costs some availability on the minority side, but it keeps the structure of the cluster correct, which is usually the right choice for a system that stores important data.

The second trade-off: fast detection against stability

Quick failure detection restores leadership sooner but reacts to short hiccups. Slow detection is calmer but leaves the cluster leaderless for longer during a real failure. The defaults aim for a balance, and changes should be based on measured behavior of your environment.

The third trade-off: dedicated masters against cost

Dedicated master nodes cost extra machines but give calm, predictable leadership. Sharing masters with data nodes saves money and works for small clusters, but a heavy query can push the master into a long pause and trigger elections. The larger and more critical the cluster, the more dedicated masters pay for themselves.

The fourth trade-off: three voters against five

Five voters survive two simultaneous failures, which matters for very critical services or for zone designs where one zone may fail while another is under maintenance. They also add a little coordination work and cost. Three is enough for most.

When this design is the wrong tool

If you need a service that stays fully writable on both sides of a network split, a majority-based design is the wrong choice, and you would look at systems built for that behavior. For search clusters that store data of record and derive it from other sources, the safe and simple election of Elasticsearch is a good match.

A calm leadership does not happen by luck. These seven habits make elections rare, fast, and boring.

Pattern 2 in action

A company has two data centers, each with one master-eligible node. On paper this gives two voters, which is no fault tolerance, because the loss of either one breaks the quorum of two. The team adds a small virtual machine in a cloud region as a voting-only node. Now there are three voters. If either data center fails, the surviving center plus the tiebreaker form a majority of two out of three, and the cluster continues. The tiebreaker never becomes master and needs almost no resources.

Pattern 7 in action

A platform team runs a failover drill every quarter. They stop the elected master in a staging cluster during a load test, measure how long clients see errors, and check that alerts fire. The first drill reveals that one application does not retry on a missing master error. They fix it. By the third drill, the failover passes without any visible effect. The drill costs an hour and replaces guesswork with evidence.

A pattern for documentation

Write a one-page runbook that lists the expected master-eligible nodes, their zones, how to identify the current master, how to check the voting configuration, and what to do if there is no master. Keep it next to your alerts. During a real incident, nobody wants to think from scratch.

These are the traps that cause most master-related incidents. Each is common and easy to avoid once you know it.

Other frequent mistakes

• Deleting the data directory of master-eligible nodes to “reset” them without understanding the consequences.
• Cloning a node with its data directory and running both copies.
• Using different seed host lists on different nodes, so that some nodes cannot find the leader.
• Following advice from old articles about minimum master nodes on modern versions.
• Ignoring the size of the cluster state until the master falls behind.
• Never testing what happens when the master disappears.

Most master problems fall into a few patterns. Each has a calm recovery, and the last resort is rarely needed.

flowchart TD
    A["Cluster reports no master"] --> B{"Are a majority of the master-eligible nodes running and reachable?"}
    B -->|No| C["Restore the nodes or the network so that a majority can talk"]
    B -->|Yes| D{"Do their logs show that they found each other?"}
    D -->|No| E["Check seed hosts, ports, firewalls, and certificates"]
    D -->|Yes| F{"Do the logs mention a different cluster identity?"}
    F -->|Yes| G["A node bootstrapped on its own: fix bootstrap settings and data paths"]
    F -->|No| H["Look for long pauses, full disks, or a full heap on the voters"]
    C --> I["Only if nodes are permanently lost: consider last resort recovery tools"]

Scenario: a single master fails

This is the normal case, covered in Chapter 15. A majority remains, an election takes seconds, and the cluster continues. No action is needed except to repair the failed node and let it rejoin. Check that alerts fired and that clients coped.

Scenario: a majority of master-eligible nodes is down

If two of three voters are lost, the last one cannot elect a leader alone. The cluster stops changing its structure and rejects writes by default. The right response is to bring the lost nodes back, either by repairing the machines or by restoring the network. When enough voters return, an election happens on its own and the cluster recovers without data loss.

Scenario: a network partition

The side with a majority keeps a master. The other side pauses. Recovery is to repair the network. When the sides can talk again, the paused nodes join the leader and catch up. Nothing needs to be deleted. Review why the network failed, and consider whether your zone layout matches the failure patterns you actually have.

Scenario: garbage collection storms and slow disks

If voters pause for long periods, heartbeats fail, leaders are lost, and elections repeat. Find the pauses in the garbage collection logs and the slow disk operations in system metrics. Move heavy work away from master-eligible nodes, right-size their heap, disable swapping, and use fast local disks.

Scenario: a node bootstrapped a separate cluster

A new node started with the initial master nodes setting and an empty data path may form its own one-node cluster with the same name. It will never join the real one, because the cluster identities do not match. Stop the node, remove the initial master nodes setting, wipe its data path if it holds nothing worth keeping, and start it again with seed hosts pointing to the real cluster.

Scenario: certificate or credential expiry

If transport certificates expire, nodes cannot complete handshakes with each other, and elections fail even though every machine is running. Renew certificates on the affected nodes, and rely on file reloading or restart as needed. Add expiry alerts so that this cannot surprise you again.

Scenario: disk full on a master-eligible node

A master-eligible node needs disk space to save its accepted states. If its disk is full, it cannot persist the state, and it may fail to acknowledge publications or even fail at startup. Free space, and put logs on a different volume so that a noisy log cannot fill the disk of the node.

Last resort: when voters are lost forever

Sometimes a majority of the master-eligible nodes are lost for good, for example because their disks were destroyed. No election can ever succeed, because the surviving voters are too few. Elasticsearch ships an offline tool, elasticsearch-node, that can force recovery in this situation. Its unsafe bootstrap command tells a surviving master-eligible node to form a new voting configuration from itself, and other nodes must be told to detach from their old cluster identity before they can join the new one.

The word unsafe is deliberate. This procedure abandons the safety guarantees of majority voting. It can lose recent committed changes, and it may bring back state that was already superseded. Use it only after you have tried everything else, with a snapshot or a copy of the data directories taken first, following the documentation for your exact version step by step. Often, the safer choice is to restore from a snapshot into a fresh cluster.

Habits that make recovery easier

• Regular snapshots of important indices, tested by restoring them.
• Copies of configuration files and certificates in a safe place.
• A record of which nodes are master-eligible and where they live.
• Runbooks for the no-master case, written and reviewed in calm times.

When something looks wrong, match the symptom to its likely meaning. This table covers the messages and behaviors that people meet most often.

A calm way to debug

Here are four common situations, with rough numbers, showing master election at work. The companies are imaginary, but the patterns are the ones teams meet every day.

A worked example with numbers

Master election is a simple idea with deep safety properties: one leader, chosen by a majority, replaced quickly when it disappears. Here is everything to remember.

Keep in mind that setting names, defaults, and tools can change between Elasticsearch versions, so confirm exact details against the documentation of the release you run. With that habit, and the ideas in this guide, you can keep your cluster’s leadership steady, understand it when it moves, and recover it calmly when it does not.