The Different Node Roles in Elasticsearch
Master, data, ingest, coordinating, machine learning, transform and more. This guide explains what each node role does, why it exists, how the roles work together, and how to combine them into a healthy production cluster.
Picture a large hospital. It does not hire a hundred identical people and hope they sort things out. It has a director who decides who works where, doctors who treat patients, nurses who prepare people before treatment, a reception desk that greets visitors and sends them to the right ward, and a laboratory for special tests. Everyone is a “staff member”, yet each has a clear job. An Elasticsearch cluster works the same way. Every machine in it is a node, and each node is given one or more roles that describe the jobs it is allowed to do. Choosing those roles well is one of the most important design decisions you will make.
1What Is a Node and What Is a Role?
Before we meet the individual roles, we need to be sure about two simple words: node and role.
Elasticsearch is a search and analytics engine. You can run it on a single computer, but in real life you run it on many computers working together. Each running copy of Elasticsearch is called a node. A group of nodes that work together and share the same cluster name is called a cluster. When you store data, the cluster splits it into small pieces called shards and spreads them over the nodes.
A node role is a label that tells a node which kind of work it is allowed to perform. Roles are set in the configuration file of each node. If you do not set any roles at all, the node takes almost every role by default. That is fine for learning or for a tiny cluster, but it becomes a problem when the cluster grows, because one machine is then trying to do many different jobs at once.
Think of a small food stall. One person takes orders, cooks, serves, and washes dishes. That works for ten customers. When the stall grows into a restaurant, you hire a manager, cooks, waiters, and dishwashers. The people are still staff, but now each has a role, and the restaurant runs better because nobody is overloaded.
Why do roles exist?
Different jobs need different resources. Deciding where shards live needs stability and a small amount of memory. Searching billions of documents needs fast disks and lots of memory. Running machine learning needs strong processors. If all of these compete on the same machine, a heavy machine learning job can slow down the very node that keeps the cluster organized. Roles let you give each job its own home.
A node can have one role or many roles. A role does not describe what a computer is. It describes what work the node is permitted to do.
2The Cluster Big Picture
Let us look at a typical mature cluster from above, so every role has a place in your mind before we study it closely.
Applications and dashboards talk to the cluster through a small group of entry nodes. Behind them sit the nodes that hold data, split into groups based on how fresh and how valuable the data is. A trio of manager nodes keeps the whole cluster organized. Special helper nodes handle document preparation, machine learning, and scheduled transformations. Not every cluster needs every group, but the picture below shows the full set.
flowchart TD
CL["Applications and Kibana"] --> CO["Coordinating Nodes"]
CO --> HOT["Hot Data Nodes"]
CO --> WARM["Warm Data Nodes"]
CO --> COLD["Cold Data Nodes"]
CO --> FRZ["Frozen Data Nodes"]
CO --> CONT["Content Data Nodes"]
IN["Ingest Nodes"] --> HOT
MS["Dedicated Master Nodes"] -.->|"cluster state and shard decisions"| HOT
MS -.-> WARM
MS -.-> COLD
MS -.-> FRZ
MS -.-> CONT
ML["Machine Learning Nodes"] --> HOT
TR["Transform Nodes"] --> HOT
Notice the dotted lines from the master nodes. Master nodes do not handle your searches or your documents. They only send decisions and updates about the layout of the cluster. The solid lines show the real traffic of requests and data.
This picture is a maximum, not a goal. A small cluster with three nodes that each hold every role is perfectly reasonable. Add separation only when a real need, such as load, cost, or stability, asks for it.
3All the Roles at a Glance
Here is the full list of roles you can assign, with the short letter that the cat nodes view uses to display each one.
| Role Name | Letter | One-Line Job |
|---|---|---|
| master | m | Can be elected to manage the cluster |
| voting_only | v | Votes in elections but can never be the master |
| data | d | Generic data node that belongs to all data tiers |
| data_content | s | Holds general content that does not age out quickly |
| data_hot | h | Holds newest, most active time-based data |
| data_warm | w | Holds older data that is searched less often |
| data_cold | c | Holds rarely searched data at lower cost |
| data_frozen | f | Holds almost archival data using a local cache over object storage |
| ingest | i | Prepares documents with pipelines before storing them |
| ml | l | Runs machine learning jobs and models |
| transform | t | Runs transforms that reshape data into summary indices |
| remote_cluster_client | r | Connects to other clusters for cross-cluster features |
| No roles (empty list) | – | Coordinating-only node that routes requests |
You can see that the list is longer than the classic three that older tutorials mention: master, data, and ingest. Data roles split into tiers, and helper roles for machine learning, transforms, and remote clusters were added over time as the product grew.
The cat nodes operation shows a role column with these letters, and a marker showing which master-eligible node is the currently elected master. It is the quickest way to see your real layout.
In the next chapters we walk through each role in detail, starting with the most important one: the master.
4The Master-Eligible Node
The master role is the brain of coordination. It decides how the cluster is organized, but it does not do the everyday work of searching.
Every cluster has exactly one elected master at any moment. This elected master is chosen from the nodes that have the master role, which are called master-eligible nodes. The elected master is responsible for cluster-wide actions. It creates and deletes indices, tracks which nodes are part of the cluster, decides which shard is stored on which node, and keeps the official record of all this information, called the cluster state.
What is the cluster state?
The cluster state is a shared notebook that describes the cluster: the list of nodes, the list of indices and their settings and mappings, and the routing table that says where each shard lives. The elected master is the only node allowed to change this notebook. Whenever it changes, the master publishes the new version to all other nodes so everyone shares the same view.
The elected master is like an air traffic controller. Planes carry the passengers (data nodes do the real work), but the controller decides which runway each plane uses and keeps the master schedule. If the controller is confused or overloaded, planes start circling, even though nothing is wrong with the planes.
What the master does not do
When you index a document or run a search, the request goes to data nodes, not through the master. The master is not on the path of normal traffic. This is exactly why a master node does not need much CPU or disk. What it needs is stability: steady memory, a steady network, and freedom from heavy competing workloads.
Index Management
Creating and deleting indices and applying mapping and setting changes.
Node Tracking
Knowing which nodes are alive, accepting new ones, and removing lost ones.
Shard Allocation
Deciding where shards are placed and moving them when nodes join or leave.
Cluster State Publishing
Sending each new version of the cluster state to every node and waiting for confirmation.
Many people assume the master node is a bottleneck for searches. It is not. It becomes a bottleneck only when the cluster state grows very large, for example from an enormous number of indices and shards, because every change must be sent to every node.
5Elections, Quorum, and the Voting Configuration
How does the cluster pick its master, and how does it avoid disaster when the network misbehaves?
Choosing a master is done by voting. The voting is not open to everyone. Elasticsearch maintains a voting configuration, which is the set of master-eligible nodes whose votes count. A decision, such as electing a master or accepting a change to the cluster state, needs agreement from more than half of the nodes in that set. This required majority is called a quorum.
Because more than half is needed, two groups can never both win. If a network problem cuts the master-eligible nodes into two groups, at most one group can hold the majority. The other group knows it does not have enough votes and stops trying to lead. This protection prevents the dangerous situation called split brain, where two parts of a cluster each believe they are in charge and accept conflicting changes.
Imagine a committee of five people that must approve every decision by at least three votes. Even if the committee splits into two rooms because of a locked door, only the room with three or more members can decide anything. The room with two members must wait. No contradictory decisions are made.
flowchart LR
subgraph A["Group with majority"]
M1["Master-eligible 1"]
M2["Master-eligible 2"]
end
subgraph B["Group without majority"]
M3["Master-eligible 3"]
end
A -->|"2 of 3 votes: can elect and publish"| OK["Cluster continues"]
B -->|"1 of 3 votes: cannot lead"| WAIT["Waits for network to heal"]
How many master-eligible nodes do you need?
The rule of thumb is three. With three, the cluster survives the loss of any one of them, because the remaining two are still a majority. With five, it survives the loss of two. With only two, there is no gain over one node, because losing either one leaves you without a majority. Elasticsearch is smart about this: if you have an even number of master-eligible nodes, it quietly leaves one out of the voting set so the count stays odd.
| Master-Eligible Nodes | Votes Needed | Failures Survived |
|---|---|---|
| 1 | 1 | 0 |
| 2 | Behaves like 1 in the voting set | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
In versions before 7.0 you had to set a minimum master nodes number by hand and getting it wrong caused split brain. Modern versions manage the voting configuration automatically as master-eligible nodes join and leave, so that dangerous manual setting no longer exists.
Do not stop half or more of your master-eligible nodes at the same time. Restart them one by one, and wait until the cluster is stable again before moving to the next.
6Voting-Only Nodes and the Tie-Breaker
Sometimes you need an extra vote but not an extra manager. Voting-only nodes solve exactly this.
A voting-only node is a master-eligible node that takes part in elections but can never become the elected master itself. It is always paired with the master role in its configuration. It only says yes or no when votes are counted. Because it never leads, it does not need to be a powerful machine.
The classic use is a tie-breaker across zones. Suppose you have two data centers. If you put one master-eligible node in each, losing one data center leaves you with one out of two votes, which is not a majority. Adding a small voting-only node in a third location fixes this. Now any two of the three locations form a majority, and the cluster survives the loss of a whole data center.
flowchart TD
Z1["Zone A: master-eligible node"]
Z2["Zone B: master-eligible node"]
Z3["Zone C: small voting-only node"]
Z1 --- Z2
Z2 --- Z3
Z1 --- Z3
A referee in a two-team match does not play but makes the decision when the teams disagree. The voting-only node is that referee: it holds a vote, but it never plays on the field.
Voting-only nodes can be very small since they hold no data and never carry the manager workload. They still must have a reliable network link to the other master-eligible nodes.
7Inside the Cluster State
The cluster state is the heart of master work. Knowing what it holds explains many strange performance problems.
The cluster state is a single structure that every node keeps in memory. It contains the list of nodes and their roles, the metadata of every index including its settings and mappings, the templates and pipelines you have defined, and the routing table that maps each shard copy to a node. When you ask any node where a document lives, it answers from this shared structure, without asking the master.
Only the elected master may change the cluster state. Every change, such as a new index, a mapping update, or a shard moving, is put into a queue of tasks. The master batches related tasks, computes the new version of the state, and then publishes it in two steps. First it sends the new state, often only as a small set of differences, to all nodes. Once enough of the voting nodes confirm they have received it, the master tells everyone to commit it and start using it.
sequenceDiagram
participant T as Change Request
participant M as Elected Master
participant V as Master-Eligible Voters
participant N as All Other Nodes
T->>M: Create index or change mapping
M->>M: Compute new cluster state
M->>V: Publish new state
M->>N: Publish new state
V->>M: Acknowledge receipt
N->>M: Acknowledge receipt
M->>V: Commit the new state
M->>N: Commit the new state
Why the size of the state matters
Each update must reach every node, so a larger state takes longer to build, send, and apply. Cluster state grows with the number of indices, shards, fields, and templates. A common trap is mapping explosion, where documents with endless different field names create thousands of fields. Every one of them enlarges the state on every node. This is why Elasticsearch limits the number of fields per index by default, and why you should design mappings on purpose instead of letting anything in.
The cluster state is like a company phone directory that must be reprinted and handed to every employee each time anyone changes desks. A small company does this in minutes. A company with a huge directory spends the whole day printing, and nobody works.
Where is it stored?
The cluster state is kept in memory, but a persistent copy is also written to disk. Master-eligible nodes keep the full cluster-level metadata, and data nodes keep the metadata of the indices they hold. This is what lets a cluster recover after a full restart, and it is also why the role of a node matters for the files that it leaves on disk.
If publishing takes too long, nodes that did not confirm in time can be removed from the cluster. The removed nodes then rejoin, which causes more updates. This loop is a typical cause of unstable clusters with far too many shards.
8The Data Node
Data nodes are the muscles of the cluster. They store shards and do the heavy lifting of indexing and searching.
A data node holds the actual shards that contain your documents. When you add, update, or delete a document, the work is done on the data node that holds the right shard. When you search, each data node searches the shards it holds and returns its part of the result. Because of this, data nodes are usually the most resource-hungry machines in the cluster, needing fast storage, plenty of memory, and good processors.
Data nodes are also where the cluster gets its ability to grow. If you need to store more data or answer more searches, you add data nodes and Elasticsearch spreads shards onto them. This is called horizontal scaling.
If the cluster is a library, data nodes are the shelves and the librarians who fetch books from those shelves. The more shelves you add, the more books you can keep, and the more librarians you have to fetch them in parallel.
Generic data role versus tier roles
The plain data role is a generic role. A node with it can hold any kind of data and belongs to every data tier. It is the simplest choice and is what you get by default. When you need to treat fresh data and old data differently for cost or speed, you instead give nodes one of the specific tier roles that we cover in the next chapter.
Avoid giving one node both the generic data role and a specific tier role. It makes the tier layout confusing, because the generic role already counts for all tiers.
Why Separate Data Nodes
- Their heavy disk and memory load stays away from the master
- You can size hardware specifically for storage and search
- Scaling storage does not require scaling other roles
- Failures of one data node are covered by replica shards
What to Watch
- Too many shards per node harms performance
- Uneven shard sizes cause uneven load
- Disk filling up triggers protective write blocks
- Losing a node causes shard recovery traffic
9Data Tiers: Content, Hot, Warm, Cold, and Frozen
Data has a life. It is exciting when it is fresh and mostly forgotten when it is old. Data tiers let the hardware match that life.
Think about logs from a website. In the first day, they are written constantly and searched all the time by people fixing problems. After a month, they are searched only sometimes. After a year, they are kept only for rules and audits. It would be wasteful to keep year-old logs on the same expensive fast machines as today’s logs. Data tiers let you place data on different kinds of nodes as it ages.
Content Tier
For data that is not time-based, such as product catalogs and user directories. It does not move as it ages. Regular indices outside of data streams land here by default.
Hot Tier
Receives new time-based data. Needs the fastest storage and strongest processors because it handles constant writing and frequent searching.
Warm Tier
Holds recent data that is no longer written to but still searched from time to time. Can use cheaper, larger disks.
Cold Tier
Holds data that is searched rarely. Can rely on searchable snapshots to avoid storing full extra copies, lowering cost.
Frozen Tier
Keeps the data in object storage and uses a local cache on the node. Searches are slower but storage cost is very low.
flowchart LR
N["New time-based data"] --> H["Hot Tier: fast disks, constant writes"]
H -->|"ages"| W["Warm Tier: cheaper disks, fewer searches"]
W -->|"ages"| C["Cold Tier: rarely searched"]
C -->|"ages"| F["Frozen Tier: cache over object storage"]
F -->|"retention ends"| D["Deleted"]
Where does new data go?
New indices that belong to a data stream start in the hot tier. New indices that are not part of a data stream, including many system indices, start in the content tier. Each index carries a tier preference that lists which tiers it may live on, in order, and the cluster places it on the first tier that has nodes available.
A supermarket keeps fresh milk at the front in the cold display, older stock in the back storeroom, and very old bulk goods in a distant warehouse. Customers can still get everything, but the most wanted items are closest to hand.
The cold and frozen tiers depend on searchable snapshots, which require an Enterprise-level license. Hot, warm, and content tiers work with the basic features.
10Moving Data Between Tiers Automatically
Nobody wants to move indices by hand. Index Lifecycle Management does it for you, according to rules.
Index Lifecycle Management, usually shortened to ILM, lets you define phases for an index: hot, warm, cold, frozen, and delete. For each phase you set conditions such as the age of the index or its size, and the actions that should happen, such as rolling over to a fresh index, shrinking, merging segments, or moving to another tier.
When an index enters the warm phase, ILM changes its tier preference so the cluster moves its shards onto warm nodes. The same happens for cold and frozen. You do not tell the cluster which specific node to use. You only state which tier, and the allocation system finds the right nodes.
Hot Phase: Write and Roll Over
The index accepts new documents. When it becomes too large or too old, ILM rolls over and starts a new index, leaving the old one read-only.
Warm Phase: Optimize
The index moves to warm nodes. ILM can shrink the shard count and merge segments so the data uses fewer resources.
Cold Phase: Save Cost
The index is moved to cold nodes, often turned into a searchable snapshot so that extra replicas are no longer needed.
Frozen Phase: Archive with Access
The data is served from object storage through a cache on frozen nodes. It can still be searched, only more slowly.
Delete Phase: Remove
When the retention period ends, the index and its data are deleted.
If a tier has no nodes at all, the tier preference lets the index fall back to the next available tier instead of getting stuck. This is helpful in small clusters, but check where your data really lives so you are not surprised by cost or performance.
11The Ingest Node
Raw data is often messy. Ingest nodes clean and enrich documents before they are stored.
An ingest node runs ingest pipelines. A pipeline is an ordered list of small steps called processors. Each processor changes the document in one way. Examples are splitting a log line into separate fields, converting text to numbers, renaming a field, adding the country that belongs to an IP address, or removing fields you do not want to store.
When you send documents with a pipeline named in the request, the receiving node passes them to an ingest node, which runs the steps and then forwards the finished documents to the data nodes that hold the shards. This means the cleaning work happens before the documents ever reach storage.
Ingest nodes are like the prep kitchen in a restaurant. Vegetables arrive muddy and whole. The prep team washes, peels, and chops them so the chefs at the stove receive clean ingredients ready to cook.
sequenceDiagram
participant App as Application
participant Co as Receiving Node
participant In as Ingest Node
participant Da as Data Node
App->>Co: Send documents with pipeline name
Co->>In: Forward documents to run the pipeline
In->>In: Apply processors one by one
In->>Da: Send cleaned documents to the right shard
Da->>App: Confirm the write
Why use dedicated ingest nodes?
Simple pipelines cost very little. Heavy pipelines with pattern matching, scripts, and enrichment lookups use real processor time. If they run on the same nodes that hold shards, they compete with indexing and searching. Dedicated ingest nodes take that load away from data nodes, and you can add more of them when ingestion grows.
Parsing
Break unstructured text into fields, such as extracting time, level, and message from a log line.
Transforming
Rename, convert, lowercase, split, or remove fields to match the shape you want.
Enriching
Add information from lookups, such as location from an IP address or details from a reference table.
Some teams do preparation outside Elasticsearch, using agents or a separate pipeline tool. Ingest nodes are the built-in option, and they are especially convenient when many different sources send to the same cluster.
12The Coordinating Node
This is the most misunderstood role, because every node already plays it. Coordinating-only nodes simply do nothing else.
Whenever a request reaches a node, that node becomes the coordinating node for that request. A search, for example, is handled in two stages. First the coordinating node sends the query to every relevant shard, which is called scatter. Each shard returns its best results. Then the coordinating node merges those partial answers into one final sorted list, which is called gather, and sends it to the client.
A coordinating-only node is created by giving it an empty list of roles. It holds no data, cannot become master, and runs no pipelines. It only accepts requests, routes them, and combines results. It behaves like a smart load balancer that understands the layout of the cluster.
sequenceDiagram
participant C as Client
participant Co as Coordinating Node
participant S1 as Data Node 1
participant S2 as Data Node 2
participant S3 as Data Node 3
C->>Co: Search request
Co->>S1: Search your shards
Co->>S2: Search your shards
Co->>S3: Search your shards
S1->>Co: Top results
S2->>Co: Top results
S3->>Co: Top results
Co->>C: Merged final answer
A coordinating node is the reception desk of a large company. Visitors do not wander through the building. The receptionist sends each visitor to the right departments, collects what each department says, and hands the visitor one clear reply.
When are they worth it?
The gather step can be expensive for searches with big aggregations, many shards, or large result sets, because the merging uses memory. Dedicated coordinating nodes take this memory pressure off data nodes. They also make a good stable front door for Kibana and for applications, and they keep the internal nodes out of direct reach of clients.
Adding too many coordinating-only nodes can hurt. Every node must acknowledge each cluster state update to the master, so extra nodes add work for the master. In many small and medium clusters, data nodes doing their own coordination are perfectly fine.
13The Machine Learning Node
Machine learning work is heavy, bursty, and different from search. It deserves its own home.
A node with the ml role can run machine learning jobs. These include anomaly detection, which learns what normal looks like for your data and flags unusual behavior, analytics jobs that classify or predict, and trained models used for language tasks such as understanding text. These features use a large part of the machine’s processor and also use memory outside of the normal Elasticsearch memory area.
Because of this, a machine learning job can easily starve a data node of memory or processing power if they share a machine. Dedicated machine learning nodes keep the impact away from indexing and searching, and they let you choose hardware that fits, for example machines with more memory or accelerators for model inference.
Machine learning nodes are like a research lab attached to a hospital. The lab runs long, complicated experiments using special equipment. You would not run those experiments in the emergency room, because they would slow down urgent care.
Anomaly Detection
Spot sudden spikes, drops, or strange patterns in metrics and logs without writing fixed rules.
Data Frame Analytics
Train and apply models on stored data for classification, regression, or outlier detection.
Trained Model Inference
Run models on incoming or stored documents, such as text understanding or embeddings for semantic search.
Many machine learning features need a paid subscription level. Also remember that the memory these jobs use lies outside the main heap, so plan machine memory by the total needs of the models, not by heap size alone.
14The Transform Node
Transforms turn detailed data into useful summaries. The transform role is the worker that keeps them running.
A transform reads data from one or more source indices, groups or pivots it, and writes a new, smaller index. For example, you might turn millions of individual purchase events into one summary row per customer with totals and last purchase date. Transforms can run once, or continuously, updating the summary as new data arrives.
Nodes with the transform role run these jobs. A transform searches the source indices and indexes into the destination, so it uses both search and write resources. On a small cluster this is no problem. On a busy cluster, dedicated transform nodes prevent these background jobs from competing with user searches.
A transform is like an accountant who reads a mountain of daily receipts and keeps a neat monthly summary sheet. Instead of counting all receipts each time someone asks, you just look at the summary.
Customer 360 View
Combine orders, returns, and support events into one profile per customer that dashboards can query quickly.
Latest State Index
Keep only the most recent record for each device out of a huge stream of sensor readings.
15The Remote Cluster Client Role
Sometimes one cluster needs to talk to another. This role gives a node permission to make that connection.
Elasticsearch supports two features that reach across clusters. Cross-cluster search lets you run one query that searches indices in several clusters and gives one merged answer. Cross-cluster replication lets one cluster copy indices from another, which helps with disaster recovery and with bringing data closer to users in other regions.
A node needs the remote_cluster_client role to open those connections to remote clusters. By default, nodes have it. But when you set roles by hand and list only the ones you want, you must remember to include it on the nodes that need it, or these features will fail with confusing errors.
This role is like being allowed to make international calls on a phone plan. The phone itself works fine locally, but without that permission, calls to other countries simply do not go through.
After moving to an explicit role list, teams often discover that cross-cluster features stopped working. The cause is usually a missing remote cluster client role on the nodes that coordinate those requests or run the replication work.
16How a Request Travels Through the Roles
Now let us put the roles together and follow a real request from start to finish.
Writing a document
An application sends a new document. The request lands on a coordinating node. If the request names an ingest pipeline, the document goes first to an ingest node. After that, the coordinating node works out which shard the document belongs to by using a formula on its identifier, and sends it to the data node that holds the primary copy of that shard. That node writes the document and forwards it to the nodes that hold the replica copies. When enough copies confirm, the client gets its answer.
Searching
The search reaches a coordinating node. It sends the query to one copy of every relevant shard, and those copies might be on hot, warm, cold, or frozen nodes. Each returns its best results. The coordinating node merges them and returns the final answer. During all of this, the master node is not involved.
Creating an index
This is a cluster-level change, so the request is forwarded to the elected master. It updates the cluster state, publishes the new state to everyone, and decides where the new shards should be placed. Data nodes then create the shards.
flowchart TD
R["Client request"] --> CO["Coordinating role"]
CO -->|"needs a pipeline"| IN["Ingest role"]
IN --> DA["Data roles"]
CO -->|"search or write"| DA
CO -->|"index creation or settings change"| MA["Elected master"]
MA -->|"new cluster state"| DA
MA -->|"new cluster state"| CO
| Request Type | Roles Involved | Master Involved? |
|---|---|---|
| Index a document | Coordinating, optionally ingest, data | No |
| Search or aggregation | Coordinating, data | No |
| Create or delete an index | Master, then data | Yes |
| Change a mapping | Master, data | Yes |
| Run a machine learning job | ML, data for reading and writing | Only to assign the job |
| Cross-cluster search | Coordinating with remote client role | No |
17The Life of a Node: Discovery, Joining, and Leaving
Roles only make sense if nodes can find each other. Here is how a node becomes part of a cluster.
Start and Read Configuration
The node starts, reads its cluster name, its roles, and a list of seed addresses, which are the addresses of some other nodes it can contact first.
Discover Peers
It contacts the seed addresses and learns about other nodes, especially the master-eligible ones.
Find or Elect a Master
If a master is already elected, the node asks to join. If there is no master, the master-eligible nodes hold an election.
Receive the Cluster State
Once accepted, the node receives the current cluster state and learns the full layout of the cluster.
Take On Work
Depending on its roles, it now receives shards, pipelines, jobs, or requests.
Stay Healthy or Leave
The master and the followers check each other regularly. A node that stops answering is removed and its work is reassigned.
There is one special step for a brand new cluster. The very first time, the initial master-eligible nodes must be told the names of the nodes allowed to form the first voting configuration. This is called bootstrapping. After the cluster has formed, this setting should be removed, and nodes that join later must not use it.
Leaving the bootstrap setting in place on nodes, or setting it on new nodes joining an existing cluster, can cause them to form a separate new cluster. Use it only for the very first start of a brand new cluster.
18Changing Roles on a Running Cluster
Roles are not permanent. But changing them on a live cluster needs care, especially for roles that hold state.
Adding a role to a node is usually simple: change the configuration and restart the node. Removing a role is trickier. A node that holds shards and loses its data role, or a node that has cluster metadata and loses its master role, is refusing to start in some cases, because Elasticsearch will not silently leave valuable data on the disk of a node that no longer manages it.
Drain the Node
Before removing the data role, tell the cluster to move all shards away from that node using allocation filtering, and wait until the node holds none.
Check Cluster Health
Confirm the cluster is green so that all replica copies exist elsewhere.
Change the Configuration
Edit the role list and restart the node. If it still has leftover data on disk, use the repurpose tool that ships with Elasticsearch to clean it safely.
Verify
Look at the node list to confirm the new role letters, remove the allocation filter, and watch the cluster settle.
When moving the master role to new dedicated nodes, add the new master-eligible nodes first, let the voting configuration update, then remove the role from the old ones one at a time. Never leave the cluster with fewer master-eligible nodes than it needs for a majority.
19Legacy Settings and Moving to the Modern Role List
Older tutorials and older clusters describe roles in a different way. Knowing the old style helps you read them and migrate.
In early versions, each role was switched on or off by its own true or false setting, such as one for master, one for data, and one for ingest. There was no simple way to describe tiers, and adding a new kind of node meant adding yet another setting. Starting with version 7.9, Elasticsearch introduced a single list of roles for each node. The old separate settings were deprecated and are no longer accepted in modern versions.
| Old Way | Modern Way |
|---|---|
| One true or false setting per role | One list that names all roles of the node |
| Only master, data, ingest, and a few helpers | Content, hot, warm, cold, frozen, voting-only, transform, and more |
| Manual minimum master nodes setting | Automatic voting configuration |
| Tiers imitated with custom attributes | Built-in tier roles and tier preference |
| Coordinating-only made by turning everything off | An empty role list |
Migrating tier layouts
Many older clusters built hot and warm layouts using custom node attributes and index settings that required indices to live on nodes with a matching attribute. Modern versions can move such indices to the built-in tiers. The usual approach is to give existing nodes the new tier roles, let lifecycle policies use the tier-based actions, and then remove the old attribute rules once all indices follow the new preference.
A guide that tells you to set a minimum number of master nodes, or to switch off roles with separate settings, was written for an older design. Check the version of the guide against your cluster before you follow it.
20Hardware Sizing for Each Role
Because each role has a different appetite, each needs different hardware. Sizing by role is where cost savings appear.
| Role | Main Need | Storage | Notes |
|---|---|---|---|
| Dedicated master | Stability, moderate memory | Small and reliable | Memory grows with cluster state size |
| Voting-only | Reliable network | Very small | Can be a tiny machine |
| Hot data | Fast CPU and fast disk | SSD or better | Handles constant writes and searches |
| Warm data | Balanced memory and disk | Larger, cheaper disks | Mostly read workloads |
| Cold data | Disk capacity | Large, cost-efficient | Fewer replicas if using searchable snapshots |
| Frozen data | Local cache and network to object store | Fast local cache disk | Small local disk relative to data served |
| Ingest | CPU | Minimal | Scales with pipeline complexity |
| Coordinating | Memory and CPU | Minimal | Memory for merging large results |
| Machine learning | CPU and large memory | Minimal | Memory outside the main heap |
| Transform | CPU and memory | Minimal | Reads and writes at the same time |
Memory rules of thumb
Elasticsearch runs on the Java Virtual Machine, and it needs a memory area called the heap. A common rule is to give the heap no more than half of the machine memory, and to stay below the size at which Java loses a memory-saving trick, which is roughly thirty gigabytes. The rest of the memory should be left to the operating system, which uses it as a file cache. This cache is what makes searches fast.
The number of shards on a node is tied to its heap, because every shard has a fixed overhead. Too many small shards is a very common problem and hurts both data nodes and the master. Aim for fewer, right-sized shards and follow the current sizing guidance for your version.
21Cluster Layout Patterns: Small, Medium, and Large
The right role layout depends on the size and importance of your cluster.
Small Cluster: Three All-Rounder Nodes
Three nodes, each holding every role. All three are master-eligible, so one can fail without losing the cluster. Data, ingest, and coordination happen on the same machines. This is simple and cost-effective for modest workloads and for getting started.
Medium Cluster: Dedicated Masters, Shared Workers
Three small dedicated master nodes, plus a group of data nodes that also do ingest and coordination. The masters are protected from heavy workloads, while the data nodes are used fully. This is a very common and balanced choice.
Large Cluster: Tiered With Helper Roles
Dedicated masters, hot, warm, cold, and frozen tiers, dedicated ingest nodes for heavy pipelines, coordinating nodes for Kibana and applications, and separate machine learning nodes. Every group scales independently, and the cost is optimized by tier.
Multi-Zone Cluster: Spread for Resilience
Nodes are spread across three availability zones. One master-eligible node sits in each zone, and shards are allocated with awareness of zones so replicas land in a different zone than their primaries. Losing a whole zone does not lose data or the majority.
flowchart LR
S["Small: 3 nodes with all roles"] --> M["Medium: 3 dedicated masters plus data nodes"]
M --> L["Large: tiers plus ingest, coordinating and ML"]
L --> Z["Multi-zone: everything spread across 3 zones"]
The message is that role separation is a journey. Start simple and split roles only when measurements show a need.
22How Roles Guide Shard Allocation
Roles decide which nodes are allowed to hold shards. A set of rules called deciders then choose the exact node.
Shard allocation is the job of the elected master. For every shard copy it asks a simple question: which node should hold this? The answer comes in two parts. First, the allocator lists which nodes could hold the shard, and here roles matter: only nodes with a data role can hold shards, and an index with a tier preference can only use nodes that belong to the wanted tier. Second, a group of deciders each vote on whether a particular node is acceptable.
Same Shard Rule
A primary shard and its replica are never placed on the same node, because that would defeat the purpose of a copy.
Disk Threshold
Refuses nodes that are too full and starts moving shards away when a node crosses the higher watermarks.
Awareness
Spreads copies of a shard over zones or racks so that losing one zone does not lose every copy.
Filtering
Lets you include or exclude nodes by name or attribute, which is how you drain a node before maintenance.
Tier Allocation
Allows an index only on nodes of the tiers in its tier preference, so hot data stays on hot nodes.
Throttling
Limits how many shard recoveries run at the same time on a node, so the cluster does not flood itself with copying.
flowchart TD
S["Shard needs a home"] --> R{"Does the node have a data role and the right tier?"}
R -->|"No"| X["Node is not a candidate"]
R -->|"Yes"| D{"Do all deciders agree?"}
D -->|"No"| X
D -->|"Yes"| B["Balancer picks the least loaded acceptable node"]
B --> P["Shard is placed and recovers its data"]
When a node leaves
If a node disappears, the cluster does not immediately rebuild all its shards elsewhere. It waits for a short delay, by default one minute, in case the node was only restarting. This avoids a large and useless copying storm during quick restarts. If the node does not return in time, replicas are promoted and new replica copies are built.
A warehouse manager who sees one worker step out does not immediately hire a replacement and move all that worker’s boxes. She waits a few minutes to see whether the worker is only getting coffee.
When a shard stays unassigned, the allocation explain operation tells you which decider said no and why. It is the best first tool for shard placement puzzles, including role and tier mismatches.
23Zones, Attributes, and Awareness
Roles say what a node does. Attributes say where it lives. Together they build resilient clusters.
Every node can carry custom labels called node attributes. A very common one names the availability zone or rack where the node runs. You then tell the cluster that this attribute is an awareness attribute. From then on, the allocation system tries to place the copies of a shard in different zones.
With two copies of each shard, one primary and one replica, and nodes in three zones, the cluster makes sure the two copies never share a zone. If a whole zone goes dark, at least one copy of every shard survives elsewhere. Awareness is therefore the bridge between the role layout and the physical layout of your data center or cloud region.
| Layer | Question It Answers | Example |
|---|---|---|
| Node role | What work may this node do? | Hot data node |
| Node attribute | Where is this node? | Zone B, rack 4 |
| Awareness | Should copies avoid the same place? | Primary in zone A, replica in zone B |
| Forced awareness | Should the cluster wait rather than crowd one zone? | Do not put both copies in zone A if zone B is down |
Forced awareness and capacity
Normally, if a zone fails, the cluster tries to rebuild the missing copies in the remaining zones. That can overload them. With forced awareness you tell the cluster the full list of zones, and it will not crowd the surviving zones with extra copies. This protects the survivors, but it also means that you must have enough spare capacity in each zone to carry the load when one is missing.
Three Zones, One Master Each
Place one master-eligible node and a share of data nodes in each zone. Losing one zone leaves two master-eligible nodes, which is a majority, and each shard still has a copy in a living zone.
Two Zones Plus a Tie-Breaker
When only two zones hold data, keep one master-eligible node in each and add a small voting-only node in a third place. Shard copies are spread over the two data zones.
Zone awareness is not free. To survive the loss of one of three zones, each remaining zone must be able to carry roughly half again its normal load. Size the nodes with that in mind, or the survivors will be overwhelmed at the worst moment.
24Thread Pools, Back-Pressure, and Resource Protection
Every node protects itself when it gets too busy. Understanding these protections tells you which role is struggling.
A node does not run every task at once. It has several thread pools, each a group of workers dedicated to one kind of work, such as searching, writing, getting single documents, or managing the cluster. Each pool has a fixed number of workers and a waiting line called a queue. When the queue is full, the node refuses new work of that kind and sends back an error that tells the client to slow down and try again. This refusal is called a rejection.
A bank has a fixed number of tellers and a waiting area with limited chairs. If the chairs are full, the guard politely asks new customers to come back later. Rejections are that guard, and they protect the bank from collapsing under a crowd.
Which role feels which pool
| Work | Pool Under Pressure | Typical Role Affected |
|---|---|---|
| Writing and bulk indexing | Write pool | Hot data nodes |
| Searching shards | Search pool | All data nodes |
| Merging results | Search coordination work | Coordinating nodes |
| Cluster tasks and state updates | Cluster management work | Master node |
| Snapshots | Snapshot pool | Data nodes |
Other guards
Besides thread pools, nodes use circuit breakers. These watch memory use and stop a request that would use too much, instead of letting the node run out of memory and crash. There is also an indexing pressure limit that caps how much memory in-flight writes may use, by default a small share of the heap. When it is reached, further writes are rejected until earlier ones finish.
These guards are good news. A rejection is a message that says this role is at its limit. If you see rejections rise on hot data nodes, you probably need more hot nodes or better bulk sizes. If you see breaker trips on coordinating nodes, you probably have giant aggregations that need redesign, or more memory for that role.
Good clients react to rejections by waiting and retrying with growing delays. Clients that retry instantly can turn a small overload into a large one.
25Capacity Planning by Role
Good role design is useless if nodes are too few or too small. Here is a calm, step-by-step way to plan.
Measure the Data
Estimate how much data arrives per day, how large a document is after indexing, and how long each age group must be kept. This gives the total data per tier.
Account for Copies
Multiply by one plus the number of replicas. A cold tier that uses searchable snapshots may need fewer local copies, which changes the arithmetic.
Add Headroom
Keep free disk well below the watermarks and leave space for merges and shard movement. Also keep spare capacity to survive the loss of a node or a zone.
Choose Shard Sizes
Aim for shards large enough to be efficient but small enough to move and recover quickly. Tens of gigabytes per shard is a common target for time-based data.
Size Each Role’s Nodes
Use the hardware needs from the sizing table. Decide how many nodes each tier needs by dividing its data volume by what one node can comfortably hold and search.
Test With Real Data
Load a realistic sample, run your real queries, and measure indexing speed, search latency, and heap use. Numbers from your own data beat any general rule.
Sizing only for storage. A node may have enough disk but not enough processors or memory for the searches you need, especially on the hot tier. Size for storage and for workload, and take the larger answer.
26Rolling Restarts and Upgrades by Role
The order in which you touch nodes matters, and roles decide that order.
A rolling restart means restarting one node at a time so the cluster keeps working. Before each restart, you prepare the cluster so it does not begin copying shards around the moment a node disappears. You then stop the node, make your change, start it again, and wait for the cluster to be healthy before moving on.
A rolling upgrade is the same idea across versions, and the order follows the roles. Older nodes must be able to talk to newer nodes during the transition, but newer nodes cannot always send everything to older ones. That is why the guidance is to upgrade the parts that are least central first, and the master last.
Prepare
Take a fresh snapshot, check the cluster is green, read the upgrade notes for your version, and pause features such as machine learning if the notes advise it.
Data Nodes by Tier
Upgrade data nodes tier by tier, starting with the frozen tier, then cold, then warm, and hot last among the data tiers. The colder tiers carry the least live pressure.
Other Non-Master Nodes
Upgrade ingest, machine learning, transform, and coordinating-only nodes next.
Master-Eligible Nodes Last
Upgrade the master-eligible nodes one at a time, waiting for stability after each. The elected master moves to an already upgraded node as needed.
Finish and Verify
Resume paused features, restore normal shard allocation settings, and confirm health and roles across the whole cluster.
Before restarting a data node, limit shard movement so that only primary shards are moved. After the node returns, allow full allocation again. This avoids a wasteful reshuffle when the node is only gone for a minute.
27Cost Optimization With Roles
Roles are your main tool for spending money only where it earns value.
The largest cost in most clusters is the data nodes, so most savings come from putting data on the cheapest tier that still meets its needs. Fresh data that people search all day belongs on hot nodes. Data that nobody has searched in weeks does not. Moving it to warm, cold, or frozen nodes can cut the cost of storage by a large factor.
Ways to Save
- Use lifecycle policies to move old data to cheaper tiers on time
- Use searchable snapshots on cold and frozen tiers to avoid extra replica copies
- Shrink and merge indices before they move to warm nodes
- Right-size dedicated master and voting-only nodes to small machines
- Turn off replicas for data that can be rebuilt from a source
- Delete data as soon as retention rules allow
False Savings
- Running masters on tiny machines that pause under load
- Removing all replicas from important live data
- Putting frequently searched data on the frozen tier
- Skipping test restores of snapshot-backed tiers
- Ignoring the cost of network traffic between zones
Also remember the cost of people. A very complicated layout with many tiny role groups takes more time to run. A cost saving that produces a week of firefighting is not a saving. Optimize in steps, starting with tiers and retention, since these give the biggest reward for the smallest complexity.
28A Troubleshooting Guide by Symptom
When something goes wrong, first ask which role is really suffering. This table maps symptoms to the likely role and the first thing to check.
| Symptom | Likely Role | First Checks |
|---|---|---|
| Master keeps changing | Master-eligible | Long garbage collection pauses, network drops, overloaded master, master log |
| Index creation is slow | Master | Pending tasks, cluster state size, number of shards |
| Writes are rejected | Hot data | Write pool rejections, bulk size, disk speed, indexing pressure |
| Searches are slow only sometimes | Data or coordinating | Hot threads, heavy aggregations, competing background jobs |
| Kibana dashboards time out | Coordinating and data | Number of shards touched, heap use, slow queries |
| Shards stay unassigned | Data | Allocation explain, tier preference, disk watermarks, filters |
| Machine learning job is pending | Machine learning | Free memory on machine learning nodes, job memory needs |
| Documents fail pipeline steps | Ingest | Failure counts per processor, pipeline logic, enrichment data |
| Old data searches are slow | Cold or frozen | Cache hit rate, object storage latency, query time range |
| Nodes drop out and rejoin | Any | Network stability, memory pressure, cluster state publishing time |
A doctor does not treat a fever without asking where it hurts. Roles are the map of the body. Once you know which role is in pain, you look at the right instruments instead of guessing.
Work from the outside in. Start with cluster health, then the node list with roles and heap, then the specific role’s metrics, and only then the logs. Change one thing at a time, and note the effect.
29Roles in Managed Services and on Kubernetes
You do not always edit configuration files yourself. Managed platforms turn the same ideas into simpler choices.
On a managed service such as Elastic Cloud, you usually choose which groups of nodes your deployment needs from a menu: hot, warm, cold, frozen, master, coordinating, and machine learning. The platform sets the roles for you, keeps master-eligible nodes spread across zones, and can add or remove capacity in each group. You still make the same design choices, such as how many zones and how much of each tier, but you do not touch the individual role lists.
On Kubernetes, the Elastic operator lets you describe your cluster as a set of node groups. Each group has a name, a count, a list of roles, resource requests, and storage. One group might be three small master nodes, another a set of hot nodes with fast volumes, another a set of warm nodes with large cheaper volumes. The operator then creates and updates the pods in a safe order.
Tiers as Building Blocks
You choose size and zones for each tier, and the platform handles the role assignment and the spread across zones.
Node Groups
Each group of pods has its own roles and resources, so scaling a role means changing one number.
Full Control
You edit the role list on each node yourself and own the sizing, upgrades, and failure planning.
Building your own cluster is like building a house from raw materials. A managed service is like renting an apartment: the walls and plumbing are done, and you decide the furniture. In both, you still need to understand rooms, which here are the roles.
Even when the platform handles the role lists, it cannot know your workload. You must still decide how much hot capacity you need, how long to keep data on each tier, and whether machine learning deserves its own nodes.
30Security and Roles
Roles are not only a performance tool. They also shrink the area an attacker can reach.
Nodes talk to each other over a transport connection and talk to clients over an HTTP connection. Modern Elasticsearch enables security features by default, including encrypted communication between nodes. Every node must present a valid certificate to join the cluster, so a stranger cannot simply start a node and join.
Limit Client Access
Only coordinating nodes, or a load balancer in front of them, should be reachable by applications. Keep masters and data nodes on an internal network.
Encrypt Node Traffic
Keep transport encryption on and manage certificates with a clear rotation plan.
Protect the Masters
Since masters guard the cluster state, restrict who can reach them and keep them free of extra workloads and plugins.
Least Privilege for Users
Give users and services only the roles and privileges they need. Cluster-level privileges should be for administrators only.
Node roles describe what a machine does. Security roles describe what a person or an application may do. They share a word but are entirely different ideas, and both should be planned deliberately.
31Monitoring, Logging, and Metrics by Role
Each role has its own warning signs. Watching the right numbers on the right nodes prevents surprises.
| Role | What to Watch | Why It Matters |
|---|---|---|
| Master | Pending tasks, cluster state update time, heap usage, election events | Slow updates delay the whole cluster |
| Data | Heap pressure, disk usage, search and write rejections, shard count | Rejections mean the node is overloaded |
| Hot | Indexing rate, merge activity, disk speed | Slow ingestion often starts here |
| Frozen | Cache hit rate, fetch latency to object storage | Low hit rate means slow searches |
| Ingest | Pipeline processing time, failed documents | Slow or failing pipelines lose or delay data |
| Coordinating | Heap and memory breaker trips | Big result merges can exhaust memory |
| Machine learning | Job memory, node assignment, job states | Jobs can fail to start when memory is full |
| Transform | Checkpoint delay, failures | Summaries fall behind the source |
Disk watermarks
Data nodes protect themselves as the disk fills up. At the low watermark, the cluster stops placing new shards on the node. At the high watermark, it starts moving shards away. At the flood stage, indices with shards on that node become read-only for writing so the disk cannot completely fill. By default these are around eighty-five, ninety, and ninety-five percent. Alert well before they trigger.
Useful views
The nodes view shows roles, heap use, and load per node. The cluster health view shows if shards are unassigned. The pending tasks view shows if the master has a backlog. The hot threads view shows what the busiest threads of a node are doing, which is a powerful way to find why a node is slow. Node stats show thread pool rejections and circuit breaker trips, which are early warnings of overload.
The master node log records elections, node joins and leaves, and slow cluster state updates. If your cluster ever shows strange behavior, reading the master log around that time is usually the first useful step.
32Failure Scenarios: What Happens When a Node Dies?
Understanding failures shows why roles are designed the way they are.
The Elected Master Fails
The remaining master-eligible nodes notice, hold an election, and choose a new master, usually within seconds. The cluster keeps serving searches and writes during this short time for most shard-level work, and cluster-level changes wait until the new master is ready.
Majority of Masters Lost
With no majority, no master can be elected. By default, the cluster then rejects writes while it has no master, and it may still serve searches from what it last knew. The fix is to restore enough master-eligible nodes to regain a quorum.
A Data Node Fails
The master notices and promotes replica shards to primaries where needed. Then it creates new replicas on other nodes to restore redundancy. During this time there is extra network and disk activity. If replicas did not exist, that data would be unavailable until the node returned.
An Ingest Node Fails
Requests that need pipelines are sent to the remaining ingest nodes. If you had only one, requests using pipelines fail until it returns. Have at least two when ingestion matters.
A Coordinating Node Fails
Clients using a load balancer or several addresses retry on another node. Nothing is lost, because coordinating nodes hold no data.
A Machine Learning Node Fails
Jobs running there are reassigned to another machine learning node with enough memory, and resume from saved state where possible. With no free capacity, they wait in a pending state.
flowchart TD
F["A node stops responding"] --> Q{"Which role?"}
Q -->|"Master"| E["Election among remaining eligible nodes"]
Q -->|"Data"| P["Promote replicas then rebuild copies"]
Q -->|"Ingest or coordinating"| R["Clients retry on other nodes"]
Q -->|"ML or transform"| A["Jobs reassigned to another such node"]
33Design Patterns That Work
Teams running Elasticsearch at scale keep arriving at a few dependable habits.
Three Dedicated Masters, Spread Across Zones
Three small, dedicated master-eligible nodes, one per zone, protect the cluster from both node failure and zone loss, and keep coordination away from workload spikes.
Hot-Warm-Cold-Frozen With ILM
Use lifecycle policies to move time-based data down the tiers automatically. This matches spending to the value of the data without manual work.
Dedicated Coordinating Front Door
Put Kibana and applications behind a small pool of coordinating-only nodes so heavy result merging never touches data nodes, and internal nodes stay unexposed.
Isolated Helper Workloads
Run machine learning and transforms on their own nodes so that background jobs never slow user searches.
Tie-Breaker Voting-Only Node
When only two zones are available for data, add a tiny voting-only node in a third location so the majority survives a zone loss.
Infrastructure as Code for Roles
Define roles in templates, such as Kubernetes node sets or configuration management, so every node of a type is built the same way and changes are reviewed.
34Anti-Patterns to Avoid
These mistakes appear in real incidents over and over. Knowing them saves you from learning the hard way.
Problem
The cluster has only two master-eligible nodes, or an even number of them.
Why It’s Harmful
Two nodes give no more fault tolerance than one, because losing either leaves no majority. An even number wastes one node in the voting set.
Correct Approach
Use three master-eligible nodes, or five for very large clusters. Add a small voting-only node if you need a tie-breaker.
Problem
In a large busy cluster, master-eligible nodes also hold data and run heavy searches and machine learning.
Why It’s Harmful
Memory pressure or long pauses on that node can make the master unresponsive, causing elections and cluster-wide instability that looks unrelated to the true cause.
Correct Approach
Move master duties to dedicated, lightly loaded nodes once the cluster grows beyond a small size.
Problem
Every node in a large cluster is master-eligible.
Why It’s Harmful
More voters make elections and cluster state publishing slower and give no extra safety past a small number of nodes.
Correct Approach
Limit master-eligible nodes to three or five, and give all other nodes explicit non-master roles.
Problem
Adding many coordinating-only nodes by default, hoping to speed up everything.
Why It’s Harmful
Each extra node adds cluster state work for the master, and in many clusters it adds one more hop without real benefit.
Correct Approach
Add coordinating-only nodes only when measurements show that merging is hurting data nodes or when you need a protected front door.
Problem
Running machine learning jobs and heavy transforms on the same nodes that serve user searches.
Why It’s Harmful
Long jobs use processors and memory that searches need, causing slow responses at unpredictable times.
Correct Approach
Give these workloads dedicated nodes, and size them by the needs of the jobs.
Problem
Stopping several master-eligible nodes at the same time during maintenance.
Why It’s Harmful
If half or more go down together, the cluster loses its majority and stops accepting cluster-level changes.
Correct Approach
Restart one master-eligible node at a time and wait for the cluster to be stable before the next.
35Best Practices and Common Mistakes
A short checklist to keep beside you when designing or reviewing a cluster.
Best Practices
- Set roles explicitly in production instead of relying on defaults
- Keep three master-eligible nodes, spread across zones
- Match tier hardware to the age and value of data
- Isolate heavy background roles such as machine learning
- Use lifecycle policies to automate tier movement
- Keep shard counts and sizes under control
- Test failure of each role before it happens for real
Common Mistakes
- Leaving every node with every role on a growing cluster
- Forgetting the remote cluster client role after listing roles by hand
- Giving too little memory to dedicated coordinating nodes
- Ignoring disk watermarks until writes are blocked
- Changing roles without draining the node first
- Creating too many tiny shards that overload masters
Whenever you plan a change, ask two questions. Which role does this work belong to? And what happens to the cluster if that role’s node disappears right now?
36Advantages and Trade-Offs of Role Separation
Separating roles is powerful, but it is not free.
Advantages
- A stable master that is protected from workload spikes
- Hardware fitted to each job, which lowers cost
- Independent scaling of storage, ingestion, search, and machine learning
- Smaller blast radius when a node fails
- Better security by limiting which nodes are exposed
- Easier troubleshooting since each node has a clear purpose
Disadvantages / Trade-offs
- More nodes to run, patch, and monitor
- Higher minimum cost, since small dedicated nodes still cost money
- More complex configuration and more places to make mistakes
- Underused capacity if a role is provisioned too generously
- Extra planning is needed to keep each role redundant
The best answer is neither extreme. A cluster of all-rounders wastes stability at scale. A cluster split into too many tiny specialized groups wastes money and attention. Let real measurements, such as heap pressure, rejections, and latency, tell you when the next split is justified.
37Real-World Scenarios
These stories show how role choices play out in practice.
A Growing Online Store
A shop starts with three all-role nodes. During a holiday sale, heavy category aggregations slow down checkout searches. The team adds two coordinating-only nodes for the storefront and moves dashboards to a separate pool. Result merging no longer competes with product lookups, and the slowdowns disappear.
A Logging Platform With Long Retention
A company keeps a year of application logs. Fresh logs live on hot nodes with fast disks, the following weeks move to warm nodes, and older logs go to searchable snapshots on cold and frozen tiers. Storage cost falls sharply, and engineers can still search old incidents when needed.
A Security Analytics Team
A security team runs anomaly detection on network logs. At first the jobs run on data nodes, and analysts complain about slow dashboards every evening. Moving the jobs to two dedicated machine learning nodes removes the slowdown, and the team can size those nodes for the models they actually run.
A Cluster Losing a Zone
A service runs across three zones with one master-eligible node in each and zone-aware shard allocation. One zone loses power. Two master-eligible nodes remain, so the cluster keeps its majority, replicas in the other zones are promoted, and users notice only a short pause.
A Kubernetes Deployment
A platform team defines separate node sets for masters, hot, warm, and ingest nodes with an operator on Kubernetes. Each set has its own resource requests, storage class, and scaling rules, making it easy to add capacity to exactly the role that needs it.
38Myths About Node Roles
Some beliefs about roles spread widely and cause real problems. Let us correct the most common ones.
The Belief
All searches and writes pass through the master node, so it must be the biggest machine.
The Reality
Searches and writes go through coordinating and data nodes. The master handles cluster-level changes only.
What To Do
Make masters stable and lightly loaded, not large. Put your budget into data nodes.
The Belief
More master-eligible nodes always means more safety.
The Reality
Safety comes from having a majority. Beyond three or five, extra voters only slow down elections and state publishing.
What To Do
Keep a small odd number of master-eligible nodes and spread them across zones.
The Belief
Coordinating-only nodes make every search faster.
The Reality
They help when merging results is heavy or when you need a protected front door. Otherwise they add a hop and extra cluster overhead.
What To Do
Add them after you measure a coordination problem, not before.
The Belief
If a node has the ingest role, all documents must pass through it.
The Reality
Only documents sent with a pipeline are processed by ingest nodes. Documents without a pipeline go straight to the data nodes.
What To Do
Use ingest nodes only where you actually apply pipelines, and size them by pipeline work.
The Belief
Replicas make snapshots unnecessary, and roles do not affect backups.
The Reality
Replicas protect against node loss, not against deletion or disaster. Also, cold and frozen tiers depend on snapshot repositories, so backups are part of your role design.
What To Do
Keep regular snapshots, and treat the repository as a critical part of any tiered layout.
39Frequently Asked Questions
No. A small cluster with three nodes that share all roles works well. Dedicated masters become important as load and cluster size grow, because they protect the coordination of the cluster from heavy work.
An empty role list creates a coordinating-only node. It holds no data, cannot become master, and runs no pipelines. It only routes requests and merges results.
Yes, and in small clusters it is common. In larger clusters it is safer to separate them, so that heavy data work cannot make the master unstable.
No. Master-eligible nodes are those allowed to be elected. Only one of them is the elected master at any time. The others stand by and vote.
Three is the smallest number that tolerates one failure with a true majority. Two is no better than one, and an even number wastes a node in the voting set.
The plain data role is generic and belongs to all tiers. Tier roles such as hot, warm, cold, and frozen let you place data on nodes that match its age and importance.
By default they go to the content tier, which is meant for data such as catalogs and directories that does not age out on a schedule.
Not necessarily. If your data is already in its final shape, you can skip pipelines. Ingest nodes are useful when you want Elasticsearch itself to parse, clean, or enrich documents.
No. Kibana can connect to any suitable node. Many teams still choose dedicated coordinating nodes as a protected, stable entry point for large deployments.
Most likely the remote cluster client role was left out of the explicit list. Add it to the nodes that need to connect to remote clusters.
You can on small setups, but heavy jobs compete for processor and memory with searches and indexing. Dedicated machine learning nodes are the safer choice for serious workloads.
It adds a vote to elections without ever becoming master. It is most often a small tie-breaker in a third zone.
Use the cat nodes view. It shows a role column with letters for each role, and it marks which master-eligible node is currently elected.
Move all shards off the node using allocation filtering, wait until it is empty and the cluster is green, then change the role. If leftover data remains on disk, use the repurpose tool to clean it safely.
Not directly, because searches do not go through it. But an overloaded or unstable master can slow cluster changes and cause shard movement problems that indirectly affect performance.
40Summary and Key Takeaways
Node roles are how an Elasticsearch cluster divides its work. Master-eligible nodes keep the cluster organized and agree on decisions by majority vote. Data nodes, split into content, hot, warm, cold, and frozen tiers, store shards and do the heavy searching, with each tier matching the value of data at a certain age. Ingest nodes prepare documents, coordinating nodes route requests and merge answers, and the machine learning, transform, and remote cluster client roles support special features. Like the hospital in our opening story, a cluster works best when every group has a clear job, enough resources to do it, and a backup ready if someone is missing.
Key Takeaways
- A role describes permitted work — nodes can hold one role or many, and unset roles default to nearly everything, which suits only small clusters.
- Masters coordinate, they do not serve traffic — they manage the cluster state, so they need stability more than raw power.
- Use three master-eligible nodes — an odd number with a majority vote prevents split brain, and a small voting-only node can act as a tie-breaker across zones.
- Tiers match hardware to data value — content, hot, warm, cold, and frozen nodes let lifecycle policies move data as it ages and cut cost.
- Every node coordinates, few should do only that — dedicated coordinating nodes help with heavy result merging and protected access, but too many add overhead.
- Isolate heavy helpers — ingest, machine learning, and transform workloads deserve their own nodes once they compete with searches.
- Change roles carefully and monitor by role — drain nodes before removing roles, watch the signals each role produces, and rehearse the failure of every role.