We begin with the everyday meaning of the word, and then look at the special way Elasticsearch splits the job into two parts.

Scaling in Plain Words

A cluster is a group of Elasticsearch servers, called nodes, that work together to store and search data. Capacity is how much work and data the cluster can hold, measured in things like disk space and memory. Scaling means changing that capacity: making it larger when the workload grows, or smaller when it shrinks. Autoscaling means the changes happen automatically, without a person watching graphs and clicking buttons.

Two Jobs, Two Owners

In Elasticsearch, autoscaling is really two jobs. The first job is deciding how much capacity is needed. Elasticsearch itself does this. It knows better than anyone how many shards it holds, how full its disks are, how fast data is arriving, and how much memory its machine learning jobs need. From this knowledge, it produces a recommendation: “for this group of nodes, you need at least this much storage and this much memory.”

The second job is acting on that recommendation: starting new servers, making servers larger, or removing servers. Elasticsearch does not do this part directly, because it does not own the machines. Instead, a separate program called an orchestrator, such as Elastic Cloud or the Kubernetes operator for Elastic, reads the recommendation and changes the cluster to match.

What Autoscaling Is Not

Autoscaling is not an instant response to a sudden burst of search traffic. Adding a node takes minutes, and moving data onto it takes longer. It is not a magic fix for a badly designed index or an overloaded query. It is not free, since every node it adds has a price. And, importantly, it does not simply watch processor usage the way some web server autoscalers do. We will come back to this last point in its own chapter, because it is the most common misunderstanding.

To see why autoscaling exists, we look at what goes wrong when a cluster is built at one fixed size and left alone.

Guessing the Future

When a team builds a cluster, someone has to guess how much data it will hold in six months or a year. That guess is hard to get right. Data volumes depend on how many customers arrive, how verbose the logs become, how many new features are launched and how long data must be kept. A small change in any of these can double the storage need.

Two Ways to Be Wrong

If the guess is too high, you over-provision: you pay for servers and disks that stay empty. If the guess is too low, you under-provision: the cluster runs out of room, and problems begin. Both mistakes cost money, one through waste and the other through outages.

A handful of words appear in every autoscaling discussion. Learn them here and the later chapters will feel simple.

The Simple Chain of Ideas

Here is how these words fit together in one sentence. A policy is attached to a group of nodes with certain roles. The policy uses one or more deciders. Each decider calculates a required capacity. Elasticsearch compares it with the current capacity, and shares the result. The orchestrator reads it and changes the number or size of nodes. Read that chain twice, and the rest of the guide becomes much easier.

There is more than one way to add capacity, and more than one way to decide when to add it. Knowing the options explains many design choices in Elasticsearch.

Vertical or Horizontal

Vertical scaling means making each node bigger, with more memory, more disk or a faster processor. Horizontal scaling means adding more nodes of the same kind. Vertical scaling is simple, but every machine has a maximum size, and one huge node is a bigger risk if it fails. Horizontal scaling can grow much further and spreads risk, but it needs Elasticsearch to move shards onto the new nodes, which takes time.

Reactive or Proactive

A reactive approach waits until the need is real. It looks at what data exists right now and makes sure there is room for it. A proactive approach looks ahead. It notices how quickly data has been arriving lately and adds room for what is about to arrive. Elasticsearch offers both styles, in separate deciders, and they are often used together.

Up Is Easy, Down Is Hard

Growing is usually safer than shrinking. When you add capacity, nothing has to be evacuated. When you remove capacity, the data on those nodes must first be moved elsewhere, and every move puts load on the cluster. For this reason, automatic scale-down is handled much more carefully than scale-up, and in some cases platforms are conservative about doing it at all. We look at this in detail later.

Let us look at the parts that work together during autoscaling, from users at the top to the machines at the bottom.

The Parts of the System

flowchart TD
    U["Users on web and mobile"] --> G["API Gateway"]
    G --> LB["Load Balancer"]
    LB --> IS["Indexing Service"]
    LB --> SS["Search Service"]
    IS --> CN["Coordinating node"]
    SS --> CN
    CN --> HOT["Hot tier data nodes"]
    CN --> WARM["Warm tier data nodes"]
    CN --> COLD["Cold tier data nodes"]
    M["Elected master node with autoscaling deciders"] --> API["Capacity API: required and current capacity"]
    HOT --> M
    WARM --> M
    COLD --> M
    ORCH["Orchestrator: Elastic Cloud or Kubernetes operator"] --> API
    ORCH --> INFRA["Infrastructure: virtual machines and disks"]
    INFRA --> HOT
    INFRA --> WARM
    INFRA --> COLD
    MON["Monitoring and alerts"] --> M
    MON --> ORCH

Reading the Diagram

Start at the top. Users reach the gateway and load balancer, which hand requests to the indexing and search services. These call the cluster through a coordinating node, which routes work to data nodes in the hot, warm and cold tiers. That is the everyday path of data.

Now follow the autoscaling loop. Data nodes report their state, such as how full their disks are, to the elected master node. The master node runs the deciders and publishes their results through the capacity interface. The orchestrator polls this interface regularly. When the required capacity is larger than the current capacity, it changes the infrastructure underneath, and new or larger nodes join the tiers. Monitoring watches both sides, so people know when decisions are made and when the platform has acted.

Follow one decision from the moment data starts growing to the moment new capacity is serving it.

sequenceDiagram
    participant App as Applications
    participant DN as Hot data nodes
    participant MN as Master node with deciders
    participant OR as Orchestrator
    participant PL as Platform infrastructure
    App->>DN: Write a growing stream of data
    DN->>MN: Report disk use and shard sizes
    MN->>MN: Deciders calculate required capacity
    OR->>MN: Ask for capacity recommendation
    MN-->>OR: Required capacity is larger than current
    OR->>PL: Add or enlarge hot nodes
    PL-->>DN: New capacity joins the cluster
    DN->>DN: Shards are placed or moved onto new capacity
    OR->>MN: Ask again later
    MN-->>OR: Required capacity now matches current

Why Decisions Are Repeated

The decision is not made once. The orchestrator keeps asking, again and again, because data keeps changing. Each answer is a fresh snapshot of what the cluster needs at that moment. This makes the system self-correcting: if a forecast was too generous or a burst ends, later answers reflect that.

Policies are how you tell Elasticsearch what to watch and how. They are the main thing you configure.

What a Policy Contains

An autoscaling policy has three parts. The first is a name, which is how you refer to it. The second is a list of roles, which decides which nodes the policy applies to, such as the nodes that hold hot data. The third is a set of deciders, each with its own optional settings. Together, they say: “For nodes with these roles, use these methods to work out how much capacity is needed.”

One Policy Per Group of Roles

Each group of node roles can have one policy. Typical setups have a policy for the hot tier, one for warm, one for cold, one for frozen and one for machine learning nodes. Nodes that have no matching policy are not autoscaled. This is deliberate: you decide which parts of the cluster are managed automatically and which stay under manual control.

Who Writes Policies

On Elastic Cloud and similar platforms, the platform creates and manages the policies for you when you switch autoscaling on, and you mostly set limits such as a maximum size for each tier. When you run the pieces yourself, for instance with your own orchestrator, you create policies through Elasticsearch’s autoscaling interface. In either case, understanding what the policy says helps you understand why the cluster grew.

Deciders are the small brains of autoscaling. Each one watches for one kind of need. Here is what each of them does, in plain language.

Reactive Storage: Making Room for the Present

The reactive storage decider is the workhorse. It knows how big every shard is, and it knows the rule that Elasticsearch stops putting shards on nodes whose disks are too full. It therefore calculates the minimum total storage so that every shard can be placed while keeping nodes below that limit. If the shards you have today would not fit, this decider says so.

The key word is “today”. It uses only the data that already exists. It does not guess about tomorrow. That makes it accurate and conservative, and it also means it can be a little late when data arrives very fast.

Proactive Storage: Making Room for the Near Future

The proactive storage decider adds a forecast. It looks at how fast data streams and their backing indices have been growing recently and estimates how much more data will arrive within a short window of time. It then requires enough storage for both the present data and this expected new data. The length of the forecast window is a setting, and it is kept short on purpose, since forecasts get less trustworthy the further ahead they look.

Think of the proactive decider as a driver who watches the speed of the car ahead and starts to slow down before the brake lights come on. The reactive decider is the driver who only brakes when the car ahead is very close.

Frozen Deciders

The frozen tier holds data as searchable snapshots, which are indices whose real data lives in cheap object storage while the frozen nodes keep a small local cache. Because the data is not fully on local disks, the sizing rules differ. Frozen deciders look at how much data must be served and how many shards are involved, and they ask for enough local cache and memory to serve them well. They also make sure that at least one frozen node exists whenever frozen data does.

The Machine Learning Decider

Machine learning jobs and deployed models need memory, and that memory is not shared easily. The machine learning decider adds up what every job and model needs and requires machine learning nodes with enough total memory, and a single node large enough for the largest job. We give this its own chapter later.

Combining Deciders

When a policy has several deciders, each produces its own required capacity, and Elasticsearch takes the largest of them. This is the safe choice, because a group of nodes must meet every requirement at once. If the reactive decider says 8 terabytes and the proactive says 9, the policy requires 9.

Autoscaling and data tiers work together. Understanding the tiers explains why different parts of the cluster grow at different speeds.

The Journey of Data Through Tiers

Not all data is equally valuable at every age. Yesterday’s logs are searched constantly. Last year’s logs are searched rarely. Elasticsearch lets you place data on different groups of nodes, called data tiers, according to its age and importance. The hot tier holds new data and gets the fastest, most expensive storage. The warm tier holds older data that is still searched from time to time. The cold tier holds data that is rarely searched, on cheaper storage. The frozen tier holds data that is almost never searched, kept mostly in cheap object storage.

flowchart LR
    IN["New data arrives"] --> HOT["Hot tier: fast disks, heavy indexing"]
    HOT --> WARM["Warm tier: older data, lighter searches"]
    WARM --> COLD["Cold tier: rarely searched, cheaper disks"]
    COLD --> FROZEN["Frozen tier: searchable snapshots, local cache"]
    FROZEN --> DEL["Deleted after retention period"]
    P1["Autoscaling policy for hot"] -. "sizes" .-> HOT
    P2["Autoscaling policy for warm"] -. "sizes" .-> WARM
    P3["Autoscaling policy for cold"] -. "sizes" .-> COLD
    P4["Autoscaling policy for frozen"] -. "sizes" .-> FROZEN

Why Each Tier Grows Differently

The Hot Tier Is the Usual Star

The hot tier is where new data lands, so it is where surges hit first. Autoscaling on the hot tier usually matters most, and it is where the proactive storage decider earns its place. Warm and cold tiers change more slowly, since they receive data only as it ages out of the tier above.

Storage autoscaling is closely tied to three disk safety limits inside Elasticsearch. Knowing them makes the recommendations easy to understand.

The Three Watermarks

A watermark is a level of disk usage, given as a percentage of the disk, at which Elasticsearch takes protective action. By default there are three:

• Low watermark, 85 percent by default: Elasticsearch stops placing new shards on a node whose disk is fuller than this.
• High watermark, 90 percent by default: Elasticsearch tries to move shards away from the node to other nodes that have more room.
• Flood stage, 95 percent by default: Elasticsearch makes indices that have shards on the node read-only for writes, so that no further data can be added until space is freed.

How Autoscaling Uses the Watermarks

The storage deciders do not ask for the disk to be filled to the brim. They ask for enough storage that all shards fit while keeping each node below the low watermark. The requirement is therefore larger than the raw size of the data. If you hold 1 terabyte of shards, the requirement will be noticeably more than 1 terabyte, so the disks are never pushed to the line.

This surprises many people who compare the recommendation with the sum of their index sizes and think it is too large. The difference is the safety room that Elasticsearch needs in order to work smoothly. Shards also need free space for merges and for moving between nodes, which temporarily makes copies.

Large Shards and Node Size

Another effect is that the largest shard determines the smallest useful node. If one shard is 400 gigabytes, no node with a smaller usable disk can ever hold it. Autoscaling therefore also reports a required capacity per node, not just in total. Very large shards can push the recommended node size up, and this is one reason to keep shards to a healthy size.

Machine learning workloads are a different kind of guest in the cluster. They need memory, not disk, and they arrive and leave in bursts.

What Machine Learning Nodes Do

Elasticsearch can run anomaly detection jobs, which learn what normal looks like for a stream of data and flag unusual behavior. It can also run data frame analytics jobs and host trained models, for example models that add meaning to text. These tasks run on nodes with the machine learning role, and each one needs a certain amount of memory while it is open or deployed.

How the Decider Thinks

The machine learning decider adds up the memory that all open jobs and deployed models require, plus a small amount of overhead for the node itself. It also checks that at least one node is big enough for the largest single job or model, since a job cannot be split across nodes. From these numbers, it derives the total and per-node memory that the machine learning nodes must have.

Scaling Down Is More Natural Here

Machine learning workloads come and go. A job is opened, runs and later closes. When jobs close, the memory they used is free again, so the requirement drops. The decider supports scaling down after a delay, which avoids removing capacity the moment a job closes and then re-adding it minutes later. The delay acts as a cooling-off period. Nodes that do not hold shards of data are also easier to remove, since there is nothing to evacuate.

Jobs That Wait

A related idea is lazy nodes. Normally, if a job needs to start and there is no room on any machine learning node, it fails to open. With a setting for lazy machine learning nodes, the job can instead wait until capacity appears. Autoscaling then sees the waiting job, calculates the capacity it needs and asks the platform to add it. This lets a job start “before” the machine exists, which is very convenient for bursty workloads.

The calculation is part of Elasticsearch, but the action needs an orchestrator. Which orchestrator you have decides how much of this works for you out of the box.

Why Self-Managed Is Harder

The Elasticsearch calculation only makes a recommendation. In a self-managed setup, you must build the program that reads the recommendation, creates or removes servers and disks, joins them to the cluster with the right roles and handles failures along the way. That is real engineering. Most teams that want autoscaling choose a platform that already includes the acting side.

Limits Are Your Safety Rope

Whatever the platform, you set limits. A minimum makes sure capacity never drops below a level you need, for example to keep enough nodes for high availability. A maximum stops runaway growth from surprising your budget. Autoscaling works between these limits. If the requirement rises above your maximum, the platform will not go beyond it, and the cluster can still run out of room, which is why alerts on hitting the maximum are so important.

The details differ by platform, but the general order of steps is the same everywhere.

Start Small and Learn

A wise first step is to enable autoscaling on one non-critical tier or one test deployment. Watch how it behaves for a few weeks: how often it changes, how big the steps are and whether your limits make sense. The habits and dashboards you build there will carry over to the more important parts of the cluster.

Autoscaling reacts to data. Lifecycle management shapes the data. Together, they keep the cluster steady and the bill under control.

A Quick Look at the Two Helpers

A data stream is a structure for time-based data such as logs and metrics. Behind a single name, it manages a series of hidden backing indices and creates a new one as the current one fills. Index Lifecycle Management, or ILM, is a feature that moves indices through stages as they age. It can roll over to a new index when the current one reaches a size or age, move older indices to warm and cold tiers, and finally delete them.

Why Lifecycle Rules Come First

Storage autoscaling responds to the total amount of data. Lifecycle rules decide how much data there is, in which tier, and for how long. If you keep everything on the hot tier forever, autoscaling will happily buy more and more expensive hot storage. If you move data down the tiers and delete it on schedule, the same autoscaling will buy far less. Lifecycle rules are the steering wheel, and autoscaling is the engine.

Healthy Shards Make Better Decisions

Rollover conditions keep shards at a sensible size. Predictable shard sizes make the storage deciders’ work more accurate and make it easier to place shards on nodes. A single enormous shard, or thousands of tiny ones, both cause trouble: the first forces very large nodes, and the second wastes memory. As a rule of thumb, many teams aim for shards in the range of tens of gigabytes, but you should check current guidance from Elastic for your version and use case.

The most common mistakes come from expecting autoscaling to do things it was never designed for. This chapter clears up those expectations.

It Does Not Watch Query Traffic

Elasticsearch autoscaling is driven by capacity needs, mainly storage and memory. It is not driven by how many searches per second arrive or how hot the processors are. If a sudden crowd of users sends heavy queries, and CPU usage rises, the storage deciders will see no change, because the data has not grown. The cluster will not automatically add search capacity for that reason.

This is the single biggest difference from the autoscaling that many people know from web servers, where more traffic means more machines. If you need to scale for search throughput, you must plan for it separately, for example by adding replicas and nodes ahead of known events, or by using scheduled changes, or by using a platform that offers other kinds of scaling. Check what your platform supports today.

It Is Not Instant

A new node must be created, started, joined to the cluster and filled with shards. Minutes are typical, and larger data moves take longer. Autoscaling is therefore for trends over hours and days, and for smoothing out surprises, not for millisecond reactions.

It Cannot Fix Bad Design

• Too many small shards waste memory, and adding nodes only hides the waste at a higher price.
• Heavy, unbounded queries can overload a node whatever its disk size.
• Missing retention rules make data grow forever, and autoscaling will keep paying for it.
• Mapping explosions, where thousands of fields are created by accident, use memory in ways that storage deciders do not measure.

It Does Not Override Your Limits

Autoscaling works inside the minimum and maximum you set. It will not spend money beyond your maximum, even if the calculation says more is needed. This protects your budget, and it also means that a maximum set too low can quietly cause the exact outage you wanted to avoid.

Growing a cluster is fairly gentle. Shrinking it is delicate. Understanding why explains many of the rules you will meet.

Why Scale-Up Is Easy

When capacity is added, nothing has to be moved out of the way. New nodes join, Elasticsearch places new shards on them and may rebalance a few existing ones, and the load spreads out. Even if something goes a little wrong, the old nodes are still there and still holding the data.

Why Scale-Down Is Hard

To remove a node that holds shards, Elasticsearch must first copy those shards to other nodes and only then remove the node. Every shard that moves means reading it from disk, sending it over the network and writing it elsewhere. If you shrink a busy hot tier at a bad time, this extra work can slow down indexing and searching just when you least want it.

flowchart TD
    A["Required capacity falls below current capacity"] --> B["Wait for a cooling-off period"]
    B --> C{"Still lower after the wait?"}
    C -- "No" --> Z["Keep current capacity"]
    C -- "Yes" --> D["Choose which node to remove"]
    D --> E["Move its shards to other nodes"]
    E --> F["Wait for shards to finish moving"]
    F --> G["Remove the empty node"]
    G --> H["Check cluster health and disk levels"]

How Platforms Handle It

Because of this, platforms are more cautious about scaling down data nodes than about scaling up. Some tiers may only scale up automatically, leaving shrinking to a deliberate action by a person, while workloads that hold no data, such as machine learning nodes, are easier to scale down. Details differ by platform and version, so check the current documentation for what your platform does automatically.

Cooling-Off Periods

To prevent “flapping”, where a cluster grows and shrinks repeatedly in a short time, scale-down decisions use a delay. The requirement must stay low for a while before capacity is removed. Think of a thermostat that does not switch the heater on and off every few seconds. A short delay avoids wasteful changes and keeps the cluster stable.

Autoscaling is one option among several. Each approach fits a different situation, and comparing them helps you choose wisely.

These Are Not Rivals

Most production systems combine approaches. A team might use autoscaling for storage growth, scheduled scaling before a known sale for search throughput, and manual review for large structural changes. The skill is in matching each kind of demand to the tool that fits it.

Autoscaling helps a cluster grow, but the way the cluster is designed decides how well it grows.

Shards Are the Building Blocks

Everything in scaling comes back to shards. Elasticsearch places whole shards on nodes, so the number and size of shards decide how evenly data can spread. Too few large shards make balancing hard and force big nodes. Too many small shards waste memory, because each shard carries overhead in the node’s memory whether it holds a lot of data or a little.

The Cost of Rebalancing

When new nodes join, Elasticsearch may move existing shards to even out the load. This rebalancing uses network and disk on both the sending and receiving nodes. Elasticsearch limits how many shard moves happen at once and how fast, so that everyday work is not starved. These limits keep the cluster responsive, and they also mean that a large scale-up can take a while to fully settle.

Speed of Ingest Versus Speed of Scaling

Think of two speeds racing each other: how fast data arrives, and how fast new capacity becomes available. As long as capacity arrives faster than the data does, the cluster stays healthy. If a huge burst arrives faster than the scaling process can react, disks can fill up before help arrives. To stay ahead, keep a safety margin of free space and use the proactive decider on tiers that receive bursts.

Each scaling change moves through a set of stages, from a quiet cluster to a resized one. Knowing them helps you read your dashboards.

flowchart LR
    S["Steady: capacity matches need"] --> G["Growing: need rises"]
    G --> R["Recommendation: required is above current"]
    R --> P["Planning: orchestrator checks limits"]
    P --> A["Acting: nodes created or enlarged"]
    A --> J["Joining: new nodes enter the cluster"]
    J --> B["Balancing: shards placed and moved"]
    B --> S
    P --> L["Blocked by maximum limit"]
    L --> AL["Alert: needs human decision"]

What You See at Each Stage

During growth, you would see free disk space falling and possibly the first low-watermark warnings. During the recommendation stage, the capacity interface shows a gap. During acting, the platform’s activity log records a plan being applied and nodes being created. During balancing, shard relocation counts rise and then settle. After balancing, disks fall to a comfortable level and the gap closes. If the gap does not close, something is blocking the process, and it is time to investigate.

Keep a Timeline of Events

Record each scaling event with its time, tier, size before and after, and the reason. Over a few months, this record becomes a rich picture of how your data grows. It helps with forecasting budgets, with spotting unusual jumps and with explaining changes in cost to the people who pay the bill.

A short story with numbers makes the ideas concrete. Here is an invented example of a logging cluster and how autoscaling handles a normal month and a bad night.

The Setting

A streaming service stores application logs in a cluster with a hot tier and a warm tier. On a normal day, the logs add about 500 gigabytes to the hot tier. Data stays hot for two days, then moves to warm, where it stays for thirty days before deletion. Each index has one replica. All figures here are invented to illustrate the reasoning and are not from a real company.

Sizing by Hand First

These numbers show what the storage deciders are doing behind the scenes. They add the space for replicas and also the room that Elasticsearch needs to breathe. The recommended capacity is noticeably higher than the sum of index sizes.

A Normal Month

With autoscaling on and reasonable limits, the warm tier grows day by day for the first thirty days as data accumulates, and then levels off when deletion starts to match arrival. The platform adds warm capacity in a few steps during that first month. The hot tier stays flat, since data leaves after two days. Cost tracks real need, with no large idle reserves.

A Bad Night

One night, a failing service floods the logs, and volume jumps to eight times normal, about 4 terabytes per day at the peak rate. The proactive decider sees the fast growth and raises the hot requirement within minutes. The platform adds hot capacity, and new nodes join. Disk use on hot peaks near 75 percent and then eases. By morning, the incident is fixed, volume returns to normal and, two days later, the extra data has moved to warm. The warm tier grows a little to hold it, and thirty days later, the data is deleted.

The Bill

The extra hot capacity existed only for a short time, since the surge data moved out after two days. The warm tier held the surge data for thirty days, then dropped back. Without autoscaling, the team would have needed either permanent hot capacity for the worst night, wasting money every day, or a manual scramble at three in the morning.

Automatic growth is wonderful until the bill arrives. Guardrails keep the growth inside a budget you have chosen.

Where the Money Goes

Cluster costs come from four main places: the servers or virtual machines, the disks, the data moved across networks or between zones, and the licenses. Autoscaling changes the first two directly, and it can change the third indirectly, since more nodes can mean more replication traffic. Understanding each helps you predict what growth will cost.

Setting the Maximum Wisely

The maximum is a balancing act. Set it too low, and a real surge can hit the ceiling and cause an outage. Set it too high, and a runaway process, such as a logging loop, could grow the cluster to a size you cannot afford before anybody notices. A useful method is to set the maximum to your worst realistic need plus a margin, and to set an alert at seventy or eighty percent of the maximum, so humans get time to react.

Autoscaling creates and removes machines automatically, and automation needs strong access controls. Security must grow along with the cluster.

New Nodes Must Be Trusted Nodes

Whenever a new node joins, it becomes part of the cluster and can receive copies of your data. The joining process must therefore be secured. Nodes should use encrypted connections between them, using TLS, the same technology that puts the padlock in a web browser, and should prove their identity with certificates. Platforms that manage autoscaling usually handle this for you. In self-managed setups, your automation must issue and install certificates for every new node.

Protect the Automation

The orchestrator has power. It can create machines, which costs money, and it can delete them, which can affect data. Give it only the permissions it needs, and protect its credentials. Use the least privilege rule: every person or program receives no more access than its job requires. Store secrets in a proper secrets manager, rotate them on a schedule and log everything the orchestrator does.

Access to the Autoscaling Interface

Creating and changing autoscaling policies requires special cluster permissions, and reading the capacity recommendation needs another. Grant them to the orchestrator and to a small number of administrators. A person who can change a policy can change how big and how expensive your cluster becomes, so treat that ability like a financial control as well as a technical one.

Data on New Disks

Every new disk that receives shards must be encrypted in the same way as the old ones, if you use encryption at rest. Make encryption a default in your machine templates, so it cannot be forgotten. When nodes are removed, their disks should be wiped or destroyed according to your rules, since they may hold copies of sensitive data.

Autoscaling helps availability by preventing full disks, and it also adds moving parts that can fail. Plan for both sides.

Keep Enough Nodes to Survive

High availability means the system keeps working when parts fail. Elasticsearch achieves it with replicas and with nodes spread across zones. Your minimum limits should always leave enough nodes, in enough zones, for the cluster to survive the loss of one zone or one node. A minimum that allows the cluster to shrink to a single node during a quiet night would remove that safety.

When the Automation Itself Fails

If the orchestrator stops working, the cluster does not stop. It continues to run at its current size, and the capacity interface continues to make recommendations that nobody carries out. That is why it is valuable to alert on the gap between required and current capacity, so people know when the automation has stopped listening. Have a manual procedure ready for adding capacity in an emergency, and rehearse it.

Autoscaling changes your cluster without asking, so you need to know what it is doing and whether it is working.

The Numbers That Matter

Where to Find Them

The capacity interface itself is the primary source for required and current capacity, and it includes the reasoning of each decider. Elasticsearch’s cluster and node statistics report disk use, shard counts and relocations. Kibana’s monitoring pages, and the activity logs of your platform, show the history of changes. Send these into the same dashboards and alerting tools that your team already uses, so autoscaling alerts arrive in a familiar place.

Alerts That Help

• Low watermark reached on any node. It means autoscaling is late or blocked.
• Required capacity above current for more than a set time. It means the recommendation is not being applied.
• Tier at seventy or eighty percent of its maximum. It gives people time to raise the limit or reduce data.
• Unusual growth in data volume. A sudden jump often reveals a runaway source before it costs much money.
• Frequent scale changes. It suggests flapping or a noisy workload.

Autoscaling solves real problems and also creates a few new ones. A fair look at both helps you decide when it is worth using.

Key Trade-offs to Decide

Safety versus cost. A higher minimum and a proactive decider keep you safer during surges, but they hold more capacity when things are quiet. Decide how much safety your workload deserves.

Automation versus control. Fully automatic scaling saves effort, but it takes decisions out of human hands. Many teams keep automatic scale-up and require a person to approve large scale-downs or to raise maximum limits.

Big nodes versus many nodes. Larger nodes are simpler and need fewer shards to move, but they take a bigger share of the cluster when one fails. More, smaller nodes spread the risk, but add coordination overhead.

Simplicity versus precision. One policy per tier is easy to understand. Fine-tuning many settings might save a little money, but makes the system harder to reason about. Start simple, and add detail only when measurements show a need.

Teams that have run autoscaled clusters repeat the same good habits, and the same avoidable mistakes.

Patterns That Work

Anti-Patterns to Avoid

A short list of habits that separate smooth autoscaling deployments from painful ones.

Common Mistakes

• Assuming traffic-based scaling. Expecting more search capacity when query load rises, when the feature follows storage and memory.
• Forgetting the license or platform requirements. Discovering late that a tier or environment does not support autoscaling.
• Comparing the recommendation with raw index sizes. Forgetting replicas and watermark room, and concluding the number must be wrong.
• Leaving minimums too low. Allowing the cluster to shrink to a size that cannot survive the loss of a zone.
• Ignoring shard layout. Growing nodes to hide the cost of oversized or too many shards.
• No plan when the ceiling is hit. Having no alert or procedure for the moment the maximum limit blocks growth.

Turning autoscaling on is a day’s work. Living with it for years is the rest. These are the routine chores that keep it healthy.

Reviewing Limits Regularly

Data grows, budgets change and workloads evolve. Every quarter, look at each tier’s minimum and maximum. Ask whether the maximum is still right for your worst realistic need and your current budget. Ask whether the minimum still keeps you safe. Adjust when the answers change, and record why.

Reading the Capacity Reasons

Each decider explains its answer. Make a habit of reading these explanations when a big step happens. You may discover that a single huge shard forced a larger node size, that a new data stream doubled the hot requirement, or that a machine learning job needs more memory than expected. Every discovery suggests a fix that costs less than simply paying for more capacity.

Upgrades and Changes

Upgrading Elasticsearch or the orchestrator can change how recommendations are made or applied. Read the release notes for autoscaling changes, and test upgrades in a copy of your environment. Consider pausing scale-down during an upgrade window, so that shard moves do not pile up on top of the upgrade work.

Handling Planned Events

Before a big sale, a product launch or a migration, review your capacity. It is often wise to raise the minimum a little ahead of time so that the cluster starts the event with room to spare, then lower it again afterward. Autoscaling deals with the unexpected, and planning deals with the expected.

Cleaning Up

Retire data streams and indices that no longer serve a purpose. Remove policies for tiers that no longer exist. Review machine learning jobs that were opened for a test and never closed, since they keep holding memory and keep the machine learning tier larger than needed.

When autoscaling seems to misbehave, an ordered set of checks finds the cause faster than guessing. Work from the calculation toward the machines.

Common Symptoms and First Guesses

The stories below are illustrative composites showing how different kinds of organizations use autoscaling. They are not accounts of specific companies.

Half-true ideas about autoscaling cause wasted effort and bad surprises. Here are the ones that appear most often.

A quick reference for the important words in this guide, in plain language.

Quick answers to the questions people ask most often when they first meet autoscaling.