Before touching a calculator, we need a clear idea of what we are actually trying to predict.

A simple definition

Capacity planning is the practice of estimating the resources a system needs to meet its goals, and then arranging those resources so the system keeps meeting the goals as demand changes. For Elasticsearch, the resources are disk space, memory, processor time, disk speed and network bandwidth. The goals are usually expressed as promises: searches finish within a certain time, new data becomes searchable within a certain delay, and the cluster stays available even when a machine fails.

Notice that capacity planning is not a one-time purchase decision. It is a loop. You estimate, you build, you measure, you learn where the estimate was wrong, and you adjust. The best teams treat their first estimate as a hypothesis and their production metrics as the experiment.

Why Elasticsearch needs its own careful planning

Elasticsearch is not a simple storage box. It is a distributed engine that builds many small search structures, keeps copies for safety, merges data in the background, and caches heavily in memory. This creates several surprises for newcomers.

What goes wrong without a plan

Under-provisioned clusters show symptoms that look like software bugs: slow searches, rejected write requests, nodes leaving the cluster, and shards refusing to allocate because disks are nearly full. Over-provisioned clusters hide their problem in the invoice. Both are avoidable, and both are far more expensive to fix in an emergency than in a calm planning session.

flowchart LR
  A["Understand workload and goals"] --> B["Estimate storage, memory, CPU, IO"]
  B --> C["Design shards, nodes and tiers"]
  C --> D["Benchmark with realistic data"]
  D --> E["Deploy with headroom"]
  E --> F["Monitor real usage"]
  F --> G["Adjust and re-plan"]
  G --> A
        

A short refresher on the words we will use in every later chapter.

The write path in one picture

When a document is indexed, it does not go straight into a finished file. It is first sent to the primary shard, written to a transaction log for safety and buffered in memory, and after a short refresh interval it becomes visible to searches as a new small segment. The primary then forwards the operation to its replicas. Understanding this path explains why writes cost CPU, memory and disk speed all at once.

sequenceDiagram
  participant C as Client
  participant N as Coordinating node
  participant P as Primary shard
  participant R as Replica shard
  C->>N: Bulk request
  N->>P: Route documents to primary
  P->>P: Buffer and write transaction log
  P->>R: Forward operation
  R->>R: Buffer and write transaction log
  R-->>P: Acknowledge
  P-->>N: Acknowledge
  N-->>C: Response
        

Every capacity problem is ultimately one of these four wells running dry.

The bottleneck rule

A cluster is only as capable as its scarcest resource. When you size by storage alone, you often discover that memory or CPU is the real limit. When you size by CPU alone, you discover a full disk. A good plan calculates each of the four separately, and then takes the largest resulting node count as the answer.

Hardware answers come from workload questions. Ask the questions before you shop.

Three broad workload families

The workload questionnaire

Before sizing, gather the following facts, even if some are only rough estimates. For each one, note how sure you are, because uncertain inputs deserve wider safety margins.

Storage is the most concrete calculation, so it is the natural place to begin.

From raw data to disk space

The disk footprint of your data is built up in stages. Each stage changes the number, and forgetting any one of them is the most common reason clusters run out of space early.

flowchart LR
  A["Raw data per day"] --> B["Indexed size: raw times expansion factor"]
  B --> C["Add replicas: multiply by copies"]
  C --> D["Multiply by retention days"]
  D --> E["Add growth allowance"]
  E --> F["Divide by target disk utilization"]
  F --> G["Total disk to provision"]
        

Stage by stage

Things that quietly consume extra space

Shards are the unit of scale in Elasticsearch, so shard planning is capacity planning.

Why shard size matters

Every shard is a self-contained search engine, and the cluster moves, recovers and balances shards as whole units. Very small shards waste memory and coordination effort because each carries fixed overhead. Very large shards are slow to recover after a failure, slow to move between nodes, and can concentrate load on one machine. A commonly cited healthy range is roughly ten to fifty gigabytes per shard, with the exact sweet spot depending on your workload and version. Always compare against the current official guidance, since recommendations have evolved.

Why shard count matters

The shard count multiplies costs across the cluster. Each shard uses heap for its metadata and file handles, adds entries to the cluster state that every node holds, and adds work for every search that touches it. Elasticsearch has a default guardrail limiting shards per data node to one thousand, and an older rule of thumb suggested keeping shards per gigabyte of heap low, such as around twenty. Treat these as warning signs rather than targets. A well-planned cluster sits far below them.

Choosing the number of primary shards

Start from the size of the data an index will hold and divide by the target shard size. For a time-based index that rolls over daily, use the expected daily indexed volume. For a fixed catalog index, use its total indexed size plus growth. Then check two more things. There should be enough primaries to spread indexing load across your data nodes, and not so many that each search fans out to a crowd of tiny shards.

Rollover and lifecycle: keeping shards healthy automatically

For growing data, rollover creates a new index when the current one reaches a size, age or document count. Combined with lifecycle policies, it keeps shards in the target range without manual work and makes retention a simple matter of deleting whole old indices. Data streams package this pattern for time-based data and are the modern default for logs and metrics.

Memory is where Elasticsearch performance is often won or lost.

Two memory pools, one machine

On each data node, physical memory is shared between the Java heap and the operating system file cache. The heap holds cluster metadata, shard structures, caches, aggregation buffers and request working space. The file cache holds recently read pieces of your index files, so repeated searches read from fast memory instead of slower disk.

The widely followed guidance is to give the heap no more than about half of the machine’s RAM and to keep it below the size at which the Java virtual machine loses an efficient pointer optimization, roughly thirty gigabytes. The remaining memory is deliberately left to the operating system, because that is what makes searches fast.

What consumes heap

Memory-to-data ratios

A useful planning aid is the ratio of memory to the data a node holds. Older Elastic guidance cited approximate ratios such as one gigabyte of RAM per thirty gigabytes of data for hot data, around one to one hundred sixty for warm data, and much lower still for cold data. Newer features such as searchable snapshots on the frozen tier change these numbers dramatically. Use ratios only as a first pass, then validate through benchmarking, because your query patterns decide how much of your data must stay in cache.

Processor capacity decides how much work the cluster can do at the same moment.

What uses CPU

Elasticsearch spends processor time on parsing and analyzing incoming documents, building and merging segments, executing queries, scoring results, and computing aggregations. Indexing is generally processor hungry because text must be broken into terms and structures must be built. Searching is processor hungry when queries are complex, touch many shards, or run heavy aggregations.

Thread pools: the cluster’s work queues

Each node handles work through named pools of threads, with separate pools for searching, writing, and other tasks. The number of threads is tied to the number of processor cores, and each pool has a bounded queue. When work arrives faster than the pool can process it, the queue fills, and once it is full new requests are rejected. Rejections are Elasticsearch’s way of saying “I am at capacity, slow down”, and a steady stream of them is a clear scaling signal.

Estimating CPU for search

A useful mental model is to count shard-level requests. A query that touches ten shards creates ten small pieces of work. If your cluster serves a certain number of queries per second, and each touches a certain number of shards, the total shard-requests per second is the product. Benchmarking tells you how many shard-requests per second one core or one node can sustain at your latency target, and that number converts directly into a node count.

Estimating CPU for indexing

Indexing capacity depends on document size, number of fields, analysis complexity, ingest pipelines and the replica count. Because every replica repeats the indexing work, doubling the replicas roughly doubles the total processing for writes. The most reliable method is to load a realistic sample into a test cluster, measure documents per second per node, and scale from there.

Capacity is not just how much can be stored, but how quickly it can be moved.

Choosing disk types

Because searches read many small pieces of files at random, disk latency matters a great deal. Solid-state drives and NVMe drives are the standard choice for hot data because they handle random reads and heavy writes gracefully. Spinning disks can be acceptable for older, rarely searched data where slower reads are tolerable, but they make merges and recoveries much slower.

Disk redundancy and layout

Because Elasticsearch already keeps replicas of every shard on other nodes, you do not necessarily need expensive redundancy inside each machine. Many teams stripe multiple local drives together for speed, accepting that losing a drive means the node recovers from replicas. This is a design decision to make consciously, weighing recovery time against hardware cost.

Network: the quiet consumer

Network capacity is used by client traffic, replication, search fan-out and result merging, shard recovery and rebalancing, and snapshots. For most moderate clusters, a fast local network is plentiful. But large clusters with heavy indexing, or clusters that must recover several terabytes after a failure, can be limited by network speed. When replicas live in different availability zones, traffic crosses zone boundaries, which can be both slower and, in cloud environments, chargeable.

Not every node needs to do every job, and separating roles is a capacity tool.

The master node rule

For production, use three dedicated master-eligible nodes, an odd number so the cluster can always form a majority to elect a leader and avoid a split brain, where two halves both believe they are in charge. Dedicated masters do not need much disk or a large heap for most clusters, but they need reliability and reasonably fast processors. Very large clusters with huge cluster states may need stronger masters.

flowchart TD
  CL["Clients and applications"] --> CO["Coordinating nodes"]
  CL --> IN["Ingest nodes"]
  CO --> H["Hot data nodes"]
  IN --> H
  H --> W["Warm data nodes"]
  W --> C["Cold and frozen nodes"]
  M["Three dedicated master nodes"] -. "manage cluster state" .-> H
  M -. "manage cluster state" .-> W
  M -. "manage cluster state" .-> C
        

When to keep it simple

Very small clusters often combine roles on the same nodes to save cost. That is acceptable, but as soon as the cluster carries important or heavy workloads, dedicated masters are one of the cheapest and most effective reliability upgrades. Coordinating nodes are worth adding when dashboards run large aggregations or when the number of shards per query is high, because merging results consumes heap that you would rather not take from data nodes.

The cheapest terabyte is the one you store on the right kind of machine.

Most time-based data is very valuable when it is fresh and much less valuable as it ages. Index lifecycle management lets you encode this fact as a policy that moves data automatically from expensive fast hardware to cheap slower hardware, and eventually deletes it. Done well, this can cut hardware costs dramatically without hurting the searches people actually run.

flowchart LR
  A["Hot: new data, fast disks"] -->|"After rollover or a few days"| B["Warm: older data, cheaper disks"]
  B -->|"After weeks"| C["Cold: rarely searched"]
  C -->|"After months"| D["Frozen: object storage plus cache"]
  D -->|"At retention limit"| E["Delete"]
        

What each phase can do for capacity

Capacity effects to remember

Let us walk through a complete, illustrative calculation from start to finish.

The scenario

A company collects application and infrastructure logs. About three hundred gigabytes of raw logs arrive per day. Logs must be searchable for thirty days: the newest seven days on fast hot nodes and the following twenty-three days on cheaper warm nodes. They want one replica for safety, expect twenty percent growth over the coming year, and want to survive the loss of any single node.

Step 1: storage

Step 2: turn storage into nodes

Suppose each hot node offers two terabytes of usable fast storage. Seven point nine terabytes divided by two is about four nodes. Suppose each warm node offers eight terabytes. Twenty-six divided by eight is about three and a quarter, so four nodes. Now add one spare node in each tier so the loss of a machine does not push the survivors past their watermarks, giving five hot nodes and five warm nodes.

Step 3: check throughput

Three hundred gigabytes per day is about three and a half megabytes per second on average, or, if each log line is around one kilobyte, roughly thirty-five hundred events per second. Assuming the peak is about three times the average, plan for roughly ten thousand events per second. Spread across five hot nodes with one replica, that workload is modest for modern hardware, but you must confirm it in a benchmark, because parsing, mappings and pipelines can change the answer.

Step 4: check shards

Indexed primary data is 330 gigabytes per day. Aiming for a shard size around forty gigabytes suggests about eight primary shards per day. With one replica that is sixteen shard copies per day, and over thirty days about four hundred eighty shards across ten data nodes, or about forty-eight per node. That is comfortably far below the default guardrail, leaving room to grow.

Step 5: check memory

If each hot node has sixty-four gigabytes of RAM with a heap around thirty gigabytes, the remaining memory serves the file cache. Two terabytes of data on a sixty-four gigabyte node is roughly one gigabyte of memory per thirty-one gigabytes of data, consistent with older hot-tier guidance. A warm node with the same memory and eight terabytes is about one to one hundred twenty-five, which is acceptable for less frequently queried data but should be tested against real dashboards.

Step 6: add the supporting cast

Add three dedicated master nodes with modest resources. Add two coordinating nodes to absorb dashboard fan-out and result merging. If ingest pipelines are heavy, add ingest nodes or move the parsing upstream.

When users type into a search box and expect instant answers, the sizing questions change.

What is different

Search-centric workloads such as product catalogs, document search or site search typically hold a stable, moderate amount of data but face demanding latency goals and bursts of traffic. Storage is rarely the constraint. Instead, the questions become: can the important data stay in memory, can we handle peak queries per second, and how do we keep response times steady under load?

Scaling reads with replicas

For search workloads, replicas are not only for safety, they add read capacity. Each complete set of shards can answer queries independently, so adding a replica gives the cluster another full copy to spread load across. This scales throughput almost linearly at first, then gives diminishing returns as other limits, such as the coordinating node or the network, appear.

An illustrative sizing method

A benchmark turns your estimate into evidence.

Why benchmarking is non-negotiable

Two clusters with identical hardware can behave very differently depending on the documents, mappings and queries they handle. Generic published numbers do not know your data. Benchmarking with your own documents and queries is the only way to learn your real expansion factor, indexing throughput, and query capacity.

flowchart TD
  A["Define goals: throughput, latency, retention"] --> B["Prepare realistic data and queries"]
  B --> C["Test one node and one shard"]
  C --> D["Scale up load until latency goal breaks"]
  D --> E["Record the knee point and resource use"]
  E --> F["Extrapolate to full cluster size"]
  F --> G["Test the full cluster including failures"]
  G --> H{"Meets goals?"}
  H -->|"No"| I["Adjust design and repeat"]
  I --> C
  H -->|"Yes"| J["Approve capacity plan"]
        

The single-shard method

A classic technique is to test the smallest meaningful unit first. Create an index with a single primary shard and no replicas on a single node, load realistic data, and measure how much it can index and how fast it can search. Push the load until latency starts to rise sharply, and note where that happens. That point, sometimes called the knee of the curve, is the shard’s practical capacity. Scaling to the full cluster then becomes a matter of multiplying by the number of shards and nodes, with adjustments for coordination and replicas.

What a good benchmark includes

Tools and metrics

Elastic provides a benchmarking tool called Rally, designed for exactly this purpose, and many teams also use general load testing tools to replay their own query traffic. Whichever tool you use, capture throughput, latency percentiles such as the median, ninety-fifth and ninety-ninth, error and rejection counts, CPU use, heap use and garbage collection behavior, disk read and write rates, and network traffic.

A cluster that only works when nothing breaks is not really sized, it is merely lucky.

Failure is a capacity event

When a node disappears, its shards must be served from other copies and, over time, rebuilt elsewhere. The surviving nodes suddenly carry more data and more traffic than before. If they were already near their limits, the failure of one machine can trigger a chain reaction: survivors hit their disk watermarks, shards cannot be reallocated, and the cluster stays in a degraded state. Capacity planning must therefore include the extra load that comes after a failure.

The simple arithmetic of survivors

If a tier has a number of equal nodes and one fails, the survivors share its data, so each carries the original amount multiplied by the total node count divided by the survivors. With four nodes at seventy percent full, losing one leaves three nodes at about ninety-three percent, beyond the default high watermark. With five nodes at about fifty-five percent full, as in the worked example of Chapter 12, losing one leaves four nodes at about sixty-nine percent, which is healthy. This is exactly why the spare node in that plan matters.

Surviving a whole zone

Many production clusters spread nodes across availability zones, using shard allocation awareness so that a primary and its replica never share a zone. Losing an entire zone is a much larger event than losing one node. With three zones, the two survivors must absorb the third’s data, raising their load by half. To stay under the default low disk watermark after such a loss, the normal utilization should be under roughly fifty-six percent. With two zones the survivor must hold everything, so normal utilization must be under about forty-two percent if you want full redundancy to be restored in a single zone.

flowchart LR
  subgraph Z1["Zone A"]
    A1["Primary shards set 1"]
    A2["Replica shards set 3"]
  end
  subgraph Z2["Zone B"]
    B1["Primary shards set 2"]
    B2["Replica shards set 1"]
  end
  subgraph Z3["Zone C"]
    C1["Primary shards set 3"]
    C2["Replica shards set 2"]
  end
  Z3 -. "Zone C lost: replicas promoted, shards rebuilt in A and B" .-> Z1
  Z3 -. "Survivors take about 50 percent more data each" .-> Z2
        

Recovery speed and its limits

Elasticsearch throttles shard recovery so that it does not overwhelm normal traffic. That protects users but lengthens the time you spend without full redundancy. Recovery time is roughly the amount of data to rebuild divided by the effective recovery rate, so smaller nodes and faster networks recover sooner. Include this in your risk assessment: the larger each node, the longer the exposure window after a failure.

Capacity planning does not end at launch. Production tells you how right you were.

The vital signs

Watch trends, not just thresholds

A disk that is at sixty percent is fine today, but if it was at fifty percent last month and forty percent the month before, you will hit trouble in a predictable number of weeks. Track growth rates and project them forward. Set alerts that fire early, with enough lead time for procurement, approvals and change windows, not alerts that fire when the fire has started.

Turning signals into decisions

When the plan says you need more, there are several ways to get it, and some cost far less than buying hardware.

Vertical or horizontal

Most teams favor scaling out with moderately sized nodes, because it improves resilience and recovery. Scaling up still makes sense to reach the right memory-to-data ratio or to add faster disks.

Before buying, try these levers

Automation and elasticity

Managed offerings and Kubernetes operators can scale certain tiers automatically based on storage and load signals, using Elasticsearch’s autoscaling capabilities. This helps with unpredictable growth but still relies on your having sensible limits and a sound shard design. Automation scales your mistakes as happily as your successes, so confirm the design first.

flowchart TD
  A["Monitoring shows pressure"] --> B{"Which resource?"}
  B -->|"Disk"| C["Tier data sooner, compress, trim retention, add data nodes"]
  B -->|"Heap"| D["Cut shard count, fix heavy queries, add nodes"]
  B -->|"CPU or rejections"| E["Tune bulk and pipelines, add nodes or replicas"]
  B -->|"Search latency"| F["Add memory for cache, add replicas, simplify queries"]
  C --> G["Re-benchmark and update the plan"]
  D --> G
  E --> G
  F --> G
        

A short catalogue of the mistakes that show up again and again.

Best practices at a glance

Five common situations, and how the ideas in this guide apply to each.

A repeatable planning routine

There are formulas for each resource, such as storage from volume, expansion, replicas, retention and utilization. But the inputs, especially the expansion factor and query capacity, must come from testing your own data, so treat formulas as a first estimate.

A commonly cited range is roughly ten to fifty gigabytes, but the best value depends on your workload and version. Aim for shards large enough to avoid overhead and small enough to move and recover quickly, and check current official guidance.

There is a default limit of one thousand per data node, but healthy clusters usually run far below it. Heap use, cluster state size and recovery time are better guides than the limit itself.

About half of the machine’s memory, and no more than roughly thirty gigabytes. The rest should remain available to the operating system for the file cache.

At least one for most production data, so a node failure does not lose data. Add more when you need extra read capacity or greater fault tolerance, remembering each replica multiplies storage and write work.

Indexing builds search structures and stores extra representations of fields, which can enlarge data, while compression can shrink it. The result depends on your mappings, so measure with a real sample.

For production clusters, yes, three of them is the standard recommendation. They protect cluster management from being starved by heavy data node workloads. Very small development clusters can combine roles.

Before you reach your limits. Set alerts on trends in disk, heap, rejections and latency so that you have lead time, and remember that rebalancing itself takes time and resources.

Many moderately sized nodes usually improve resilience and shorten recovery, while very large nodes concentrate risk and slow recovery. The right balance depends on your memory-to-data needs and operational overhead.

Size to the measured peak with headroom, use replicas to add read capacity, buffer writes upstream, and consider temporary scaling ahead of known events.

No. Autoscaling reacts to signals, but it depends on a sound shard design, sensible limits and a budget. Planning tells you what the system should look like, and automation helps it get there.

Yes. Managed services handle machines and upgrades, but you still choose data volumes, tiers, shard design and retention, which determine performance and cost. The concepts in this guide carry over directly.

Capacity planning for Elasticsearch is the art of matching hardware and structure to a real workload, with enough room for peaks, growth and failure. It starts by understanding your data volume, retention, write and query patterns, and service goals. It continues by estimating storage, memory, processing and input and output separately, and taking the largest node count as the answer. It shapes shards into healthy sizes, separates node roles, and uses data tiers so that cheap hardware holds cold data. It is confirmed through benchmarks with real data and real queries, including failure tests. And it never really ends, because monitoring and regular reviews keep the plan honest as your system changes. Build the plan, test it, and keep correcting it.