Most Elasticsearch performance problems that appear at scale trace back to a shard decision made months earlier.

The quiet decision that shapes everything

When you create an index, Elasticsearch asks how many primary shards it should have. Many beginners accept the default and move on, because everything works perfectly with a small amount of data. The trouble is that this choice stays hidden for a long time. The cluster runs happily for months, then data grows, more teams arrive, and suddenly the master node is slow, restarts take hours and dashboards stutter.

The reason the problem appears late is that a shard count is baked into an index when it is created. You cannot simply edit it afterwards. Changing it later means shrinking, splitting or reindexing, and each of these takes planning, spare capacity and time. A little thought on day one saves a very stressful month later.

What “at scale” really means

At a small scale, almost any shard layout works. Scale begins when the number of shards, indices or nodes is large enough that the fixed cost of each shard becomes visible. That can mean tens of terabytes of data, thousands of shards, or dozens of teams sharing one cluster. At that point, small mistakes are multiplied by thousands, and a wasteful pattern that cost nothing in a test environment becomes the single biggest drain on the system.

Every term below is used again and again, so it is worth defining each one carefully.

The relationship between them

An index is divided into primary shards. Each primary shard can have zero or more replica shards. All these shards are spread across the data nodes of the cluster. A search request has to visit one copy of every relevant shard, and an indexing request has to reach the primary shard that owns the document and then its replicas. So the number and size of shards directly decide how much work each request creates and where that work happens.

Why each shard is a full search engine

This point is the root of the whole topic. Because a shard is a complete Lucene index, it has its own segment files, its own memory structures, its own caches and its own transaction log. Creating a shard is therefore never free. Even an empty shard has a small cost, and the cost does not shrink when the shard holds only a few documents. That is why many tiny shards are so wasteful, and it is the seed of the guidance you will read in the next chapters.

To pick a good size, first learn what you are paying for every time you create a shard.

The heap link

Elasticsearch documentation has long offered a rule of thumb that a node should hold no more than about twenty shards per gigabyte of heap. On a node with thirty gigabytes of heap, that suggests around six hundred shards as an upper comfort level. Newer versions reduced the memory footprint of a shard, and the true limit depends on how many fields and segments each shard has, so treat the figure as a sanity check rather than a law. Still, the direction is clear: the more shards a node holds, the less room remains for searching, aggregating and caching.

Remember also that Elasticsearch recommends keeping the heap at no more than half of the machine’s memory and below the size where the Java virtual machine loses a memory-saving trick, which is roughly thirty gigabytes. The rest of the memory is left for the operating system to cache files. So you cannot fix a shard explosion by simply buying a bigger heap.

The thread link

Searches run on a thread pool whose size is based on the number of processor cores. Each thread works on one shard at a time. If a single search touches four hundred shards on a node with eight cores, most of those shard tasks wait in a queue. Fewer, larger shards let the same threads finish the job with less waiting, up to the point where a single shard becomes too large to search quickly.

Oversharding means having far more shards than your data and hardware can efficiently support.

How it usually happens

Oversharding rarely comes from one bad decision. It builds up from many reasonable ones. A team creates one index per day. Another creates one index per customer. A third uses the older default of five primary shards for every index. Each choice looks harmless alone, but they multiply. Consider two hundred services, each writing a daily index with five primary shards and one replica, kept for thirty days. That is two hundred times five times thirty, which equals thirty thousand primary shards, or sixty thousand with replicas.

Now suppose the whole cluster holds nine thousand gigabytes of primary data. Divided across thirty thousand shards, the average shard holds only about three tenths of a gigabyte. The cluster is carrying sixty thousand units of overhead to store data that would fit comfortably in a couple of hundred well-sized shards.

flowchart LR
  A["200 services"] --> B["Daily index each"]
  B --> C["5 primary shards each"]
  C --> D["30 day retention"]
  D --> E["30000 primary shards"]
  E --> F["About 0.3 GB per shard"]
  F --> G["Heap pressure and slow master"]
        

The symptoms

The opposite mistake is just as real, though it is quieter and less discussed.

What goes wrong with huge shards

Because a shard is the smallest unit Elasticsearch can move, copy or recover, a very large shard turns every routine event into a long operation. Moving a two hundred gigabyte shard to another node takes far longer than moving a forty gigabyte one, and during that time the network and disks of both nodes are busy. If the node holding the only copy fails, the time to rebuild the replica is proportional to the shard size.

A simple calculation shows the effect. A fifty gigabyte shard copied at one hundred megabytes per second takes about eight and a half minutes. A two hundred gigabyte shard at the same speed takes about thirty-four minutes. Multiply by dozens of shards after a node loss and you see why very large shards prolong periods of reduced safety.

Other problems with giant shards

A large shard cannot be searched by more than one thread at a time in the usual case, so one heavy query on one enormous shard cannot be sped up by adding more cores. Merges of large segments require large amounts of temporary disk space and produce long bursts of input and output activity. Uneven data placement also becomes likely, because a cluster of nodes holding a few huge shards each cannot balance itself smoothly. One node may hold three giants while another holds one.

There is also a hard technical ceiling. A single Lucene index cannot hold more than about two billion documents. Long before you reach that limit you will have suffered from slow merges and slow recovery, but it shows that unlimited growth of a single shard is impossible in any case.

Between too small and too large lies a comfortable range that works for most clusters.

The widely used range

Elastic’s own guidance suggests aiming for shards between about ten and fifty gigabytes for many search and logging workloads. Shards near the top of this range are typical for time-based data such as logs, where most reads cover recent windows and merging efficiency matters. Shards nearer the lower end are common for search-heavy workloads where each query is latency sensitive and users expect very quick answers. It is also wise to keep the number of documents per shard below a couple of hundred million, because document count affects search cost independently of gigabytes.

Why a range and not one number

The best size depends on several things at once: how fast the disks are, how much memory the node has, how complex the queries are, how much recovery time you can tolerate, and how the data is queried. A cluster on fast local drives with quick recovery can safely run bigger shards than a cluster on slow network disks. A workload of simple filtered searches tolerates bigger shards than one running heavy aggregations. So the published range is a starting point, and your own measurements are the final judge.

flowchart TD
  S["Start: estimate data per index or per day"] --> T{"Is the data time-based?"}
  T -->|Yes| U["Use data streams and roll over by primary shard size"]
  T -->|No| V["Estimate total size and growth for one to two years"]
  U --> W["Target 30 to 50 GB per shard"]
  V --> X["Target 10 to 30 GB per shard"]
  W --> Y["Benchmark with real data and real queries"]
  X --> Y
  Y --> Z["Adjust and document the decision"]
        

Sometimes the number of shards is chosen not for size but to spread the write load.

How writes use shards

Every document written to an index is routed to exactly one primary shard, decided by hashing the document’s routing value, which by default is its identifier. The primary shard processes the write and forwards it to the replicas. Because different primaries live on different nodes, a large number of primaries lets many nodes index in parallel. A single primary shard, in contrast, handles all writes for its index through one node’s resources.

For a busy write index, such as the current index of a big log stream, a single shard can become a bottleneck long before it becomes big. A common approach is to set the number of primaries for the write index close to the number of hot data nodes, or a small multiple of it, so the indexing work is spread evenly. When that index later rolls over, the old one can be shrunk to fewer shards for cheap long-term storage.

Balancing the two goals

You now have two forces. Size guidance pushes towards fewer shards, while write throughput pushes towards more. The trick is to treat them as belonging to different stages of the index’s life. During the hot stage, use enough primaries to spread writes. During the warm stage, shrink to the number that suits the final data size. The shrink step makes it possible to have both.

Replicas multiply your shard count, so they belong in every sizing calculation.

What replicas give you

A replica gives you two benefits. First, resilience: if a node fails, a copy of its shards exists elsewhere, and the cluster can promote it to primary. Second, read capacity: searches can be served by either the primary or any replica, so more copies means more places to run queries. A cluster serving a lot of search traffic can raise the replica count to spread reads across more machines.

What replicas cost

Every replica stores a full copy of the data, so one replica doubles the disk needed. Each replica also repeats the indexing work in normal Elasticsearch, so heavy write loads pay a price in processor time and network traffic. And each replica is another shard in the cluster state, adding to all the overhead described earlier. At scale, the cost of an unnecessary replica is multiplied by thousands of shards.

Choosing replicas by data value

Fresh data on the hot tier normally deserves at least one replica, since it is precious and heavily read. Older data on cheaper tiers may keep fewer, particularly when searchable snapshots provide an independent durable copy. Data that can be re-created easily from another source, such as a rebuildable analytics index, may run with no replicas at all, accepting a rebuild if something fails. The point is to decide deliberately rather than leave the default in place everywhere.

For logs, metrics and events, the best sizing tool is not a fixed number of shards, but a rule about when to start a new index.

The problem with one index per day

Creating an index every day looks tidy, but data volume is not tidy. A quiet weekend day may fill only a few gigabytes, giving you a tiny wasteful shard. A traffic peak may fill several hundred gigabytes, giving you a giant one. With hundreds of sources, the mixture of sizes is chaotic, and you cannot choose one primary count that suits them all.

Rollover by primary shard size

Rollover solves this by watching the index instead of the calendar. A lifecycle policy can roll the write index over when the largest primary shard reaches a chosen size, such as fifty gigabytes, or when the index reaches a maximum age, or when a document count limit is reached. Whichever happens first triggers a new index. The result is that every finished index has shards close to the target size, whether the day was quiet or busy.

Newer versions also allow minimum conditions, so that an index does not roll over merely because it is old if it is still nearly empty. That protects you from creating a stream of tiny indices during quiet periods. If your version does not support this, choose a maximum age long enough that a slow source still produces a reasonable shard.

sequenceDiagram
  participant App as Log source
  participant DS as Data stream
  participant W as Write index
  participant ILM as Lifecycle policy
  App->>DS: Send documents
  DS->>W: Store in current write index
  ILM->>W: Check primary shard size and age
  W-->>ILM: Largest shard reaches target size
  ILM->>DS: Roll over to a new write index
  DS->>W: Old index becomes read-only for writes
        

A worked example

Imagine a platform that produces three hundred gigabytes of indexed primary data each day and keeps thirty days online. That is nine thousand gigabytes. With a target of fifty gigabytes per shard, you need about one hundred and eighty primary shards in total, or three hundred and sixty with one replica. Compare this with the thirty thousand primaries in the oversharded example of Chapter 4. The difference in overhead is nearly two orders of magnitude, while the searchable data is identical.

Product catalogs, customer profiles and document libraries do not roll over, so they need a different way of thinking.

Estimate total size and growth

With data that lives for years and is updated in place, you cannot rely on rollover. Instead, estimate the size the index will reach within a planning horizon, perhaps one to two years, and pick a primary count so that shards stay in your target range at that size. A catalog that holds twenty gigabytes now and is expected to reach sixty gigabytes might do well with two or three primaries. One that will stay under five gigabytes forever is best with a single shard.

Do not oversize for the far future

It is tempting to plan for ten times growth by creating twenty primaries on day one. That would give twenty tiny shards today, each carrying full overhead and each slowing every query, all for a future that may never arrive. Elasticsearch offers a safer route: the split operation can multiply the primary count of an index later, when it is truly needed, as long as the original index was created with enough routing space. Planning for that flexibility is far cheaper than paying for unused shards every day.

Seeing the journey of a request explains why shard count and size affect latency so strongly.

The journey of a search

A search arrives at any node, which becomes the coordinating node for that request. It determines which indices match, and from them which shards could hold results. It sends the query to one copy of each shard. Each shard runs the query on its own segments, keeps the best results, and returns them to the coordinator. The coordinator then merges all these partial lists into one final list and, in a second step, fetches the actual documents for the winning results.

sequenceDiagram
  participant U as User
  participant C as Coordinating node
  participant S1 as Shard A
  participant S2 as Shard B
  participant S3 as Shard C
  U->>C: Search request
  C->>S1: Query phase
  C->>S2: Query phase
  C->>S3: Query phase
  S1-->>C: Top results
  S2-->>C: Top results
  S3-->>C: Top results
  C->>C: Merge and rank
  C->>S1: Fetch phase for winners
  C-->>U: Final response
        

Why many shards slow the merge

The coordinating node must hold and merge results from every shard. With three shards, that is a tiny task. With three thousand, the memory needed to hold the partial results grows, and the coordinator may need to reduce results in batches. The response time is also set by the slowest shard, so the more shards there are, the higher the chance that at least one is slow at that moment, a pattern often called tail latency.

Skipping shards that cannot match

Elasticsearch includes a lightweight pre-check step. Before running the real query on many shards, it asks each shard whether the query could possibly match, using information such as the range of timestamps stored in the shard. Shards that cannot match are skipped. This step is very helpful for time-based data, which is one more reason to always include a time filter in queries on data streams.

The journey of a write

A write request also lands on a coordinating node. The node hashes the routing value to find the correct primary shard, forwards the document there, and the primary writes it into memory buffers and the transaction log, then sends it to the replicas. The write is confirmed once the required copies have accepted it. Regular refresh operations make the new data visible to searches, by default about once per second on active indices. Each shard has its own refresh and its own merge activity, so thousands of shards means thousands of small background jobs.

A cluster has both soft rules of thumb and hard guardrails that limit how many shards it can carry.

The guardrail settings

By default, Elasticsearch limits each normal data node to about one thousand open shards, and each frozen data node to about three thousand, because frozen shards are lighter. If creating an index would exceed the total allowed by the cluster, the request is refused. These limits exist to protect the cluster from a slow slide into instability, so they should be treated as a warning bell, not as an obstacle to remove.

The master node view

The master node has its own scaling concern, since it must maintain and publish the cluster state. Elastic’s guidance suggests keeping the number of indices roughly proportional to the master heap, on the order of a few thousand indices per gigabyte of master heap. A dedicated master with a small heap can be overwhelmed by a large state, and its trouble is felt by every node because every change waits for the master.

Making the count work for you

Take the nodes you plan to run, multiply by a safe shards-per-node figure based on your heap, and compare the result with the shards your design creates. If the design needs more, you have three choices: enlarge the shards, reduce the number of indices, or add nodes. Adding nodes is the most expensive fix and often not the right one, since it treats the symptom rather than the cause.

Even perfectly sized shards cause trouble if they are placed unevenly across the nodes.

How the cluster places shards

The master runs an allocation process that decides which node receives each shard. It follows rules, such as never placing a primary and its replica on the same node, respecting tier roles, and obeying disk watermarks. Within those rules, it tries to balance the cluster by spreading shards and, in newer versions, by considering factors such as disk usage and write load, not only shard counts.

What a hot spot looks like

A hot spot happens when one node does much more work than its neighbours. The classic cause is that several busy write shards land on the same node, because the balancer only saw them as equal shards. That node then shows high processor use, a long indexing queue and slow responses, while other nodes sit idle. Users see the slowness as random spikes, which makes the cause hard to find.

flowchart LR
  subgraph Uneven
    N1["Node 1: 3 active write shards"]
    N2["Node 2: 1 active write shard"]
    N3["Node 3: 0 active write shards"]
  end
  subgraph Balanced
    M1["Node 1: 1 active write shard"]
    M2["Node 2: 1 active write shard"]
    M3["Node 3: 1 active write shard"]
  end
  Uneven --> Balanced
        

Tools to keep placement healthy

If your shards are already the wrong size, you have several repair tools, each with its own rules.

Shrink: fewer, larger shards

The shrink operation creates a new index with fewer primary shards from an existing one. The target count must be a factor of the source count, so an index with eight primaries can become four, two or one. The source index must be read-only, and all its shards must first be gathered on a single node, which is why enough free disk on that node matters. Shrink is efficient, because it can reuse the underlying files instead of re-reading every document. It is the natural fix for oversharded read-only data and a standard step in the warm phase.

Split: more, smaller shards

Split does the reverse, creating a new index with more primaries. The target count must be a multiple of the source count, and the operation depends on the routing space that was set when the index was created. It is useful when a growing index has shards that have become too large but still needs to keep the same name and content.

Force merge: fewer segments

Force merge does not change the number of shards. It rewrites the segments of each shard into fewer, larger ones, reducing overhead and improving search efficiency for read-only data. It uses a lot of input and output activity and temporary disk space, so it should only be applied to indices that are no longer receiving writes.

Reindex: the universal fix

When nothing else fits, reindexing copies documents from the old index into a new one with the right settings. It is slow and resource-hungry, but it is always possible, and it lets you also fix mappings and other design errors at the same time. For huge datasets, it should be planned like a migration, with monitoring, throttling and a clear plan for switching over.

Serving thousands of customers is where shard sizing choices are tested hardest.

Three common designs

Software companies that host many customers in one cluster generally pick one of three designs. The first is an index per tenant. The second is one shared index for all tenants, with a tenant identifier filtering each query. The third is a hybrid where big tenants get their own indices and small tenants share. Each has strengths, and each interacts with shard sizing in a different way.

flowchart TD
  T["Incoming tenant data"] --> D{"How large is the tenant?"}
  D -->|Very large| A["Dedicated index with its own shards"]
  D -->|Medium| B["Shared index group with routing by tenant"]
  D -->|Small| C["Large shared index with filtered alias"]
        

Routing to reduce fan-out

In a shared index with many shards, a query for one tenant would normally visit every shard. By supplying a routing value equal to the tenant identifier at both write and read time, the query goes only to the shard that holds that tenant’s data. This can cut latency and resource use dramatically. The trade-off is uneven shard sizes when tenants differ greatly in size, because a huge tenant places all of its data on a single shard. That is why the hybrid approach gives very large tenants their own indices.

Protecting tenants from each other

Filtered aliases give each tenant a named view of the shared index, with the filter built in, which reduces the chance of a query forgetting the tenant condition. Role-based security can restrict users to their alias or to documents matching their tenant. Resource limits, quotas and separate lifecycle policies help stop a noisy tenant from harming the rest.

The ideal shard is not the same at every stage of an index’s life.

Hot tier

On hot nodes the priority is write throughput and quick access to fresh data. Shards are sized so they can be written to in parallel across nodes, and they often reach the upper end of the size range before rollover. Replicas are common for safety and read capacity. Merges are ongoing, so disk speed matters.

Warm tier

When an index is no longer written to, it can be reshaped. Shrinking reduces the number of primaries, force merge reduces the segments, and read-only status protects the data. The aim is fewer, well-compressed shards that use little memory per gigabyte, since warm nodes are expected to hold a lot of data per machine.

Cold and frozen tiers

With searchable snapshots, the shard layout of the snapshot becomes the layout of the mounted index. Large, well-merged shards work well here, because every shard means extra requests to the repository and extra overhead in the cache. Frozen nodes tolerate a higher number of shards per node than normal nodes, because their shards are cache-backed and light, but that does not mean shard count is free.

Shard size decides how quickly the cluster heals, so it is a reliability decision as much as a performance one.

Recovery time and risk

When a node fails, its shards must be rebuilt on other nodes from surviving copies. The time needed grows with shard size and with the number of shards on the failed node. During that window, some data exists in fewer copies than intended, and a second failure could cause real loss. Smaller shards recover in parallel and in shorter individual times, which narrows the danger window. Recovery speed is also throttled to protect normal traffic, so raising the limit trades safety speed against search performance.

Snapshots and shard count

Snapshots copy shard files to the repository. Each shard adds bookkeeping, so tens of thousands of tiny shards make snapshots slow to start and to finish, and make repository listings heavy. On the other hand, very large shards make each snapshot task long. Well-sized shards make snapshots both quick to organise and quick to restore, which matters when you need to recover from a real disaster.

Restart behaviour

During a rolling upgrade, each node stops, restarts and rejoins. Its shards recover while the next node waits. If each node holds a manageable number of moderate shards, this takes minutes. If it holds huge numbers of tiny shards, the recovery phase alone can take many times longer, and upgrades that should fit in a night stretch across days.

Published ranges are a starting point. The real answer for your data comes from a small, careful experiment.

What to measure

Shard sizing is not a one-time task. Data changes, so you must keep watching the numbers.

The numbers worth tracking

A troubleshooting flow

flowchart TD
  P["Symptom: slow cluster or unstable nodes"] --> Q{"Many pending master tasks?"}
  Q -->|Yes| R["Check total shard and index count, look for oversharding"]
  Q -->|No| S{"Some nodes much busier than others?"}
  S -->|Yes| T["Check shard placement and hot spots"]
  S -->|No| U{"Slow recovery or long merges?"}
  U -->|Yes| V["Check for very large shards"]
  U -->|No| W["Check queries, mappings and hardware"]
        

Useful diagnostic views

Elasticsearch provides compact listing views for shards, indices and allocation, which show sizes, document counts and node placement. It also provides cluster health, pending task and node statistics views. Together with the allocation explain feature, these answer most questions about why shards are where they are and whether they are healthy. Monitoring dashboards in Kibana present the same data over time, which is essential for spotting slow trends before they become emergencies.

Sizing decisions touch security more than people expect, especially in shared clusters.

Isolation and its price

Strong isolation between teams or tenants is easiest with separate indices, since permissions can be granted per index. But separate indices for every team creates the small-shard problem. Shared indices reduce overhead but rely on document-level security and filtered aliases to keep data apart. Choosing between them is a balance of risk, cost and operational effort, and it should be decided together by the architecture and security teams, not by the shard count alone.

Protecting the platform from careless users

Because index creation is cheap for the user but costly for the cluster, uncontrolled creation of indices is a genuine risk. Index templates, naming rules and permission limits can prevent teams from creating thousands of tiny indices by accident. Applications that create an index per request or per user session are a well-known source of shard explosions, and a governance rule against them saves a great deal of pain.

Data protection duties

Deleting one customer’s data is much simpler when it lives in its own index than when it is mixed inside a shared one, where it needs a delete-by-query and a later cleanup of segments. If your legal duties require prompt erasure, that should influence your tenancy and shard design. Encryption in transit and at rest, audit logging, and guarded snapshot repositories apply regardless of the shard design, but they must scale with it.

Patterns that keep shard counts under control, and traps that keep appearing in real clusters.

Patterns worth adopting

Anti-patterns to avoid

A checklist of habits that keep a large cluster calm.

These illustrative examples show how the ideas combine in different kinds of organizations.

These scenarios are illustrations, not measurements from named companies. Your own results depend on your data and hardware, which is why Chapter 18 recommends running your own tests.

A shard sizing strategy takes effort, so let us be honest about what it returns and what it demands.

When you can keep it simple

If your data is small, your cluster has only a handful of indices, and growth is slow, the default single shard is often perfectly good. Elasticsearch’s own default of one primary shard per index in recent versions reflects that most indices are small. Deliberate strategy pays off when volume, number of indices or number of teams grows, and it is best to start the habit before the pain arrives.

A common starting range is about ten to fifty gigabytes per shard, with larger values for logs and smaller for latency-sensitive search. The right value for you comes from testing with your own data and queries.

Not by simply editing it. You can use shrink, split or reindex to create a new index with a different count. That is why planning at creation time matters.

Yes. The replica count can be changed at any time on a live index, so it is a flexible tool for adjusting resilience and read capacity.

No. More shards add parallelism up to a point, but each one carries overhead. Beyond that point, extra shards only add coordination work and memory use.

A common comfort level is around twenty shards or fewer per gigabyte of heap, and by default a normal data node is limited to about one thousand. The practical number depends on shard content and workload.

Usually it is better to use data streams and roll over by size, so that each index holds a sensible amount of data no matter how busy the day was.

Shrink reduces the number of primary shards by creating a new index. Force merge keeps the same shards but reduces the number of segments inside each one.

Not always. It suits a small number of large tenants. With thousands of small tenants, it creates too many tiny shards, and a hybrid design is usually better.

It sends all documents that share a routing value to the same shard, so queries with that value visit one shard instead of all. The risk is uneven shard sizes when groups differ greatly.

Typical signs are slow cluster state updates, high heap use with little data, slow restarts, many tiny shards, and errors about shard limits.

Search cost relates to how many documents a shard holds, not only its gigabytes. A shard with tiny documents can reach very high counts at a modest size, so both should be watched.

It reduces the operational work, and some newer serverless offerings hide much of it. Still, in many deployments the shard design remains yours, so the principles here continue to apply.

Shard sizing strategy is the discipline of choosing how big each shard should be and how many the cluster should carry. Too many small shards drain memory, slow the master and make restarts painful. Too few huge shards make recovery, movement and merging slow and risky. The healthy middle, guided by ranges such as ten to fifty gigabytes and checked by your own tests, is reached by using data streams, rollover by size, tier-aware reshaping, careful replica choices, and a tenancy model that matches your customer count. Add monitoring, governance and regular reviews, and you get a cluster that stays fast and affordable as it grows.