The shortest possible answer: a merge policy is the set of rules that tells a shard which of its small data pieces, called segments, should be combined into a bigger one, and when.

Let us build the definition slowly. Elasticsearch is a search and analytics engine. Companies use it to search products, explore application logs, count events, and draw charts. Data lives in indices, which are like big labeled cabinets of documents. Each index is cut into pieces called shards, so that many servers can share the work.

Inside every shard, Elasticsearch relies on a search library called Lucene. A Lucene index stores its data in many small, read-only bundles of files called segments. Once a segment is written, it is never changed. New documents go into new segments. A search must look at every segment in the shard and combine the results.

Because segments pile up as data arrives, something must clean up. The cleaning job is called a merge: several segments are read, their live contents are written into one new, larger segment, and the old segments are deleted. The merge policy is the decision maker for this job. It looks at the current list of segments, and it answers three questions: is a merge needed now, which segments should take part, and how large may the result be?

The default policy

Elasticsearch, like Lucene, uses a policy named the tiered merge policy by default. It groups segments into tiers by size, like sorting coins into small, medium, and large piles, and merges the members of a tier when the pile gets too tall. Most of this guide explains how that policy thinks and how its settings shape its behavior.

Why you rarely notice it

Merges happen quietly in the background, dozens or hundreds of times a day on a busy shard. When the policy suits the workload, you never see them. You only notice when something goes wrong: writes slow down because merges cannot keep up, searches are sluggish because too many segments exist, or disks are fuller than the number of documents suggests.

What a merge policy is not

A merge policy does not decide which documents match a search, it does not move shards between servers, and it does not copy data to replicas. It only reorganizes the files inside one shard copy. It also does not fix a bad design: a shard with millions of tiny documents and huge mappings will still be heavy after a perfect merge.

Where you will meet it

You will meet the merge policy when you tune a bulk data load, when you plan how old time-based indices are compacted, when you see log lines about throttled indexing, when a busy update-heavy index carries many deleted documents, and when someone suggests running a force merge and you must judge whether that is a wise idea. Knowing how the policy thinks turns each of these into a clear decision.

Segments never change once written. That gives Elasticsearch speed and safety, but it creates a housekeeping problem that only merging can solve.

Why does Lucene store data in read-only segments at all? Because read-only files are easy to cache, easy to share between many searches at once, and impossible to corrupt halfway through an edit. A search can read a segment without locking anything, and the operating system can keep its contents in memory very efficiently. These benefits are a large part of why Elasticsearch searches so fast.

The price is that updates and deletes cannot change a segment in place. To change a document, Elasticsearch marks the old version as deleted and writes a new version somewhere else. To delete a document, it only marks it. Chapter 5 explains this trick. The result is that segments slowly fill with dead entries, and new segments keep appearing.

Three problems that pile up

• Too many segments. Each search must visit every segment in the shard. Every visit has a fixed overhead, so a hundred tiny segments are far slower to search than one big segment with the same content.
• Too many files. Each segment is several files on disk. Thousands of files strain the operating system, which limits how many files a process may keep open.
• Wasted space. Deleted documents still occupy their old segments. Without cleaning, the disk keeps carrying data that no search will ever return.

The alternative that would be worse

A different design would edit files in place. That approach needs locks, careful recovery after crashes, and slower caching. Lucene chose immutability plus merging. This kind of design is sometimes called a log-structured approach: write new data quickly in small pieces, and reorganize it later in the background. Many modern databases use the same idea, because it makes writing fast and reading safe.

An illustrative story

A short history

Lucene has used segment merging since its earliest days. Over time, its default policy has improved. The tiered policy became the default in the 3.x era of Lucene, replacing older “log” style policies. In more recent years, Lucene added a limit on the share of deleted documents that a segment may carry and improved how it treats very large segments. Elasticsearch exposes the important controls as index settings and keeps sensible defaults so that most users never touch them.

To follow the rest of this guide you need eight simple ideas: shard, segment, deleted document, refresh, flush, translog, merge, and the two roles called policy and scheduler.

Two kinds of document counts

When you ask Elasticsearch how many documents an index holds, it can report two numbers: the count of live documents and the count of deleted ones that still occupy space. The gap between them is a direct sign of how much cleaning work the merge policy has waiting. Keep these two numbers in mind, because they come back in Chapters 5, 24, and 26.

What is inside a segment

A segment is not a single file. It is a small family of files, each with its own job. There is a structure that maps words to the documents containing them, used for text search. There is a compact column store of values, used for sorting and grouping. There are stored copies of the original document source. There are helper structures for numbers, geographic points, and vectors. And there is a small file recording which documents are deleted. A merge rewrites all of these together, which is why it costs real reading and writing.

Where merging sits in the data path

The diagram shows how a document travels from a client to a shard and where merging takes place. The client usually reaches the cluster through a load balancer or an API gateway, which passes the request to a node. The node routes it to the primary shard, which writes it to its memory buffer and translog, and later turns the buffer into segments. Merging then works on those segments in the background.

flowchart TD
    A["Client application"] --> B["Load balancer or API gateway"]
    B --> C["Elasticsearch node"]
    C --> D["Primary shard"]
    D --> E["In-memory indexing buffer plus translog"]
    E -->|Refresh| F["New small segment, now searchable"]
    F --> G["Merge policy checks the list of segments"]
    G -->|Merge needed| H["Merge scheduler runs a background merge"]
    H --> I["One larger segment replaces several small ones"]
    D -.->|Copies the write| R["Replica shard: keeps its own segments and its own merges"]

Notice the dotted line to the replica shard. A replica receives the same writes, but it builds and merges its own segments independently. Two copies of the same shard usually contain the same documents but have different segment layouts. This matters for caching and for monitoring, and Chapter 18 returns to it.

Merging only makes sense once you see where segments come from. Every segment starts life in a memory buffer and is born at a refresh.

When a document is indexed, three things happen almost at once. It is added to an in-memory buffer, where Lucene collects new documents. It is written to the translog, the safety diary, so that a crash does not lose it. And it is sent to the replica copies. At this moment the document is safe but not yet searchable.

The refresh: birth of a segment

A refresh takes the current contents of the memory buffer and writes them as a brand-new segment. From that moment, searches can see those documents. The setting index.refresh_interval controls how often refresh happens, and its default is one second for indices that are actively searched. That means a busy shard creates a new small segment roughly every second, and each of those segments is tiny.

The memory buffer can also fill up before the timer fires. The node has a budget for indexing buffers, controlled by indices.memory.index_buffer_size, which defaults to ten percent of the heap. When the buffers of the shards on a node together exceed that budget, Elasticsearch writes some of them out as segments even without a refresh.

The flush: a safe checkpoint

A flush is different. It asks Lucene to make all segments durable on disk with a commit, and then it clears the parts of the translog that are no longer needed. Flushes happen automatically when the translog grows too large or after a period of time. Flushing does not directly create the small segments that merging cares about, but it does make the current set of segments permanent.

How many segments does a busy shard create?

A shard that receives steady traffic with the default one-second refresh interval creates about one new segment per second. In an hour, that is thousands of tiny segments if nothing merged them. This is why merging is not optional: without it, a busy shard would drown in files within minutes. It also shows why the refresh interval is such a strong lever: lengthening it from one second to thirty seconds creates thirty times fewer tiny segments, and the merge policy has far less to do.

Segments and searches

Every search opens a consistent view of the shard called a reader, which lists the segments that exist at that moment. If a merge finishes while a search is running, the search keeps using its old view, and the old segments remain on disk until the last search using them ends. This is why merges never disturb running searches and why disk space from merged segments is freed only after the readers that use them are closed.

Idle shards and the refresh timer

Modern Elasticsearch pauses scheduled refreshes on shards that have not been searched for a while, and refreshes them the next time a search arrives. This reduces the number of tiny segments created on shards that nobody is reading, and it helps large clusters with many quiet time-based indices.

Because segments never change, a deleted document is only marked. The bytes stay until a merge rewrites the segment without them.

When you delete a document, Elasticsearch does not remove it from its segment. Instead, it records the deletion in a small side structure called the live documents bit map. Searches consult this bit map and skip the marked documents. The document’s bytes are still on disk, still part of the segment, until a merge produces a new segment that leaves the dead ones out.

An update is a delete plus an insert

Updating a document works the same way. Elasticsearch marks the old version as deleted and indexes the new version as a fresh document, in whichever segment is currently being built. So an update-heavy index produces a constant stream of deleted documents in old segments and new documents in new segments. If ten percent of documents are updated each day, ten percent of the space in the older segments becomes dead each day.

sequenceDiagram
    participant C as Client
    participant S as Shard
    participant Old as Old segment
    participant New as New segment
    participant M as Merge
    C->>S: Update document 42
    S->>Old: Mark version 1 of document 42 as deleted
    S->>New: Write version 2 of document 42
    Note over Old: Dead bytes remain on disk
    M->>Old: Later, read live documents only
    M->>New: Write live documents into a merged segment
    M->>Old: Old segment removed, dead bytes reclaimed

How much waste is normal?

Some waste is normal and even healthy: merging costs time and disk activity, and reclaiming every dead byte immediately would be wasteful. The merge policy tolerates a certain percentage of deleted documents before it becomes eager to clean. The setting index.merge.policy.deletes_pct_allowed controls this, and its default in current versions is twenty percent. A shard that shows more than that share of deleted documents across its segments is a candidate for cleaning by the policy.

Some deleted documents are kept on purpose

Elasticsearch uses a feature called soft deletes to keep a short history of changes on each shard. Replicas and cross-cluster replication use that history to catch up after a temporary disconnection, without copying the whole shard. A document that was deleted may therefore be retained in a special form for a while, held by a retention lease with a default period of twelve hours. During that time, merges keep the history, and the count of deleted documents may not drop right after a merge. This is normal and not a sign of a broken policy.

Deleting whole indices is different

When you delete an entire index, Elasticsearch removes its files at once. There is no merge involved and no reclamation delay. This is the reason why time-based designs, where old data is removed by deleting whole daily or weekly indices, are so efficient: they avoid the slow, expensive process of removing individual documents.

Why this matters for capacity planning

A common surprise is that the disk usage of an index is larger than the size of its live data would suggest. For append-only data such as logs, the difference is small. For catalogs, profiles, or any workload with frequent updates, it can be large. When you plan storage, include a margin for deleted documents and for the temporary space that merges need, which Chapter 17 explains.

A merge is a careful copy. It reads several old segments, writes one new one, and only then lets the old ones go. Knowing the steps explains its costs and its benefits.

Let us follow one merge from start to finish. Imagine a shard with ten segments of similar size, and the policy decides that they should be combined.

flowchart LR
    subgraph Before["Before the merge"]
        S1["Segment A: 100 MB, 15 percent deleted"]
        S2["Segment B: 100 MB, 10 percent deleted"]
        S3["Segment C: 100 MB, 20 percent deleted"]
    end
    Before --> M["Merge: read live documents, rebuild structures"]
    M --> After["After the merge: one segment of about 255 MB, no deleted documents"]

The costs of a merge

• Disk reads and writes. All live data in the merged segments is read once and written once. For big merges, that is many gigabytes of input and output.
• CPU work. Rebuilding search structures and compressing data uses processor time. Fields with heavy structures, such as vectors or complex text analysis results, cost more.
• Temporary disk space. The new segment exists together with the old ones until the switch, so a merge needs extra space roughly equal to the size of the merged output.
• Memory pressure on the file system cache. New files compete for the operating system’s cache with files that searches use. After a large merge, some search data may have to be read from disk again.

The benefits of a merge

• Fewer segments. Each search visits fewer places, so it wastes less time on fixed per-segment overhead.
• Reclaimed space. Deleted documents are removed for good, unless they must be kept as history.
• Better compression and locality. Larger segments compress better, and related data sits closer together on disk.
• Fewer files. The operating system keeps track of fewer open files, which lowers the risk of reaching limits.
• More useful caches. Some caches, such as the node query cache, only consider segments above a size threshold, so bigger segments can make them work.

A merge is a copy, not a move

The most important idea is that a merge always writes new data. It never edits old segments. This is why merging is safe: if the process is interrupted, the old segments are still there and still correct, and the half-written new segment is simply discarded. It is also why merging has a cost: every byte that survives is rewritten.

Merges keep documents in order

Segments preserve the order in which documents were added within each segment. A merge concatenates the surviving documents of each source segment in a chosen order. If an index uses index sorting, a feature that keeps documents ordered by certain fields, the merge sorts the combined set, which is more expensive but produces benefits for queries that match the sort order.

Two separate parts work together: the policy chooses which segments to merge, and the scheduler decides how and when the merges actually run.

People often say “merge policy” when they mean the whole system. It helps to separate the two parts, because they have different settings and different failure modes.

How they talk to each other

The conversation happens inside the Lucene writer that owns the shard’s segments. Whenever the set of segments changes, because of a refresh, a flush, or a finished merge, the writer asks the policy whether any merges are needed. The policy returns a list of proposed merges. The writer hands the list to the scheduler, which starts merge threads within its limits. When a merge finishes, the writer asks the policy again, because the new segment list may call for another merge.

sequenceDiagram
    participant W as Shard writer
    participant P as Merge policy
    participant S as Merge scheduler
    participant T as Merge thread
    W->>W: New segment created by refresh or flush
    W->>P: Any merges needed for the current segments?
    P-->>W: Yes, merge these ten segments
    W->>S: Please run this merge
    S->>T: Start a merge thread within the limits
    T->>T: Read live documents and write the new segment
    T-->>W: Merge finished
    W->>P: Any more merges needed now?
    P-->>W: No

Why the separation is useful

Because the two parts are independent, you can tune them separately. If merges are choosing the wrong segments, you adjust the policy. If merges are right but too slow or too disruptive, you adjust the scheduler. Many operational problems that look like “merge policy trouble” are really scheduler trouble, such as too few merge threads on fast disks or too many on slow ones.

Other policy types

Lucene supports other policies, such as ones that merge neighboring segments in order of creation. Elasticsearch uses the tiered policy by default because it handles mixed workloads well. Some newer versions of Elasticsearch offer an alternative choice for special time-ordered data, where keeping neighboring documents together has extra benefits. If you use such features, read the documentation of your version for the options that apply. For most users, the default is the right answer.

The tiered policy sorts segments into size tiers, like coins into piles, and merges a tier when its pile grows too tall.

The main idea is simple. Small segments are cheap to merge and are created constantly, so they should be merged often. Big segments are expensive to rewrite and change little, so they should be merged rarely. The tiered policy creates this behavior by organizing segments into tiers of roughly similar size, where each tier is about ten times bigger than the one below it.

Four numbers that shape the policy

How tiers form over time

Watch a busy shard. Each refresh adds a tiny segment. When about ten tiny segments of a couple of megabytes each exist, the policy merges them into one segment of around twenty megabytes. When ten of those exist, they merge into one of around two hundred megabytes. When ten of those exist, they merge into one of around two gigabytes, and the pattern continues until segments reach the ceiling.

flowchart LR
    T1["Tier 1: about 10 segments of 2 MB"] -->|Merge| T2["Tier 2: about 10 segments of 20 MB"]
    T2 -->|Merge| T3["Tier 3: about 10 segments of 200 MB"]
    T3 -->|Merge| T4["Tier 4: about 10 segments of 2 GB"]
    T4 -->|Merge| T5["Tier 5: segments near the 5 GB ceiling, merged rarely"]

Write amplification: how often is each byte rewritten?

Every time a document climbs a tier, it is copied once. A document that starts in a two-megabyte segment and ends in a multi-gigabyte segment is therefore rewritten about four or five times over its life. This repeated copying is called write amplification. It is the price of keeping the segment count low. Doubling the refresh interval does not double this cost, but making segments start bigger, with a longer interval, removes the lowest tiers altogether.

What this means for segment counts

A healthy shard therefore carries roughly ten segments per active tier. A young, small shard has a handful. A shard of tens of gigabytes typically settles at a few dozen segments. A shard that has grown to several hundred gigabytes has many segments near the ceiling, because these large segments are no longer merged with each other, so the count grows slowly with size. These figures are approximations that depend on how the data arrived, but they are useful for judging whether a count looks reasonable.

The “floor” trick for tiny segments

Without a floor, segments of a few kilobytes would each form their own tier and would never be merged with anything of similar size. The floor makes all segments smaller than two megabytes count as two megabytes. That way, ten segments of ten kilobytes each are treated as ten segments of two megabytes, and the policy merges them into a proper segment. This is a small but important detail for workloads with very short refresh intervals.

When a merge is due, the policy does not pick at random. It scores possible groups and chooses the best one, favoring balanced, cheap, and cleaning merges.

After each change to the segment list, the policy asks a first question: does this shard have more segments than it should, given its total size and the per-tier setting? If yes, it looks for the best group of segments to merge. If no, it asks a second question: are there segments with too many deleted documents? Only if both answers are no does it do nothing.

The allowed number of segments

The policy calculates how many segments it would accept for the current size of the shard. It starts with the floor size and asks how many segments of that size fit into the data, then repeats with segments ten times bigger, and so on, allowing up to the per-tier number at each level. The sum of these is the allowed segment count. When the real count exceeds the allowed count, a merge is due.

The score of a candidate merge

To choose which segments to merge, the policy looks at candidate groups of up to the “merge at once” number and gives each a score. Lower scores are better. The score rewards three qualities.

• Balance. Merging segments of similar size is better than mixing one large and many small ones, because a large segment would be rewritten just to absorb a few small ones. The policy measures this as the “skew” of the group.
• Cost. Smaller total size means less data to rewrite, so cheaper merges are preferred.
• Reclaimed space. Groups with more deleted documents free more space, so they receive a better score.

Big segments and the half-ceiling rule

The largest segments need special care. Rewriting a segment of several gigabytes just to remove a few deleted documents is a poor use of resources. So, the policy treats segments that are larger than about half of the ceiling as too big for ordinary merging. They are considered only when their share of deleted documents rises above the allowed percentage, at which point cleaning them is worth the cost.

The deleted documents limit

The setting index.merge.policy.deletes_pct_allowed caps the share of deleted documents that the policy tolerates across the shard. When the share goes above the limit, the policy prefers merges that reclaim the most dead space, even if a segment is large. Lower values keep the index leaner but cause more rewriting. Higher values save effort but let dead bytes accumulate.

flowchart TD
    A["Segment list changed"] --> B{"More segments than the allowed count?"}
    B -->|Yes| C["Score candidate groups: balance, cost, reclaimed space"]
    B -->|No| D{"Deleted documents above the allowed percentage?"}
    D -->|Yes| E["Choose segments that free the most dead space"]
    D -->|No| F["Do nothing now"]
    C --> G["Propose the best group of up to ten segments"]
    E --> G
    G --> H["Scheduler runs the merge"]

Why the policy sometimes seems lazy

Operators sometimes see a shard with more segments than they expected and conclude that merging is broken. Usually the policy is doing what it should: the count is within the allowed number for that data size, and the extra segments are too big to be worth combining. A merge would cost more than the benefit. The right judgment is to compare the actual count with the allowed count for your data, not with an ideal of one segment.

Merges are proposed, not guaranteed to run at once

The policy only proposes merges. The scheduler starts them subject to its thread limits. When the shard is busy, proposals may wait in a queue. This delay is normal and usually short. If it grows long, the scheduler is the place to look, as explained in the next chapter.

Six settings shape the tiered policy. All are set per index, all can be changed on a live index, and the defaults are good for most workloads.

What happens when you change them

Golden rules for changing settings

• Keep the segments per tier at least as large as the merge at once. If the per-tier value is smaller, the policy would want to merge more segments than it is allowed to combine at once, causing repeated small merges.
• Change one setting at a time. Two changes together hide which one had the effect.
• Change on one index first. Test on a single index that represents your workload, and compare merge statistics and search timing before and after.
• Use index templates for future indices. Settings on an index do not follow it to new indices. A template applies them at creation time.
• Write down the reason. Non-default settings are easy to forget, and the reason for them is more important than the numbers.

Related settings around the policy

A few other settings have a strong effect on merging even though they are not part of the policy. The refresh interval controls how many tiny segments are born. The codec, set with index.codec, chooses how stored data is compressed: the default is fast, while the best compression option uses more CPU during writes and merges in exchange for smaller files. The number of replicas multiplies the merging work across the cluster, because every copy merges independently.

Setting the values in practice

These are index settings, so they are changed through the update index settings operation, or placed in an index template. Because they are dynamic, a change takes effect for the next merge decision, and it does not rewrite existing segments by itself. To bring an existing index to a new shape, you would rely on natural merging over time or on a planned force merge, which Chapter 13 discusses.

The scheduler is the crew boss. It limits how many merges run in parallel, adjusts their speed, and, when merges fall far behind, slows down indexing to let them catch up.

Elasticsearch uses the concurrent merge scheduler. It runs merges in separate background threads, so that indexing and searching keep going while merges work. Three controls matter.

Why merges are throttled

Merges compete with searches and indexing for the same disks and processors. If merges ran at full speed all the time, they would steal resources and make searches sluggish. The automatic throttle keeps merges at a speed just fast enough to keep up with the incoming data. When a big merge is running but no backlog exists, it goes gently. When merges start piling up, the speed limit rises.

When indexing is throttled

If incoming data creates segments faster than merges can combine them, the number of pending merges grows. Left alone, this would end with thousands of segments and a broken shard. To prevent this, when the backlog passes a limit, Elasticsearch throttles indexing on that shard down to a single thread. The logs show a message that says indexing is being throttled, together with the number of merges in flight and the maximum. When the backlog clears, another message says that throttling has stopped.

stateDiagram-v2
    [*] --> MergesKeepUp
    MergesKeepUp --> MergesFallingBehind: Segments arrive faster than merges finish
    MergesFallingBehind --> MergesKeepUp: Automatic throttle raises merge speed and backlog clears
    MergesFallingBehind --> IndexingThrottled: Backlog passes the maximum merge count
    IndexingThrottled --> MergesFallingBehind: Merges catch up
    IndexingThrottled --> MergesKeepUp: Backlog cleared and throttling stops

Spinning disks and solid-state disks

The defaults assume modern solid-state storage, which handles parallel reads and writes well. On old spinning disks, several merges in parallel force the disk head to jump around, and the total speed drops. For such disks, the traditional advice is to set the maximum thread count to one. Most current clusters use solid-state storage, where the default is appropriate.

Signs that the scheduler is the problem

• Log lines about throttled indexing appear during normal load.
• The number of running merges stays at the maximum for long periods.
• Segment counts grow steadily over hours while the policy settings are unchanged.
• Disk usage on the data path shows sustained saturation during merges.

What to do about a lagging scheduler

First reduce the incoming pressure on the shard: lengthen the refresh interval, reduce the number of shards that share one disk, and check bulk request sizes. Then check that storage is fast enough for the write rate. Only after these checks consider raising the number of merge threads, and only if disks and processors have spare capacity. Turning up merge parallelism on a saturated machine makes searches worse without solving the cause.

Follow one shard from its first document to its last search. The merge policy shapes each stage, and knowing the stages helps you judge what is normal.

flowchart LR
    A["Empty shard"] --> B["Many tiny segments from refreshes"]
    B --> C["Tiered merges combine similar sizes"]
    C --> D["Steady state: churn at the bottom, stable large segments"]
    D --> E["Aging: deleted documents accumulate"]
    E --> F["Cleaning merges rewrite the big segments"]
    D --> G["Writes stop: read-only shard"]
    G --> H["Optional one-time force merge"]

An illustrative growth example

The numbers below are invented to show the shape of the curve, not to predict your cluster. Imagine a shard that grows by about five gigabytes per day, with a one-second refresh interval and default merge settings.

Notice that the count does not grow linearly with size. Big segments stop merging, so the number rises slowly. If you see counts that are several times higher than this shape for your data size, investigate whether merges are lagging.

The importance of the small end

Almost all merge activity in a steady state involves the smallest segments, since they are created constantly. This is why the refresh interval has such a large effect: it decides how fast the smallest tier fills. A change that reduces the number of tiny segments has a much larger impact on total merge work than any change to the top tiers.

What “healthy” looks like

• A modest number of segments, in line with the size of the shard.
• A share of deleted documents below the allowed percentage on most segments.
• Few running merges at any moment, and none of them running for hours without progress.
• No indexing throttle messages in the logs.
• Search latency that does not drift upward as the shard grows.

The force merge operation lets you ask Elasticsearch to compact an index right now. It is a powerful tool with a sharp edge, and it belongs to indices that no longer receive writes.

The force merge operation asks every shard of an index to merge its segments down to a target number that you choose, using the parameter max_num_segments. A value of one asks for a single segment per shard. A separate option, only_expunge_deletes, asks the shards to merge only the segments that carry a large share of deleted documents, controlled by the expunge setting from Chapter 10.

Why force merge exists

For an index that has stopped receiving data, further natural merging will never happen, because the policy only acts when there is pressure. Such an index may carry dozens of segments and some dead bytes for the rest of its life. A one-time force merge tidies it, gives it its best possible compression, and lightens every future search. This is the ideal use.

The danger on an active index

If you force merge an index that is still being written to, two bad things happen. First, the merge produces a very large segment, far above the normal ceiling. Because of the half-ceiling rule from Chapter 9, such a giant segment is not eligible for normal merging unless it accumulates a large share of deleted documents. New writes and updates keep marking parts of it as deleted, and the policy cannot clean it cheaply. Second, the force merge itself uses a lot of I/O and CPU, which can disturb live traffic.

When to use it

• After a time-based index has rolled over and stopped receiving writes, typically as part of a warm phase.
• Before taking a long-term snapshot or before mounting an index as a searchable snapshot, so that fewer, larger files are stored.
• After a one-time bulk load into an index that will then be used mostly for reading.
• After changing the compression codec, to rewrite old segments with the new choice.

How to do it safely

flowchart TD
    A["Should I force merge this index?"] --> B{"Is the index still receiving writes or updates?"}
    B -->|Yes| C["No. Let the merge policy work, or tune the policy instead"]
    B -->|No| D{"Is there plenty of free disk space?"}
    D -->|No| E["Free space first, or merge in smaller steps"]
    D -->|Yes| F{"Is it off-peak or on a warm tier?"}
    F -->|No| G["Wait for a quiet time"]
    F -->|Yes| H["Force merge with a sensible segment target"]
    H --> I["Verify segment count, size, and search timing"]

Choosing the target

Merging to one segment gives the smallest overhead and the best compression, and it is a good target for an index that will never change. Some teams choose a slightly higher target when the shards are very large, to avoid a single enormous segment and to keep the merge less heavy. There is no universal right answer. Test on a representative index.

Force merge and deleted documents

A force merge also removes deleted documents, except for those kept as soft-delete history. For indices with a large share of dead data that will still receive writes, a lighter option is to ask for expunging deletes only. Even that should be used sparingly, because it rewrites every segment that crosses the threshold.

Force merge is not a fix for slow searches on a busy index

Operators sometimes reach for force merge when searches on an active index feel slow. That rarely helps for long and can make things worse. Look at queries, shard sizes, caches, and resources first. If segment count really is the issue, improve the policy or the refresh interval, so that the solution is lasting.

Logs, metrics, and events arrive in time order and age gracefully. A lifecycle plan uses merging at the right moment: gently while data is fresh, and once, decisively, when data is old.

Many clusters store data that only grows and is rarely changed: application logs, security events, sensor readings. These are usually kept in a data stream or in a series of indices that roll over by size or by age. The newest index receives all the writes, and older ones are read-only. Elasticsearch’s index lifecycle management, or ILM, automates the moves between phases.

flowchart LR
    A["Hot phase: index takes writes, natural merges only"] -->|Rollover| B["Warm phase: index is read-only"]
    B --> C["Force merge to a low segment count, optional better compression"]
    C --> D["Cold or frozen phase: cheaper storage, searchable snapshot"]
    D --> E["Delete phase: whole index is removed"]

What happens in each phase

Why the warm phase is the right moment

After rollover, the index no longer receives writes. Its segments can never become worse, so a compaction pays off for the rest of its life. Lifecycle management offers a force merge action for the warm phase, where you can set the target segment count and even switch to the higher-compression codec, so the merge rewrites data with better compression in the same pass.

Keep hot indices simple

Resist the urge to tune the hot phase heavily. Its natural merging is designed for exactly this workload. The main levers are the refresh interval, the shard size, and the hardware. Avoid force merging hot indices, as Chapter 13 explained.

Why deleting whole indices helps

Because deletion happens at the level of whole indices, none of the per-document cleanup burden appears. There are no deleted documents to merge away and no slow reclamation. This is one of the strongest reasons for time-based designs: they replace expensive document-level deletion with cheap file-level deletion.

Shrinking and merging

Some lifecycles also shrink an index, which reduces the number of primary shards, and this too requires a read-only index. Shrinking and force merging complement each other: fewer, larger shards with fewer segments are cheaper to keep and faster to search. The lifecycle order matters, so follow the documented order of actions for your version.

Fewer, larger segments make searches cheaper. But the benefit has limits, and merging itself can disturb searches while it runs.

Why fewer segments help

A search does its work segment by segment. For each segment, it looks up terms in that segment’s dictionary, reads postings, applies filters, and collects results, and then it combines the partial results. The steps for one segment have a fixed cost, no matter how small the segment is. With a hundred small segments, the fixed costs repeat a hundred times. With ten larger ones, they repeat ten times.

Other benefits of larger segments

• Better compression. Larger segments compress data better, so the same information takes fewer bytes to read from disk.
• More cache-friendly filters. The node query cache only considers segments that are big enough, above a minimum number of documents and a minimum share of the shard. Tiny segments do not qualify.
• Cheaper aggregations. Some structures used for grouping are built per segment, and fewer segments mean less repeated setup.
• Fewer open files. Less pressure on the operating system’s file handling.

The cost of merging to searches

Merges use disks and processors, so they take resources away from searches. A large merge reads and writes gigabytes, and this can add latency to searches that need to read from disk. New segments also start cold, meaning their data is not yet in the operating system’s file cache. The automatic throttle limits how hard merges push, so that searches stay responsive, and this is why it is best left on.

Diminishing returns

Going from two hundred segments to twenty gives a large improvement. Going from twenty to one gives a smaller one for most queries, and on an active index it is not sustainable. Measure before you invest effort in reducing segment counts further. On busy indices, the cost of rewriting data often outweighs the last few percent of search speed.

Effect on aggregations and sorting

Some grouping operations use structures called global ordinals, which map values from all segments into a single numbering. These are rebuilt whenever segments change, at refresh time. With frequent refreshes and many segments, rebuilding can be a noticeable cost for large grouping fields. A longer refresh interval, and fewer segments, both reduce this cost.

Vector fields

Indices that store dense vectors for similarity search keep a graph structure inside each segment. Merging these structures is more expensive than merging plain text data, because the graphs need to be combined. On such indices, merges take longer and use more CPU, so it is even more valuable to avoid creating many tiny segments and to watch merge times closely.

Every document you write is copied several times by merges before it reaches its final home. Understanding this hidden cost is the key to fast, steady indexing.

The hidden second write

When you measure disk writes on a node that indexes data, you will notice that the disk writes far more bytes than your clients send. The extra writing comes from merges. As Chapter 8 explained, each document is rewritten roughly once per tier. So if a client sends one gigabyte of documents, the disk may see several gigabytes of writes, depending on tiers, replicas, and the translog. This multiplier is called write amplification, and it is the main reason why indexing speed depends so much on disk speed.

Levers that reduce merge pressure

A bulk load recipe

Loading a large amount of data into a new index is the situation where merge tuning pays off most. A common sequence follows.

Update and delete workloads

Workloads with many updates create dead documents as fast as they create new ones. The tiered policy handles this by tolerating up to the allowed percentage and cleaning beyond it. If such an index causes heavy background merging, consider these approaches: reduce how often documents are rewritten by batching changes, use partial updates that do not touch large fields when possible, keep segments smaller with a lower maximum merged size so that cleaning does not rewrite giants, and check that a document really needs to be updated rather than appended as a new event.

Watch for signs of trouble

• Bulk requests start to be rejected or slow down while merges are at their thread limit.
• The log reports that indexing is being throttled.
• Disk write rates stay at their ceiling even when the incoming rate drops.
• Segment counts rise during the load and do not fall afterward.

A merge needs room to breathe. Planning for temporary space, disk speed, and processor time keeps merges from causing trouble elsewhere.

Temporary disk space

Because a merge writes the new segment before deleting the old ones, the disk holds both at the same time. The peak extra space is roughly the size of the merged output. A merge of ten segments of five hundred megabytes each needs several gigabytes of extra room. A force merge of a whole shard to a single segment needs, at the worst moment, about as much free space as the shard itself. Several merges running in parallel add up.

Disk watermarks

Elasticsearch protects nodes from running out of disk with three levels called watermarks. At the low watermark, new shards are no longer allocated to the node. At the high watermark, shards are moved away. At the flood stage, indices on the node become read-only. Merges can push a node across these lines by creating temporary space use. Keep enough headroom that a heavy merge cannot trigger them. Leaving a healthy share of every data disk free is a good habit for this reason alone.

Disk speed

Merges read and write large amounts of data in long streams. They benefit from disks with high sustained throughput. In cloud environments, disks may have a burst allowance, which is fast for a while and then falls to a lower baseline. A steady merge workload can use up the burst credits, after which everything slows down, including searches. Choose storage whose baseline speed matches your steady merge load, not just its peak.

CPU and memory

• CPU. Merges compress, decompress, and rebuild structures. On heavy indexing nodes, merge threads may use a large share of the processors.
• Memory. Merges mostly use the operating system’s file cache rather than the Java heap. A large merge may push some useful data out of the cache, which briefly slows searches.
• Heap. Merges use a moderate amount of heap for buffers. Extremely wide indices with many fields can increase this.

File handles

Each segment is several files, and every open segment holds file handles. Elasticsearch packs segments that are small relative to the shard into compound files, which bundle several files into one, to reduce the number of handles. Still, a shard with thousands of segments can run into the operating system’s limits. Keeping segment counts reasonable, and configuring the file descriptor limit as the installation guide recommends, avoids this class of problems.

Planning capacity

As a planning rule, plan for the live data size, plus the deleted share you tolerate, plus the working room for merges and force merges, plus a margin under the watermarks. For a workload with frequent updates or planned force merges, this margin needs to be larger than for pure append-only logs.

Every copy of a shard merges on its own. That simple fact explains surprising differences between replicas, the size of snapshots, and the speed of recovery.

Each shard copy merges independently

Elasticsearch replicates documents to replica shards, not segment files. A replica receives the same writes as the primary and builds its own segments. Because the timing of refreshes and merges differs slightly, the primary and its replicas contain the same live documents but usually have different segment layouts. A segment count of 42 on the primary and 38 on a replica is normal.

Consequences for caches

Caches that depend on segments, such as the node query cache, are per shard copy. A request served by a replica may find a cold cache even if the primary’s cache is warm. This is one reason why steering related requests to the same shard copies improves cache hit rates.

Peer recovery copies files

When a new replica is created or a lost one is rebuilt, Elasticsearch may copy segment files from the primary, followed by the recent operations from the translog. In that case, the new replica starts with the same segment layout as the primary at that moment, and afterward, they drift apart again. If history is kept long enough through retention leases, a returning replica can catch up by replaying operations instead of copying files, which is much faster.

Snapshots and segment churn

Snapshots to a repository are incremental at the segment file level. The first snapshot uploads all segment files. Later snapshots upload only files that did not exist in the earlier ones. Because segments are immutable, a file that survives untouched is not uploaded again. But a merge creates brand-new files. So an index with heavy merging produces a stream of new files that every snapshot must upload, while an index whose big segments stay untouched produces small incremental snapshots.

Order of operations for long-term storage

For data that you plan to keep for a long time, the sensible order is: stop writes, force merge, then take the snapshot, or mount it as a searchable snapshot. That way the repository stores the final, compact form once, instead of several intermediate forms.

Cross-cluster replication

A follower index in another cluster receives operations and builds its own segments, which merge on the follower’s own schedule. The merge behavior of the leader and follower are independent, and settings on one do not apply to the other automatically.

Restores and merges

A restored index arrives with the segments that were snapshotted. It does not need an immediate merge unless it will receive writes again. If it does, natural merging works from the restored segments, and huge restored segments follow the same half-ceiling rule as any others.

Merging is a bargain: you spend disk and processor time now to save search time and space later. Here is an honest scorecard so you can decide how much to spend.

The central trade-off: write cost against read cost

Merging more eagerly gives fewer segments and faster reads, but every extra merge rewrites data and burns disk and CPU. Merging less eagerly saves that effort but leaves more segments to search. The default settings sit at a point that suits mixed workloads. Write-heavy systems slide toward less merging, and read-heavy systems toward more.

The second trade-off: freshness against efficiency

A short refresh interval gives near-instant visibility of new data but creates many tiny segments and much merging. A long interval gives less merging and higher indexing efficiency but delays visibility. The right value is a business decision about how fresh the data must be.

The third trade-off: space against effort

Tolerating more deleted documents saves rewriting effort but wastes disk. Being strict keeps disks lean but costs more merging of big segments. Cheap, plentiful disks favor tolerance. Tight or expensive storage favors strictness.

The fourth trade-off: many small segments against one giant one

A single giant segment is the most efficient to search, but it cannot be changed cheaply. Any deleted documents inside it stay until it is rewritten in full. A moderate number of medium segments is easier to maintain. This is why the ceiling exists and why the force merge is best kept for read-only data.

When merge tuning is the wrong tool

If your searches are slow because of complex queries, bad mappings, oversized results, or too few resources, merge tuning will not help much. If your cluster is unstable because of too many shards, more merging makes it worse. Fix the design first, and tune merging last.

Start from the symptom, not from the setting. This chapter maps common problems to the least risky fix.

flowchart TD
    A["Symptom observed"] --> B{"Indexing slow or throttled?"}
    B -->|Yes| C["Lengthen refresh interval, check disk speed, reduce busy shards per disk"]
    B -->|No| D{"Searches slow with very many segments?"}
    D -->|Yes| E["Check that merges keep up, then consider a lower segments per tier"]
    D -->|No| F{"Disk larger than live data suggests?"}
    F -->|Yes| G["Check deleted share, consider a lower deletes percent allowed or a lower max merged segment"]
    F -->|No| H{"Old read-only index still heavy?"}
    H -->|Yes| I["Plan a one-time force merge in the warm phase"]
    H -->|No| J["Merging is probably not the problem: look at queries, mappings, and resources"]

Recommendations by workload

A safe experiment method

Good merging behavior is mostly the result of good design choices upstream. These seven habits keep merge work small and predictable.

Pattern 2 in action

A platform team stores application logs in a data stream. Each backing index rolls over at fifty gigabytes per shard or one day, whichever comes first. The hot phase uses a ten-second refresh interval and default merge settings. After rollover, the index moves to warm nodes, is made read-only, and is force merged to one segment per shard, with stronger compression. After thirty days, the whole index is deleted. The team has not touched a single merge policy setting in two years, and the disks are lean.

Pattern 5 in action

A product team originally updated a stock counter inside a large product document every time an item sold. That produced thousands of rewrites per hour and a high share of deleted documents in large segments. They moved the counter to a small separate document and reduced the update rate by batching. The deleted share dropped, background merging calmed down, and the team reduced disk headroom accordingly.

A pattern for documentation

Keep a one-page description of your merge-related choices: refresh intervals per index family, lifecycle rules, any non-default merge settings with the reason and the evidence, and the baseline numbers. When somebody joins the team or a cluster is rebuilt, this page prevents the loss of hard-earned knowledge.

These are the traps behind most merge-related incidents. Each is common and easy to avoid once you know it.

Other frequent mistakes

• Changing several merge settings at once and losing track of what worked.
• Expecting a single segment per shard on an active index.
• Comparing segment counts between primary and replica as if they must match.
• Copying merge settings from an old article written for a different version.
• Creating thousands of tiny indices, each of which merges on its own.
• Forgetting that a template only affects new indices and leaving old ones with old settings.

Merging seems like a pure performance topic, but it touches security in two ways: who is allowed to trigger heavy operations, and when deleted data physically disappears.

Deleted data is not immediately gone

When a document is deleted or replaced, its old bytes remain in a segment until a merge rewrites that segment. If a privacy rule says that personal data must be erased on request, marking a document as deleted does not physically remove it at once. In addition, some deleted documents are kept as soft-delete history for a retention period, and copies of the data may exist in snapshots and on replicas until their own cycles complete.

Design for erasure

If you must be able to erase data reliably and quickly, design for it. Separate data by tenant or by time so that whole indices can be deleted. Keep snapshot retention short enough for your obligations. For the rare case of deleting a single record from an active index, understand that removal from disk follows merging, and involve your compliance team in deciding what is acceptable.

Who may trigger heavy operations

Force merge and settings changes are administrative operations, and they require specific privileges on the index. A careless or malicious force merge on a big active index can consume disk and I/O and harm the cluster. Give these privileges to a small group of trusted operators and to lifecycle management, and audit their use. The same applies to changes of merge policy settings, which could deliberately or accidentally degrade performance.

Protecting the files

Segments are files on disk and contain your data in a searchable form, including deleted documents that have not yet been merged away. Protect them with disk or volume encryption and strict file permissions. Merges create new files with the same protections as the rest of the data directory, so no special configuration is needed, but the whole directory must be secured.

Resource abuse and availability

Availability is part of security. Uncontrolled merging can exhaust disk space, and disks that are full make nodes stop accepting writes. Alerts on disk space, limits on who can run heavy operations, and lifecycle policies that pace compaction all protect the cluster from accidents and abuse.

A merge policy that works is invisible. A few numbers tell you when it stops working, usually long before users notice.

Where to look

Important log lines

Metrics worth charting

• Segment count per shard, on average and at the maximum. Rising trends over days deserve a look.
• Deleted document share per index. Useful for update-heavy indices.
• Running merges and merge time. Show the effort behind the scenes.
• Merged bytes per hour. A direct measure of write amplification when compared with incoming bytes.
• Indexing throttle time. The clearest sign that merges cannot keep up.
• Disk free space and I/O wait. The resources merges depend on.
• Search latency percentiles. The customer-facing effect of everything above.

Alerts that help

Create a short list of alerts. First, an alert on indexing throttling lasting longer than a few minutes. Second, an alert when average segment count per shard rises far above its baseline. Third, an alert on the deleted document share for indices where you expect it to be low. Fourth, an alert when free disk space drops below a margin above the watermarks. Fifth, an alert when a merge or force merge has been running for an unusually long time. Each alert should link to a runbook.

Comparing before and after

When you change anything about merging, capture the same set of numbers before and after: segment counts, merged bytes per hour, throttle time, deleted share, indexing rate, and search latency. A change that improves one number and worsens others may not be a real improvement.

Merging is designed to be safe, so most failures come from the environment around it: disks, limits, and load. Each has a calm recovery.

flowchart TD
    A["Merge-related trouble"] --> B{"Errors about disk space?"}
    B -->|Yes| C["Free space, move shards, add capacity, then retry"]
    B -->|No| D{"Indexing throttled or merges lagging?"}
    D -->|Yes| E["Reduce write pressure, lengthen refresh interval, check disk speed"]
    D -->|No| F{"Too many open files or huge segment counts?"}
    F -->|Yes| G["Fix file limits, find why merges did not keep up"]
    F -->|No| H{"Corruption or checksum errors in logs?"}
    H -->|Yes| I["Let the shard recover from a healthy replica, investigate hardware"]
    H -->|No| J["Compare with baseline and look at queries, mappings, and resources"]

Scenario: the disk fills up during a merge

A big merge or force merge needs temporary space. If the disk runs out, the merge fails with a “no space left” error, and the shard may fail as well. Elasticsearch discards the incomplete new segment, and the old segments remain intact, so no committed data is lost. Recovery is to free space, for example by deleting old indices, moving shards to other nodes, or adding capacity, and then allow the shard to recover. Afterward, review your free space margin and the size of merges you run.

Scenario: merges lag and indexing is throttled

Under heavy write load, merges may not keep up. The scheduler throttles indexing to protect the shard. Recovery means reducing the pressure: lengthen the refresh interval, slow the sources, spread shards across more nodes, or move to faster storage. Once merges catch up, the throttle lifts by itself. If it happens often, treat it as a capacity or design problem, not as an incident.

Scenario: too many open files

If a shard accumulates thousands of segments, the process may run into the operating system’s limit for open files. Errors mention “too many open files.” Recovery is to make sure that the file descriptor limit is configured as the documentation recommends, and to find out why merges did not keep up, since a healthy shard should never have such a high count.

Scenario: a node crashes during a merge

Because merges write new files and switch them in at the end, a crash in the middle leaves the old segments valid. On restart, Elasticsearch removes incomplete files and recovers from the last commit and the translog. No special action is needed.

Scenario: a force merge is interrupted

The force merge request may run for a long time. If the client disconnects, the merge normally continues in the background. If the node restarts, the merge is cancelled, and the index remains in its previous valid state. You can start it again later. Running it with the option that returns a task lets you track and, if needed, cancel it.

Scenario: corruption found during a merge

Merging reads whole segments and checks their checksums. If a segment on disk is damaged, the merge notices, and the shard is marked as failed. The cluster then rebuilds that shard copy from a healthy copy on another node. This is a good reason to keep at least one replica of important indices, and to investigate the hardware that produced the corruption.

Scenario: burst credits run out

On cloud storage that has burst allowances, a long merge or a big force merge can use up the credits. After that, the disk slows to its baseline, and merges, searches, and indexing all suffer. Symptoms are a sudden drop in throughput without any change in load. Recovery is to reduce merge activity, wait for credits to return, or move to storage with a higher baseline.

Habits that make recovery easier

• At least one replica for indices that matter.
• Regular snapshots that you have practiced restoring.
• Alerts on disk space and throttle time.
• A runbook for disk full, merge lag, and file limit errors.
• A rule that heavy operations wait until free space and load are checked.

When something looks wrong, match the symptom to its likely meaning. This table covers the situations people meet most often.

A calm way to debug

Here are four common situations, with rough numbers, showing merge policy at work. The companies are imaginary, but the patterns are the ones teams meet every day.

A worked example with numbers

A merge policy is a simple idea with a big effect: combine small immutable pieces into fewer big ones, at a pace the hardware can sustain. Here is everything to remember.

Keep in mind that setting names, defaults, and available policies can change between Elasticsearch versions, so confirm exact details against the documentation of the release you run. With that habit, and the ideas in this guide, you can keep your indices tidy, your disks lean, and your searches quick without ever making merging a daily worry.