The Lucene Segment in Elasticsearch
Learn how Elasticsearch stores every search index as a pile of small, sealed, never-edited pieces called segments, why that odd design makes searching so fast, and how refresh, merge and delete behaviour all follow from it.
Imagine a huge public library that receives new books every hour. The librarian could squeeze each new book into the main catalogue by rewriting the printed pages, but that would be slow, messy and dangerous, because readers are using the catalogue at the very same time. So the librarian does something clever instead. New books arrive in sealed boxes. Each box gets its own small printed catalogue that is never changed after it is printed.
To find a book, a reader checks every box’s catalogue, one after another, and combines the answers. When a book is withdrawn, the librarian does not reprint anything. A separate sheet simply lists the crossed-out books. Late at night, the librarian takes several small boxes, throws away the crossed-out books, and prints one new, larger box with a fresh catalogue. Each of those sealed boxes is a Lucene segment, and this whole routine is how Elasticsearch keeps data searchable while it keeps changing.
1The Problem: Searching Fast While Data Keeps Changing
Segments exist because fast search and constant change pull in opposite directions.
Elasticsearch is a search and analytics engine. People use it to find products in an online shop, read application logs, investigate security alerts and analyse metrics. Two promises make it useful: searches come back in milliseconds, and new data becomes searchable within about a second. Keeping both promises at once is much harder than it sounds.
To answer a search quickly, the engine builds special lookup structures ahead of time. The most famous one is the inverted index, which works like the index at the back of a textbook. Instead of reading every page to find the word “engine”, you look up “engine” in the index and it tells you the pages. The catch is that these lookup structures are packed tightly and sorted carefully. Changing one small entry in the middle can force you to shuffle huge amounts of data around it.
Think of a dictionary printed on thin paper with every word in alphabetical order. If you want to add a new word starting with “M”, you cannot just squeeze it in. You would have to reprint every page after it. Now imagine doing this many times per second while thousands of readers are flipping the pages. That is the problem segments solve.
The clever trick: never edit, only add
Lucene, the search library that sits inside every Elasticsearch shard, solves this problem by refusing to edit anything it has already written. New documents are gathered together and written out as a brand-new, self-contained mini-index. Old mini-indices are never touched. Searching means looking through all the mini-indices and combining the answers. The mini-indices are called segments.
Once you understand this one idea, many puzzling behaviours of Elasticsearch stop being puzzling. It explains why an update is really a delete plus an insert, why deleted data still uses disk space for a while, why a document is not searchable the instant you index it, and why forcing everything into one segment is sometimes wonderful and sometimes terrible.
2What Is a Lucene Segment?
A plain definition first, then the words you will keep meeting.
A Lucene segment is a small, self-contained, immutable index that holds a batch of documents together with every lookup structure needed to search them. A shard in Elasticsearch is really one Lucene index, and a Lucene index is really a collection of segments plus a small file that lists which segments currently count.
Each segment can be searched entirely on its own. It has its own dictionary of terms, its own lists of which documents contain each term, its own stored copy of the original documents, and its own column-style data for sorting and counting. Nothing inside one segment refers to anything inside another segment. That independence is the source of both its speed and its simplicity.
Words you need to know
Document
One record you store, such as one product, one log line or one customer. It is like one card in a card box.
Lucene
The open-source search library at the heart of Elasticsearch. Elasticsearch adds clustering, an HTTP API and much more on top.
Segment
One sealed mini-index inside a shard. It is like one sealed box with its own printed catalogue.
Immutable
Cannot be changed after creation. Segments are written once and then only read, or eventually replaced.
Commit point
A small file that names exactly which segments make up the index right now. It is the official table of contents.
Refresh
The action that turns freshly indexed documents into a new searchable segment.
Merge
The background action that combines several segments into one larger segment and drops deleted documents.
Live docs
A per-segment record of which documents are still valid and which have been marked as deleted.
People often say “segment” and mean two different things. Inside Elasticsearch documentation, a segment always means a Lucene segment inside a shard. It has nothing to do with customer segments or network segments.
3Where Segments Live: Index, Shard, Lucene Index, Segment
Four nested layers are easy to confuse, so let us separate them clearly.
When you create an Elasticsearch index, it is divided into shards, and the shards are spread across the nodes of the cluster. Each shard copy is, under the hood, a complete Lucene index. That Lucene index is stored as many segments. So the nesting goes from the outermost container to the innermost piece: Elasticsearch index, then shards, then segments, then documents.
graph TD
I["Elasticsearch index"] --> S1["Shard 0"]
I --> S2["Shard 1"]
S1 --> L1["Lucene index inside the shard"]
L1 --> G1["Segment _0"]
L1 --> G2["Segment _1"]
L1 --> G3["Segment _2"]
G1 --> D1["Documents, terms, doc values, stored fields"]
G2 --> D2["Documents, terms, doc values, stored fields"]
G3 --> D3["Documents, terms, doc values, stored fields"]
| Layer | What it is | Who decides it | Everyday comparison |
|---|---|---|---|
| Elasticsearch index | A named collection of similar documents | You | A whole library |
| Shard | A slice of the index, held on one node | You choose the count, Elasticsearch places them | One wing of the library |
| Lucene index | The search engine data inside one shard copy | Elasticsearch | The catalogue system of that wing |
| Segment | One immutable mini-index inside that Lucene index | Elasticsearch and Lucene, automatically | One sealed box of books with its own catalogue |
| Document | One stored record | You | One book |
You do not manage segments directly
You choose how many shards an index has, and you decide when data arrives. You do not create segments by hand. Elasticsearch and Lucene create them as a natural result of indexing activity, and they merge them without asking you. What you can do is influence how many segments appear and how quickly they are merged, through settings and habits that we will explore in later chapters.
A shard is not the same as a segment. A shard is a logical slice that lives on a node. A segment is a physical piece inside a shard. One shard can hold a few segments or hundreds, and their number changes constantly as data arrives and merges happen.
4Why Segments Are Immutable
Refusing to edit sounds like a limitation, yet it is the secret of the design.
Immutable means “cannot be changed after it is created”. Once Lucene has written a segment to disk, no program changes a single byte of it. If something must change, Lucene writes a new segment and eventually retires the old one. This sounds wasteful, but it brings several large advantages.
No locking for readers
Because nobody edits a sealed segment, many searches can read it at once without waiting for each other or worrying about half-finished writes.
Friendly to the operating system cache
Unchanging files stay in the memory the operating system keeps for recently read files, so repeated searches are served from memory instead of disk.
Heavy compression
Data that never changes can be packed tightly and sorted perfectly, because nobody will need to squeeze something into the middle.
Stable caches
Lookups cached for a segment never become wrong, because the segment never changes. Only a retirement of the whole segment clears them.
Simple crash safety
A segment is either completely written and listed in the commit point, or it is ignored. There is no such thing as a half-updated segment.
Easy backups
Snapshots can copy segment files once and reuse them next time, since the files never change. That makes snapshots incremental.
A printed newspaper is immutable. Once it is on the doorstep, nobody edits it. If there is a mistake, the paper prints a correction in tomorrow’s issue. Readers never wonder whether the copy in their hands secretly changed overnight, and the paper never needs to chase down each reader to fix a copy.
The price you pay
Immutability has costs. Deleting a document cannot remove it from the segment, so deleted documents linger until a merge. Updating a document cannot rewrite it in place, so an update writes a whole new copy. New documents are not searchable until a new segment exists, so search is near-real-time and not instantaneous. And because segments pile up, Elasticsearch must spend effort merging them. Almost every performance topic in this tutorial is a question of managing these costs.
Elasticsearch is built for data that is written once and read many times. The more your workload behaves like that, the better segments serve you. Workloads with constant heavy updates to the same documents fight the design and need extra care.
5Inside a Segment: The Data Structures
A segment is not one thing. It is a bundle of specialised structures, each good at a different job.
If you opened a segment and looked inside, you would find several different ways of storing the same documents. This is deliberate. A structure that is excellent for finding which documents contain a word is poor at sorting by price, and one that sorts well is poor at rebuilding the original document. So Lucene stores several, side by side.
Inverted index
Maps every unique term to the list of documents that contain it. This is what makes full-text search fast. It is the index at the back of the book.
Stored fields
Keeps the original field values of each document, including the original source, so results can show the full record. It is the book itself.
Doc values
A column-oriented store that lists one field’s value for every document in order. It powers sorting, aggregations and scripts.
Norms
Small numbers that record how long each field is, used to score how relevant a match is. A word in a short title counts for more than in a long essay.
Points
Special tree structures for numbers, dates and geographic locations, so range questions like “price between 10 and 50” are quick.
Vector structures
Graph-like structures for similarity search over numeric vectors, present when you use dense vector fields.
Live docs
A simple record of which documents in the segment have been deleted. It is the only part that changes over time, and it is kept as a separate small file.
Segment metadata
Facts about the segment itself, such as how many documents it holds, which Lucene version wrote it and which fields it contains.
Why several structures for the same data
A cookbook has the recipes themselves, an index at the back listing which recipes use tomatoes, and a table listing every recipe’s cooking time in one column. Each helps with a different question. Finding tomato recipes uses the back index. Finding the quickest recipe uses the time table. Reading a recipe uses the page itself. A segment carries all of these together.
This is why Elasticsearch field settings matter for disk use. Turning off doc values for a field you never sort by, or turning off norms for a field where relevance scoring is pointless, means the segment simply does not contain those structures for that field.
6Segment Files on Disk
If you look at a shard’s folder, you will see files that map neatly to the structures above.
A segment is stored as a group of files that share a segment name. Lucene names segments with short base-36 identifiers, so you will see names such as _0, _1, _a and _1f. The shared name tells you which files belong together. The file extensions tell you what each file holds.
| File kind | Typical extension | What it holds |
|---|---|---|
| Segment info | .si | Basic facts about the segment |
| Field info | .fnm | The list of fields and how they are indexed |
| Term dictionary and its index | .tim and .tip | The sorted list of terms and a small map to find them fast |
| Postings | .doc, .pos, .pay | Which documents contain each term, at which positions |
| Stored fields | .fdt, .fdx, .fdm | The original document values |
| Doc values | .dvd and .dvm | Column data for sorting and aggregations |
| Norms | .nvd and .nvm | Field length information for scoring |
| Points | .kdd, .kdi, .kdm | Tree data for numbers, dates and geo shapes |
| Live documents | .liv | Which documents are still alive |
| Compound container | .cfs and .cfe | Several of the above bundled into one file |
| Commit point | segments_N in the shard folder | The list of segments that make up the index |
Exact extensions and the presence of some files change between Lucene versions. Treat the table as a guide to the idea, not as a fixed specification, and never edit or delete these files by hand, since Elasticsearch relies on them being exactly as it wrote them.
The compound file format
Each segment can have a dozen or more files. Thousands of segments across many shards would mean a very large number of open files, and operating systems limit how many files a program can hold open. To reduce the count, small segments are often packed into a compound container, which is one or two files that hold everything. Large merged segments usually skip this, because packing them costs more than it saves. Elasticsearch controls the choice with a setting called index.compound_format.
A sealed box might contain twelve separate booklets, or the booklets could be bound together in a single folder. The information is the same, but a single folder is easier for the librarian to carry. For tiny boxes, binding is worth it. For huge boxes, the binding effort is not.
Elasticsearch documentation asks operators to raise the operating system limit on open files well above the default. Segments are one of the reasons. Many shards with many segments each can otherwise exhaust the limit.
7Document Numbers Inside a Segment
Every document gets a small private number, and understanding it explains several odd behaviours.
Inside a segment, Lucene gives each document a small whole number called a document ID, starting from zero and counting upward in the order the documents were added to that segment. These numbers are private to the segment. Segment _0 has its own document 0, and segment _1 also has its own document 0. They are unrelated.
This is different from the _id field that you see in Elasticsearch, which is the identifier you or Elasticsearch assigned to the document. The internal number is only a compact position inside the segment. It is used everywhere inside the search engine because small numbers compress beautifully and can be stored in fast lists.
In every sealed box, books are numbered one, two, three from the top. Book one in box A and book one in box B are different books. A reader wanting a specific book uses its title, which is the public _id, and the librarian translates that into the box and the position number.
Why internal numbers change after a merge
When several segments are merged into a new one, documents are renumbered from zero in the new segment. Documents that were deleted are simply not copied, so the new numbers pack tightly without gaps. This is why nobody should rely on the internal number, and why Elasticsearch never exposes it as a stable identifier.
Nested documents live as a block
If you use nested fields, each nested object is stored as its own hidden Lucene document, placed right next to its parent inside the same segment, as one adjacent block. Because the block must stay together, updating one nested item means rewriting the whole parent block as new documents. This is one reason why heavy updates to documents with many nested items are expensive, and why the document count you see internally can be larger than the count of top-level documents you indexed.
Statistics that count Lucene documents include nested hidden documents. If you index one thousand orders that each have ten nested items, the internal document count can be far larger than one thousand. Keep this in mind when reading segment statistics.
8The Life of a Document: Buffer, Refresh, Flush
Follow one document from the moment it is sent until it is safely stored in a segment.
When you send a document to Elasticsearch, it does not instantly become a segment. Creating a segment for every single document would produce millions of tiny segments and crush the system. Instead, documents travel through stages, and the segment is created when a batch of them is ready.
flowchart TD
A["Client sends a document"] --> B["Primary shard receives the request"]
B --> C["Document goes into the in-memory indexing buffer"]
B --> T["Same operation is appended to the translog for safety"]
C --> R{"Refresh happens?"}
R -->|Yes| S["New segment is written and opened for search"]
S --> Q["Document is now searchable"]
S --> F{"Flush happens?"}
F -->|Yes| K["Segments are fsynced, commit point is written, translog is trimmed"]
F -->|Not yet| M["Segments wait, protected by the translog"]
Q --> G["Background merges combine small segments later"]
Arrival at the primary shard
The request is routed to the correct primary shard, which checks it, assigns it a version and prepares to index it.
Buffering in memory
The document is analysed, meaning its text is broken into terms, and added to an in-memory structure. It is not yet searchable, and not yet a real segment on disk.
Writing the translog
At the same time, the operation is recorded in a simple append-only log called the translog, so it can be replayed if the node crashes before the segment is safely stored.
Refresh creates a segment
On a schedule, the buffered documents are written out as a new segment, which is opened for searching. From this moment the document appears in results.
Replication
The primary also forwards the operation to replica shard copies, and each copy builds its own segments from it independently.
Flush makes it durable
Periodically, a flush forces segment files to durable disk storage, writes a new commit point, and lets the translog discard the operations that are now safely stored.
Merging cleans up
Over time, background merges combine small segments into bigger ones and drop deleted documents.
Many indexing threads, many small segments
Lucene lets several indexing threads work at the same time, and each thread builds its own in-memory piece. When a refresh happens, each piece that holds data is written as its own segment. So a busy shard can create several segments in a single refresh. This is normal, and it is one more reason why merging exists.
A refresh is not the only trigger. If the in-memory buffer fills up before the next scheduled refresh, Lucene writes a segment early to free the memory. The size of the buffer is controlled by an indexing buffer setting that by default is a fraction of the node’s heap.
9Refresh: Making New Documents Searchable
This is why Elasticsearch is called near-real-time.
A refresh creates a new segment from the recently buffered documents and opens it for search. By default, Elasticsearch refreshes each shard once every second, using the index setting index.refresh_interval. That is why a newly indexed document typically appears in search results after about one second, not instantly.
A refresh is cheaper than a full commit. It writes segment files to the file system, where they sit in the operating system’s cache and can be read immediately. It does not force them onto durable disk, which is a separate and slower step handled by the flush. Splitting these two jobs is what lets Elasticsearch offer both speed and safety.
Refresh is like a shop assistant placing newly arrived items on the display shelf so customers can see them at once. Flush is like the evening stocktake when the manager records everything into the permanent ledger. Customers do not wait for the ledger to see the shelf.
Settings and options you can use
| Control | What it does | When to use it |
|---|---|---|
| Refresh interval, default one second | How often a scheduled refresh creates a new segment | Keep for interactive search that needs fresh data |
| Refresh interval raised to a larger value | Creates fewer, larger segments and saves work | Log ingestion or bulk loads where a delay is acceptable |
| Refresh interval set to minus one | Turns scheduled refresh off | Initial bulk loads, followed by a manual refresh |
| Refresh parameter on a request set to true | Forces an immediate refresh after that request | Rarely, such as in tests, because it creates tiny segments |
| Refresh parameter set to wait_for | Makes the request wait until the next normal refresh | When a client must see its own write, without forcing extra refreshes |
| Manual refresh API | Creates a segment on demand for an index | After a bulk load with scheduled refresh disabled |
Idle shards rest
In modern versions, a shard that has not received any search request for a while, thirty seconds by default, stops doing its scheduled refreshes. The next search request wakes it up and triggers a refresh first. This saves a lot of work for indices that are written constantly but searched rarely, such as archived logs.
Adding the immediate-refresh option to every single indexing request feels safe, but it forces a new tiny segment for each request. The result is thousands of tiny segments, heavy merge work and slow indexing. Batch documents instead, and use wait-for behaviour when a client truly needs to read its own write.
10Translog and Flush: Making Segments Durable
A segment that is searchable is not automatically safe. Durability has its own machinery.
After a refresh, the new segment exists as files, but the operating system may still be holding the bytes in memory and may not have written them to the physical disk. If the machine lost power at that moment, the segment could be lost. Forcing every refresh onto physical disk would be far too slow. Elasticsearch solves this with the translog, short for transaction log.
The translog is a simple file that records each indexing, update and delete operation as it happens. Writing to a simple append-only log is quick. By default, Elasticsearch makes sure the translog is safely on disk before it acknowledges each request, controlled by the setting index.translog.durability. If a node crashes, on restart it replays the translog operations that were not yet part of a committed segment set.
A shop assistant quickly jots every sale on a notepad the moment it happens. The permanent ledger is only updated in the evening. If the shop is flooded overnight, the notepad, which was kept in a waterproof bag, lets the owner rebuild the ledger. The notepad is the translog.
What a flush does
A flush is the step that makes segments durable. It forces all outstanding segment files onto physical disk, writes a new commit point that lists them, and then lets Elasticsearch start a new translog generation and discard the old operations, since those are now safely inside committed segments. Flushes happen automatically, for example when the translog grows beyond a size threshold, so you rarely need to trigger one yourself.
| Step | Makes documents searchable? | Makes them durable? | Relative cost |
|---|---|---|---|
| Adding to the indexing buffer | No | No, but the translog protects it | Very low |
| Writing the translog | No | Yes, once safely on disk | Low |
| Refresh | Yes | Not by itself | Medium |
| Flush and commit | Not needed for this | Yes, in the segments themselves | Higher |
| Merge | Keeps search efficient | Produces new segments that later become committed | Highest, in the background |
Searchability and durability are separate questions. Refresh answers “can it be found?” and flush plus the translog answer “will it survive a crash?” Understanding this split removes most confusion about why documents sometimes seem to appear before they are permanently committed.
11How Deletes and Updates Really Work
Because segments never change, removing or changing a document is done by bookkeeping rather than editing.
Deleting a document
When you delete a document, Elasticsearch does not remove it from its segment. Instead, it marks the document as deleted in a separate small structure called the live-docs record. Searches consult this record and skip any document marked as deleted. The original bytes stay inside the segment, taking disk space, until a merge builds a new segment without them.
Updating a document
An update is handled as a delete plus an index. The old version is marked deleted in whichever segment holds it, and a brand-new version of the complete document is added to the current buffer, which will become a new segment at the next refresh. Even if you change one tiny field, the whole document is rewritten. The partial update feature of Elasticsearch is a convenience: it reads the old document, applies your change, and indexes the merged result as a new document.
sequenceDiagram
participant C as Client
participant P as Primary Shard
participant O as Old Segment
participant N as New Segment
C->>P: Update document 42
P->>O: Mark document 42 as deleted in live-docs
P->>N: Index the complete new version of document 42
Note over O: Old bytes remain until a merge
Note over N: New version searchable after refresh
C->>P: Search
P->>O: Skip documents marked deleted
P->>N: Return the new version
| Operation | What happens inside segments | Immediate disk effect |
|---|---|---|
| Index a new document | Added to buffer, later written to a new segment | Space grows at next refresh |
| Delete a document | Marked as deleted in the segment that holds it | No space is freed yet |
| Update a document | Old copy marked deleted, full new copy added elsewhere | Space grows, old copy remains |
| Delete by query | Finds matches, marks each as deleted | No space freed until merges |
| Merge | Copies live documents to a new segment, drops deleted ones | Space is eventually freed |
Deleted documents still leave footprints
Until they are merged away, deleted documents still occupy disk. They also still contribute to some scoring statistics, such as how many documents contain a term, so relevance scores can shift slightly after a merge cleans them out. And in modern versions, Elasticsearch keeps some deleted operations around for a retention period, using a feature called soft deletes, so that replicas and other copies can catch up by replaying recent history. These retained deletions are also cleaned up during merges once their retention ends.
A workload that updates the same documents again and again, such as a product catalogue with constant price and stock changes, can accumulate a large share of deleted documents. Watch the deleted-document count of your indices and keep the update rate in mind when planning disk capacity.
12Segment Merging: Cleaning the Shelves
Merging is the background chore that keeps immutable segments from piling up forever.
Because every refresh can add a new segment, a shard would soon hold hundreds of tiny segments if nothing cleaned up after it. A search would then have to look inside every one of them. So Lucene continuously runs merges. A merge picks several segments, reads all their live documents, writes them into one new larger segment, and then retires the old segments once the new one is ready and committed.
A librarian notices twelve small sealed boxes on the shelf. Late at night, the librarian takes ten of them, unpacks the books, throws away the crossed-out ones, and packs the rest into one fresh box with a new printed catalogue. Readers now have one catalogue to check instead of ten, and the discarded books no longer waste space.
graph LR
A1["Small segment 5 MB"] --> M1["Merge"]
A2["Small segment 4 MB"] --> M1
A3["Small segment 6 MB"] --> M1
A4["Small segment 5 MB"] --> M1
M1 --> B1["Medium segment about 20 MB"]
B1 --> M2["Later merge with similar medium segments"]
B2["Medium segment 22 MB"] --> M2
B3["Medium segment 19 MB"] --> M2
M2 --> C1["Large segment about 60 MB"]
The tiered merge policy
Deciding which segments to merge is the job of a merge policy. Elasticsearch uses Lucene’s tiered merge policy by default. It groups segments into size tiers and tries to merge segments of similar size together, so that the same big data is not copied again and again. It also gives extra attention to segments that hold many deleted documents, because merging them recovers the most space.
Segments per tier
Roughly how many segments of a similar size are allowed before a merge is wanted. The default is around ten.
Maximum merge at once
How many segments a single merge may combine. The default is around ten.
Maximum merged segment size
A ceiling on how big a naturally merged segment grows, by default about five gigabytes.
Merge scheduler threads
How many merges may run at the same time on a shard, sized from the number of processors by default.
Merging costs real resources
A merge reads a lot of data and writes a lot of data. It uses disk bandwidth and processor time, and it temporarily needs extra disk space because old segments remain until the new one is ready. That is why Elasticsearch runs merges in the background and slows them down when the disk is busy, and why running out of disk space can be dangerous during heavy merge activity.
The defaults are excellent for most users. Before changing merge settings, first check whether the real problem is too many tiny refreshes, too many tiny shards, or slow disks. Tuning merges rarely fixes those root causes.
Some teams try to disable merging to speed up indexing. This quickly creates an enormous number of segments, slows every search, and can exhaust file handles. Merging is not optional overhead. It is part of the price of immutability.
13Force Merge: A Powerful Tool to Use with Care
You can ask Elasticsearch to merge segments right now, but that is not always wise.
The force merge API tells Elasticsearch to merge the segments of one or more indices down to a chosen number of segments right away. It can also be told only to expunge deleted documents. The most common use is to merge an index down to a single segment, which yields the smallest search overhead and reclaims all deleted space.
When force merge is a great idea
Force merge shines on indices that will no longer receive writes. A daily log index from last week is a perfect example. Once nothing will be added, merging it into very few segments gives faster searches, lower memory pressure and smaller disk use. Index lifecycle management can do this automatically in a warm or cold phase, once an index has rolled over and become read-only.
When it is a bad idea
Never force merge an index that is still being written to. The natural merge policy stops merging segments once they reach its maximum size, and it treats segments above that size in a special way. A giant force-merged segment on an active index will not be merged again until most of its documents are deleted, so it slowly fills with deleted documents and wastes space. New writes then create small segments next to it, which is exactly the uneven situation the tiered policy tries to avoid.
| Situation | Force merge? | Reason |
|---|---|---|
| Rolled-over log index that is now read-only | Yes | No new writes, so one compact segment is stable and efficient |
| Static reference data loaded once | Yes | Fewer segments gives faster searching |
| The index currently receiving writes | No | Creates a giant segment that natural merges will neglect |
| Nightly job on a busy write index | No | Costly disk work with little lasting benefit |
| A read-only index with many deleted documents | Yes, possibly expunge only | Reclaims space held by deleted documents |
Force merge is heavy work that can saturate disks and temporarily double the space an index needs. Run it during quiet periods, confirm there is free disk space first, and never point it at a whole cluster without thought.
14Searching Across Segments
A search does not look at a shard as a single block. It visits each segment.
When a search reaches a shard, the search engine treats each segment as a separate mini-index. It runs the query against each segment, collects the best matches from each, and merges those partial lists into the final answer for the shard. The coordinating node then merges the shard answers into the final response. Because each segment is independent, this work splits naturally into small pieces.
sequenceDiagram
participant U as User
participant C as Coordinating Node
participant S as Shard
participant G1 as Segment A
participant G2 as Segment B
participant G3 as Segment C
U->>C: Search request
C->>S: Run query on this shard
S->>G1: Query segment A
S->>G2: Query segment B
S->>G3: Query segment C
G1-->>S: Top matches from A
G2-->>S: Top matches from B
G3-->>S: Top matches from C
S-->>C: Combined top matches, deleted documents skipped
C-->>U: Final results
Why segment count matters for search speed
Every segment adds a small fixed cost. The engine must look up each search term in each segment’s own dictionary, and it must combine more partial results. A shard with three segments does three sets of lookups and a shard with three hundred does three hundred, even if the total number of documents is identical. That is why merging gives a real speed benefit, and why very large segment counts feel slow.
Skipping deleted documents
For each match, the engine checks the live-docs record for the segment and ignores documents marked as deleted. This check is fast, but a segment with a high share of deleted documents wastes effort visiting entries that will be discarded.
Recent Elasticsearch versions can spread the work of searching groups of segments across several threads within a shard for some kinds of requests. The details depend on the version and the request type, so check the documentation for your release before relying on this behaviour.
More segments does not mean more parallel speed in a simple way. Few large segments usually give better cache behaviour and less per-segment overhead. A healthy shard has a modest number of segments, in a natural spread of sizes.
15Segments and Caching
Immutability makes caching safe, and it also decides what gets cached.
Elasticsearch keeps several caches to avoid repeating work. Many of them are tied to segments, because a segment never changes and so a cached answer about it stays true for as long as the segment exists. When a segment is retired by a merge, the cached facts about it simply disappear, and new segments start with a clean slate.
Query cache
Remembers which documents matched a commonly used filter, kept separately for each segment. Small segments are usually skipped, since caching them is not worth the overhead.
Request cache
Remembers whole results of certain searches on a shard. It is cleared whenever a refresh changes what the shard contains.
Operating system file cache
The machine keeps recently read segment files in spare memory. This is the most important cache for search speed.
Global ordinals
A lookup that lets aggregations on text-like fields work across segments. It must be rebuilt when new segments appear.
Why frequent refreshes make caches less useful
Every refresh that changes the shard makes the request cache invalid for that shard. Every new segment needs its own cache entries, and every change to the set of segments means global ordinals must be rebuilt for fields that use them. A shard refreshing every second while receiving constant writes gets much less benefit from caching than a shard that is stable. This is one more reason why bulk-heavy workloads that can tolerate a slower refresh often see better search performance too.
A librarian writes helpful notes on sticky papers attached to each sealed box: “the mystery shelf is in the third row”. As long as the box stays the same, the note stays useful. When the box is replaced by a new bigger box, the notes must be written again. Frequent box swapping means constant rewriting of notes.
Elasticsearch documentation explains that the query cache by default only considers larger segments, roughly those with at least ten thousand documents that also form a meaningful share of the shard. Tiny segments do not use it, which is why an index made of many tiny segments gains little from that cache.
16Replicas, Snapshots and Segments
Copies and backups both depend on the sealed nature of segments.
Each replica builds its own segments
Elasticsearch keeps replica copies of each shard for safety and extra search capacity. In the traditional design, replication works at the level of operations. The primary sends each indexing operation to its replicas, and each replica builds its own segments from those operations independently. Refreshes on different copies happen on their own schedules, and merges happen independently too.
As a result, the primary and its replicas contain the same documents but usually have different segment layouts. Segment counts, sizes and names differ between copies, and a document may become searchable on one copy a moment before another. This is normal, and it is why two searches on the same index can very briefly see slightly different sets of the newest documents.
Two libraries receive the same list of new books by post. Each unpacks and shelves the books on its own timetable, and each packs its own sealed boxes. Both end with the same books, but the boxes are not identical.
Snapshots love immutability
A snapshot is a backup of indices stored in a repository. Because a finished segment never changes, a snapshot only needs to copy the segment files that are not already in the repository. Segments that survived unchanged since the last snapshot are simply referenced again. That makes snapshots incremental and space-efficient without any special effort.
A side effect is that merges influence snapshot size. A merge creates new segments, and the next snapshot must copy those new files even though the documents inside are old. Very frequent merging of huge segments can therefore create larger incremental snapshots than you might expect. Feature areas that read segments directly from snapshot storage, such as searchable snapshots, also rely on segments being stable, complete files.
Force merging read-only indices before taking a long-term snapshot often reduces how much data must be stored and how long restores take, because fewer and larger segments carry less overhead and no deleted documents.
Copying a shard’s segment files by hand is not a valid backup method for a running cluster. The files change as merges and commits proceed, and a consistent picture needs the commit point machinery. Use the snapshot feature instead.
17Memory, the File System Cache and Segments
Segments are files, so the machine’s memory outside the Java heap matters a great deal.
Many people expect Elasticsearch to keep all search data inside its Java heap. It does not. Elasticsearch relies heavily on the operating system’s file system cache. When a segment file is read, the operating system keeps a copy in unused memory. The next read is served from memory, which is dramatically faster than reading from disk.
Modern versions map many segment files directly into memory, an approach called memory mapping. The default store type chooses memory mapping for the file kinds that benefit most and ordinary reads for the rest. This means much of a segment’s data, including the map used to find terms, lives outside the heap, and the operating system decides which parts stay in memory based on how recently they were used.
What this means for capacity planning
Setting an enormous Java heap starves the file system cache. Elasticsearch documentation advises leaving at least half of the machine’s memory for the operating system. A machine whose segments mostly fit in that leftover memory answers searches quickly. A machine where segments are far larger than available memory must keep going to disk, and searches slow down.
Imagine a librarian who keeps the most requested boxes on the desk instead of walking to the basement every time. The bigger the desk, the more boxes fit, and the fewer trips are needed. The heap is the librarian’s notebook, while the file system cache is the size of the desk.
Giving the Java heap most of the machine’s memory in the belief that more heap means more speed. Beyond a sensible size, this shrinks the file system cache, hurts segment reads and worsens garbage collection pauses. Follow the sizing guidance in the official documentation.
Fields you only filter or sort by can often be served entirely from doc values, which is column data in segments. Design mappings so that the hot structures you need are small, and disable structures you never use, so the cache holds what matters.
18Many Small Segments or Few Large Ones?
Neither extreme is perfect. The goal is a healthy natural spread.
| Aspect | Many small segments | Few large segments |
|---|---|---|
| Search speed | Slower, each segment adds lookups | Faster, fewer lookups and merges of partial results |
| Indexing cost | Cheap at first, then heavy merge work follows | Fewer merges needed, but a new large merge is a big job |
| Freshness | New data visible quickly, since each refresh is small | Freshness depends on refresh, not segment size |
| Disk use | Deleted documents can hide across many segments | Space is reclaimed on merge, but huge segments merge rarely |
| Open files | Many files and more file handles | Fewer files |
| Caching | Small segments often skip the query cache | Caches work well |
| Memory pressure | More per-segment overhead | Less overhead |
The healthy middle
A healthy write-heavy shard usually holds a mixture: a few big older segments, several medium ones and a handful of tiny fresh ones. This natural pyramid is exactly what the tiered merge policy aims for. A healthy read-only shard, on the other hand, is best when merged down to very few segments, since no new writes will disturb it.
A well-run warehouse has a few huge pallets of old stock, some medium crates and a few fresh cartons by the door. A warehouse made only of loose cartons is slow to search. A warehouse forced into one giant pallet cannot easily receive new goods.
Shard count and segment count work together
Each shard has its own set of segments. Splitting an index into too many small shards multiplies the number of segments, files and per-shard overhead across the cluster. So tiny shards create the same many-small-pieces problem at a higher level. Choosing a sensible shard size is often the strongest way to keep segment counts sensible, before any merge tuning is considered.
19Inspecting Segments: APIs and Statistics
You can look at segments without touching them. Doing so is the best way to build intuition.
Elasticsearch offers several read-only ways to see what segments a shard holds. The most direct are the segments API and the cat segments API. They list each segment, its size and its document counts, and they show important flags such as whether the segment is committed and whether it is searchable.
| Field you may see | What it tells you |
|---|---|
| Segment name | The base-36 name that groups the files of one segment |
| Generation | A counter showing the order in which segments were created |
| Docs count | How many live Lucene documents the segment holds, including hidden nested ones |
| Docs deleted | How many documents in the segment are marked deleted and waiting for a merge |
| Size | Disk space used by the segment |
| Committed | Whether the segment is included in a durable commit point |
| Searchable | Whether searches can currently see the segment |
| Version | The Lucene version that wrote the segment |
| Compound | Whether the segment’s files are packed into a compound container |
Other useful views
Cat indices
Shows total documents and deleted documents per index. A high deleted count signals bloat.
Index stats
Reports segment counts, merge totals, refresh totals and flush totals, useful for spotting unusual activity.
Node stats
Shows current merges, merge time and throttling, which reveal whether merges are keeping up.
Disk usage analysis
An analysis API can estimate how much space each field and structure uses inside segments, showing where your bytes go.
On a test index, index a few documents, look at the segment list, wait for a refresh, list again, then delete a document and list a third time. Watching the segment count and deleted-document numbers move teaches the whole lifecycle in a few minutes.
20Real-World Examples
Segment behaviour shows up in very different everyday workloads.
Log analytics with daily indices
A company ingests billions of log lines. Each day’s index receives heavy writes, so it holds many segments and merges constantly. Once the day rolls over, no more writes arrive. A lifecycle policy then force merges the index to very few segments, shrinks its disk use and speeds up week-old searches. The team gets cheap fast archives from the same segment machinery.
E-commerce catalogue with frequent price changes
An online shop updates prices and stock counts thousands of times an hour. Each update writes a whole new product document and marks the old one deleted. Deleted documents accumulate, and the index uses much more disk than the number of live products suggests. The team monitors deleted counts, provisions extra disk and moves rapidly changing counters into a smaller separate index.
Security monitoring
A security team indexes events and needs alerts within seconds. They keep the one-second refresh on the hot index, accept many small segments, and rely on merges to keep things tidy. Older events move to cheaper nodes and are merged down, trading freshness for compactness as data ages.
Nightly bulk load of reference data
A data team reloads a product reference index each night. They turn scheduled refresh off and set replicas to zero during the load, run a manual refresh at the end, then restore replicas. Fewer, larger segments are written and the load finishes much faster.
Metrics and time-series data
A monitoring platform stores measurements in time-based backing indices. Older backing indices are read-only, so their segments are compact and stable, while only the current index is busy. Caches stay warm for old data, since its segments never change.
Privacy deletion requests
A company must remove a customer’s data on request. The team learns that deleting documents only marks them deleted, so they verify that merges, index rewrites and snapshot retention together really remove the data from every physical copy within the promised time.
21Security and Compliance Considerations
Segments are files on disk, and files on disk raise security questions.
Deleted does not mean erased
This is the most important security fact about segments. When you delete a document, its bytes remain in the segment until a merge rewrites it, and they may also remain in older commit points, snapshots and replica copies. If a regulation requires that personal data be truly removed within a time limit, you cannot rely on a delete request alone. You must plan how merges, snapshot expiry and, if needed, index rewrites will complete the job.
Encryption at rest
Segment files contain your data in readable structures. Use disk or volume encryption so that stolen disks or copied files reveal nothing.
File permissions
Restrict the data directory so that only the Elasticsearch service account and administrators can read it.
Snapshot retention
Snapshots hold segment files for as long as the snapshot exists. Align retention with your deletion promises.
Field-level security
Access rules filter what a user sees at search time, but the segment still contains the full data. They protect querying, not raw files.
Anyone who can copy the raw shard files can read every field, whatever access rules exist inside Elasticsearch. Treat the data directory and the snapshot repository with the same care as the database itself.
Forcing physical removal
To speed removal of deleted documents from disk, operators can run a force merge that only expunges deleted documents on read-only indices, or reindex the surviving documents into a new index and delete the old one. Both approaches take real resources, so plan them. Also remember that removal from segments does not remove data from older snapshots, which must expire or be deleted separately.
22Performance Tuning, Monitoring and Metrics
Tune for the workload you really have, and measure before and after.
Tuning for heavy indexing
When your priority is getting data in quickly, the goal is to create fewer, larger segments and to avoid wasted work. The most effective steps are simple habits rather than exotic settings.
Use bulk requests
Send many documents per request. This cuts per-request overhead and lets each indexing thread build larger in-memory pieces.
Raise the refresh interval
If a delay of thirty seconds or more is acceptable, fewer refreshes create fewer tiny segments and cut merge work. For a one-time load, disable it and refresh once at the end.
Reduce replicas during the initial load
Each replica repeats the indexing work and builds its own segments. Loading with no replicas and adding them afterwards is often faster, provided you accept the temporary risk.
Let the indexing buffer be generous
A larger buffer lets segments be written larger before they are flushed to disk, which lowers the number of small segments.
Skip structures you do not need
Every field with an inverted index, doc values and norms adds work. Disable what you will never use for that field.
Tuning for heavy searching
For search-heavy indices, the goal is fewer segments, stable caches and plenty of file system cache. Merge read-only indices down, avoid needless refreshes, keep shard sizes reasonable and leave memory for the operating system. Avoid changing the index constantly if freshness is not required.
What to monitor
| Signal | Where to see it | What it suggests |
|---|---|---|
| Segment count per shard | Segments and cat segments APIs | Very high counts point to refresh or merge pressure |
| Deleted document share | Cat indices and segment listings | High share means update-heavy workload or lagging merges |
| Current merges and merge time | Index and node stats | Long-running merges can indicate slow disks |
| Throttled merge time | Node stats | Merges are being slowed to protect the disk |
| Refresh time and count | Index stats | Frequent expensive refreshes reduce indexing capacity |
| Open file descriptors | Node stats | Approaching the limit warns of segment or shard sprawl |
| Disk free space | Cat allocation and host metrics | Merges need spare room, and low disk triggers protective limits |
| Search latency by shard | Slow logs and search stats | Rising latency with rising segment counts links the two |
Keep a dashboard showing segment counts, deleted-document share and merge time per index. These three numbers together tell the story of most segment-related problems long before users complain.
23Design Patterns and Anti-Patterns
Habits that work with the segment design, and habits that fight it.
Patterns worth copying
Time-based indices with rollover
Write to one active index, then roll over. Old indices become read-only, so their segments settle and can be merged down.
Merge after write ends
Force merge only when an index is truly finished receiving writes, ideally via an automated lifecycle policy.
Bulk load then refresh once
Turn scheduled refresh off for large loads, then refresh manually to publish a few large segments.
Separate hot changing data
Keep rapidly changing small fields in a dedicated index so the huge stable data does not generate deleted-document bloat.
Right-sized shards
Choose shard sizes that avoid both tiny shards with many small segments and giant shards that are slow to move and recover.
Leave memory to the operating system
Keep the heap modest so segment files can stay in the file system cache.
Anti-patterns to avoid
Problem
A nightly job force merges the index that is currently receiving writes down to one segment, in the hope of keeping search fast.
Why It’s Harmful
The giant segment is left alone by natural merges until most of its documents are deleted. New writes create small segments beside it, and deleted documents pile up inside it, wasting disk. The heavy nightly job repeats the cost without a lasting benefit.
Correct Approach
Use rollover so that write indices stay small, and force merge indices only after they become read-only.
Problem
Every indexing request asks for an immediate refresh so that clients can search the data at once.
Why It’s Harmful
Each request creates its own tiny segment. Thousands of tiny segments overload merging, slow both indexing and search, and inflate file handle use.
Correct Approach
Use bulk requests and normal scheduled refresh. Where a client must see its own write, use the wait-for option rather than forcing refreshes.
Problem
A counter field, such as a view count or stock level, lives inside a large document and is updated many times per minute.
Why It’s Harmful
Each increment rewrites the whole large document into a new segment and marks the old copy deleted. Deleted documents and merge work explode compared with the tiny change you intended.
Correct Approach
Store fast-changing counters in a separate small document or index, or batch updates and write them less frequently.
Problem
An index is split into hundreds of tiny shards “for future growth”, each holding a few megabytes of data.
Why It’s Harmful
Every shard has its own segments, files, caches and overhead. The cluster is full of tiny segments spread across tiny shards, and searches must contact all of them.
Correct Approach
Size shards for the data you actually have, and use rollover or the shrink feature to keep the count sensible.
24Advantages and Trade-offs of the Segment Design
An honest balance sheet for immutable segments.
Advantages
- Readers never wait for writers, so heavy search and heavy indexing coexist.
- Sealed files compress well and stay valid in the operating system cache.
- Per-segment caches remain correct until the segment is retired.
- Crash recovery is simple, since a segment is either committed or ignored.
- Snapshots are incremental because finished files never change.
- Merges keep the structure healthy in the background without stopping the service.
Disadvantages / Trade-offs
- New data is searchable only after a refresh, so search is near-real-time.
- Updates rewrite the whole document, which is expensive for large documents.
- Deleted documents keep consuming space until merged.
- Merging uses disk bandwidth, processor time and temporary extra disk space.
- Too many or too few segments both hurt, so behaviour needs watching.
- Physical removal of data for compliance needs deliberate planning.
25Best Practices and Common Mistakes
A checklist worth keeping next to your cluster dashboard.
Best practices
Understand your workload’s shape
Decide whether your data is append-only, update-heavy or a mix. This tells you how much to worry about deleted documents and merge load.
Batch your writes
Use bulk requests and a sensible refresh interval instead of forcing refreshes.
Use lifecycle management
Automate rollover, merging of finished indices and eventual deletion so you do not depend on manual timing.
Watch deleted documents and merge time
Add both to your standard dashboards and set gentle alerts.
Keep free disk space
Merges temporarily need extra room. Never run disks close to full.
Protect the files
Encrypt disks, restrict permissions and manage snapshot retention deliberately.
Common mistakes
Expecting instant visibility
Assuming that a document is searchable the moment indexing returns. It becomes visible after the next refresh unless you ask to wait for one.
Believing deletes free space at once
Deleting a large amount of data and being surprised that disk use does not drop. Space returns as merges rewrite the affected segments.
Editing segment files by hand
Deleting or altering files in a shard folder can corrupt the shard. Only Elasticsearch should touch them.
Disabling merges for speed
This trades a short-term gain for a severe search and file handle problem later.
Forgetting that updates are rewrites
Treating a partial update as a cheap in-place change, when it actually indexes a whole new copy of the document.
26Troubleshooting Segment Problems
Match the symptom to the likely cause, and check the simplest thing first.
| Symptom | Likely cause | What to check |
|---|---|---|
| Searches slow down as an index grows busy | Too many small segments, or too few merges keeping up | Segment count per shard, refresh interval, merge activity |
| Disk use far higher than expected | Many deleted documents, or extra space held by unfinished merges | Deleted document counts and recent merges |
| Disk does not shrink after deleting data | Deleted documents not yet merged away | Wait for merges, or expunge on read-only indices |
| Indexing throughput drops over time | Merges falling behind or disks saturated | Merge stats, throttled time and disk performance |
| New documents not visible in search | Refresh disabled or a long refresh interval | Index refresh interval setting and idle shard behaviour |
| Too many open files errors | Sprawl of shards and segments, low operating system limit | Open descriptor stats and system limits |
| Force merge made an active index worse | A giant segment beside many small ones | Segment sizes and the index’s write status |
| Primary and replica show different segment lists | Normal, since each copy builds its own segments | Compare document counts, not segment layouts |
A calm routine
First, list the segments of the affected shards and look at counts, sizes and deleted documents. Second, compare with the refresh and merge statistics to see whether the shard is creating segments faster than it merges them. Third, check the disk for free space and slowness. Fourth, review whether some client is forcing refreshes or force merges. Only after these steps should you consider changing merge settings.
Reproduce the pattern on a test index with a small copy of your data, and watch how segments change as you index, update and refresh. Seeing the sequence with your own eyes makes real incidents much easier to reason about.
27Frequently Asked Questions
Short answers to doubts that come up again and again.
No. A shard is a slice of an index held on a node. A segment is a sealed mini-index inside a shard. One shard holds many segments.
Segments are immutable by design. This makes reading safe without locks, allows heavy compression and stable caches, and makes crash recovery simple. Changes are expressed as new segments plus deletion markers.
It becomes searchable when a refresh writes it into a new segment, which by default happens about once a second. Use the wait-for option if a client needs to see its own write.
There is no single number. An active shard typically holds a natural spread of sizes, from a few large segments to some tiny fresh ones. A finished read-only shard is best merged down to very few.
No. Deleted documents are marked in the segment and physically removed only when a merge rewrites the segment.
An update is delete plus index. The old version is marked deleted and a complete new version is written. Partial updates only save you from sending the full document over the network.
Not if the index is still receiving writes. Force merge is best for indices that are finished, such as older time-based indices, and it works well when driven by a lifecycle policy.
Refresh makes new documents searchable by creating a segment. Flush makes segments durable by forcing them to disk, writing a commit point and clearing the translog of operations that are now safe.
Each copy builds its own segments from the same stream of operations, on its own refresh and merge timing. They hold the same documents, but not identical files.
Mostly no. Segment files are read through the operating system’s file system cache, often memory mapped. Leave plenty of machine memory outside the heap for that cache.
Yes. Use the segments API or the cat segments API to list segment names, sizes, document counts, deleted counts and flags.
No. Existing snapshots keep the segment files they captured. Data disappears from a snapshot only when that snapshot is deleted or expires, so plan retention with your privacy promises in mind.
28Summary and Key Takeaways
A Lucene segment is a sealed mini-index inside an Elasticsearch shard. It bundles an inverted index, stored fields, doc values, norms, points and a record of deleted documents, and once written it is never edited. Refresh turns buffered documents into new segments, flush and the translog make them durable, deletes are markers, updates are rewrites, and merges quietly combine small segments into larger ones. Nearly every performance and disk behaviour you meet, from near-real-time search to deleted data lingering on disk, follows from this one design. Work with it by batching writes, using lifecycle management, merging finished indices, keeping memory for the file system cache and watching segment health.
Key Takeaways
- Segments are sealed mini-indices — every shard is a collection of immutable segments, each searchable on its own.
- Immutability is the source of speed — it removes locking, enables compression, keeps caches valid and simplifies crash recovery and snapshots.
- Refresh makes data searchable — a new segment appears roughly every second by default, which is why search is near-real-time.
- Translog and flush provide durability — searchability and safety are separate steps, joined by the translog.
- Deletes and updates are bookkeeping — deleted documents stay until a merge, and an update rewrites the whole document.
- Merging is essential — it keeps segment counts healthy and reclaims space, and force merge belongs only on finished, read-only indices.
- Plan for disk, memory and compliance — keep free space for merges, leave memory for the file cache, and remember that deleted data survives in segments and snapshots until they are rewritten or expired.