The Codec in Elasticsearch
Understand the hidden set of packing rules that decides how every segment is written to disk and read back, how the index.codec setting trades disk space against speed, and how to use it safely across the life of your data.
Imagine you run a warehouse that stores thousands of boxes for customers. You can pack each box in two ways. You can toss the items in loosely, which is very fast, but the box is big and bulky. Or you can vacuum-seal the items, which takes longer, but the box becomes much smaller and cheaper to store. Whichever method you use, you write it on the label of the box, so that whoever opens it later knows how to unpack it correctly.
Elasticsearch stores its data in the same way. Every sealed piece of an index, called a segment, is written using a set of packing rules, and the name of those rules is stamped on the segment. Those rules are called a codec. The word is short for “coder and decoder”, the same idea as the codec that squeezes a video for streaming. In this tutorial you will learn what the codec decides, how to choose it, how to change it without breaking anything, and what it costs you.
1The Problem: Storing a Lot of Data Without Wasting Disk
Search engines are hungry for disk space, and someone has to decide how bytes are laid out.
Elasticsearch is a search and analytics engine. Companies use it to search products, read logs, investigate security events and study metrics. To answer questions in milliseconds, it does not simply save your documents in a file. It builds many helper structures next to them, such as the inverted index, column-style value stores and small statistics tables. All of these must be stored on disk, and together they can be larger than the original data.
For a small project, disk space is not a worry. For a large one, it becomes one of the biggest costs. A company that ingests a terabyte of logs each day needs to keep weeks or months of them. If storing the same data takes twenty percent less space, that saving is repeated across every node, every replica and every backup. Yet squeezing data harder costs extra processor time and can slow down some operations. So there is always a balance between speed and size.
Think of moving to a new house. You can throw clothes into suitcases quickly, or you can use vacuum bags that shrink them to a third of the size. Vacuum bags need time and effort at packing and unpacking. If you will not open the suitcases for months, the vacuum bags are a good deal. If you need your clothes every day, the quick suitcases may be better.
Someone must define the packing rules
Whenever a segment is written, many small decisions are made. How should the list of terms be arranged? How should document numbers be squeezed? Should the original documents be compressed in blocks, and with which method? How should numbers used for sorting be laid out? These decisions cannot be improvised. They must follow a fixed, documented format, so that the data written today can still be read correctly next year. The component that holds these decisions is the codec.
2What Is a Codec?
A plain definition first, then the vocabulary that goes with it.
A codec in Elasticsearch, inherited from Lucene, is the complete set of rules that defines how every part of a segment is encoded when written to disk and decoded when read back. It is the translator between the in-memory ideas of a search engine, such as terms, documents and numbers, and the actual bytes stored in files.
A codec is not a single algorithm. It is a bundle of separate formats, one for each kind of data inside a segment. One format handles the term dictionary and lists of matching documents. Another handles the stored original documents. Another handles column-style values used for sorting. The codec is the box that holds all of these formats together and gives them one name.
Words you need to know
Codec
The named bundle of encoding rules used to write and read a segment. Its name is stored inside the segment.
Format
One rule set for one kind of data, such as the stored-fields format or the doc-values format. A codec contains many formats.
Encoding
Turning meaningful data into bytes in a specific layout. Decoding is the reverse.
Compression
Encoding data so that it takes fewer bytes. It is one of the things a format may do.
LZ4
A very fast compression method that gives a modest size reduction. It is the default choice for stored documents.
DEFLATE
A slower compression method, similar to what zip files use, that squeezes data smaller. It powers the best compression option.
Segment
A sealed, never-edited mini-index inside a shard. Every segment was written by exactly one codec.
Stored fields
The saved copy of each original document, including its source. This is the part most affected by the codec setting you can change.
Codec here has nothing to do with audio or video codecs, apart from sharing the same idea. It also is not the same as the compression used when nodes talk to each other over the network. We will separate these ideas clearly later.
3Where Codecs Sit: Elasticsearch, Lucene, Segment, Codec
Codecs live at the lowest layer of the storage stack, right next to the files.
Elasticsearch stores an index as shards, and each shard is a Lucene index. A Lucene index is made of segments. When a segment is created, for example at a refresh or a merge, Lucene asks the active codec to write each part of that segment. When a search later needs data from the segment, Lucene asks the codec to read it back. So the codec sits between the search logic above and the files below.
graph TD
A["Elasticsearch index"] --> B["Shard"]
B --> C["Lucene index"]
C --> D["Segment"]
D --> E["Codec chosen when the segment was written"]
E --> F1["Postings format"]
E --> F2["Stored fields format"]
E --> F3["Doc values format"]
E --> F4["Other formats"]
F1 --> G["Files on disk"]
F2 --> G
F3 --> G
F4 --> G
| Layer | Question it answers | Everyday comparison |
|---|---|---|
| Elasticsearch index and mapping | What data do we store and how do we describe it? | The plan of what goes into each box |
| Shard and segment | Which sealed piece holds which documents? | Which box is on which shelf |
| Codec and formats | How are the bytes inside the box arranged? | How the items are packed inside the box |
| Directory and files | Where are the files stored and how are they opened? | The shelf and the warehouse floor |
What the codec does not decide
The codec does not decide which fields you have or how text is analysed. Those choices belong to your mappings and analyzers. The codec also does not decide when segments are created or merged. It only decides how the contents are packed once that work is happening. It is a purely technical layer, which is why most users never notice it, until they need to save disk space.
Many beginners think the codec is an Elasticsearch feature added on top of storage. It is actually a core Lucene concept. Elasticsearch chooses which Lucene codec to use, and lets you switch between a small set of supported choices through one index setting.
4The Formats Inside a Codec
A codec is a team of specialists, and each specialist handles one kind of data.
Inside a segment there are several different structures, each with its own needs. The term dictionary must be searched quickly by text. Numbers must be scanned rapidly for sorting. Stored documents must be returned whole. It would be inefficient to use one encoding for everything, so the codec assigns a separate format to each.
Postings format
Writes the term dictionary and the lists of documents that contain each term. It uses compact number packing to keep long lists small and fast to skip through.
Stored fields format
Writes the original field values of each document, including the source, usually compressed in blocks. This is the format the index codec setting influences.
Doc values format
Writes column-style data used for sorting, aggregations and scripts, packing repeated or small values tightly.
Norms format
Writes small numbers describing field lengths, used to score how well a document matches.
Points format
Writes tree structures for numbers, dates and geographic points, enabling fast range and distance queries.
Term vectors format
Writes optional per-document term details for features like highlighting when enabled.
Vector formats
Write graph-like structures used for approximate similarity search over dense vector fields.
Live docs, field info and segment info formats
Write the bookkeeping records: which documents are deleted, which fields exist, and facts about the segment itself.
A moving company has separate teams. One wraps fragile dishes, another folds clothes, another disassembles furniture. Each team has its own packing method suited to its items. The moving company as a whole, with all its teams and their methods, is the codec. Each team is a format.
Why the stored-fields format matters most to you
Most of the formats are carefully tuned by Lucene’s developers and are not exposed as choices in Elasticsearch. The stored-fields format is different, because it holds the original documents, and those are often the largest part of an index. It is also the one where a clear speed versus size choice exists. That is why the main user-facing setting, index.codec, is mostly about how stored fields are compressed.
When people say “change the codec”, they usually mean “change how stored documents are compressed”. The rest of the codec quietly stays on its well-tested defaults.
5How a Segment Remembers Its Codec
The label on the box is the reason old data stays readable.
When a segment is written, its codec name is recorded in the small segment information file. Each data file also carries a short header at its start and a footer at its end. The header identifies the kind of file, the codec name and the format version. The footer holds a checksum, which is a number computed from the file contents that lets Lucene detect corruption.
sequenceDiagram
participant W as Writer at refresh or merge
participant C as Active Codec
participant F as Segment Files
participant R as Reader at search time
W->>C: Write segment data
C->>F: Encode each part with its format
C->>F: Record codec name in segment info
C->>F: Add header and checksum footer to files
R->>F: Open segment and read codec name
R->>C: Find the codec with that name
C->>R: Decode data using the matching formats
Looking up the codec by name
When Lucene opens a segment, it reads the recorded codec name and asks its registry of installed codecs for a match. This registry lookup uses a standard Java mechanism for discovering components by name. As long as the software still knows a codec of that name, it can read the segment. That is how one shard can hold segments written by different codecs at the same time, each read using the correct rules.
Each sealed box has a label saying “vacuum-sealed, method B, version 3”. The receiving clerk looks in the manual of known methods, finds method B, and follows its unpacking steps. If the clerk’s manual has no such method, the box cannot be opened safely.
Checksums protect against silent damage
Because each file ends with a checksum, Elasticsearch can verify files when it recovers a shard, copies data to another node, or takes a snapshot. A file whose checksum no longer matches is treated as corrupt, and the system can repair it from another copy. This is a quiet, valuable benefit of having a strict codec format.
Never edit, rename or replace segment files by hand. The headers and checksums are part of the codec’s format. A file modified outside Elasticsearch will fail verification and can make the shard unusable.
6The index.codec Setting
One setting, two well-known choices, and a very clear trade-off.
Elasticsearch exposes the codec choice through the index-level setting index.codec. In its documented form it has two main values. The value default compresses stored data with LZ4, which is quick. The value best_compression uses DEFLATE for stored data, which produces smaller files but is slower for the parts that read stored documents.
| Aspect | default | best_compression |
|---|---|---|
| Stored fields compression | LZ4, light and fast | DEFLATE, stronger and slower |
| Disk space used by stored data | Larger | Smaller, often noticeably so |
| Speed of fetching documents in search results | Faster | Slower, because blocks must be decompressed more heavily |
| Indexing and merging processor cost | Lower | Higher, since more compression work is done |
| Searches that return no documents, such as counts and aggregations | Not affected much | Not affected much |
| Typical use | Hot data and interactive searching | Older, rarely fetched data and cost saving |
It is a static setting
The codec setting is static, which means it can be given when an index is created, or changed only while the index is closed. You cannot change it on an open index. This makes sense, because switching the encoding rules while writes are happening would be risky.
Expecting the codec change to shrink your existing data instantly. The new codec applies to segments written after the change. Old segments remain in their original encoding until they are merged. We cover this carefully in a later chapter.
Newer Elasticsearch releases may add further codec options or change details of the existing ones. Always check the index modules page of the documentation for your exact version before planning a large change.
7How Stored-Field Compression Works
The difference between the two options lies in how documents are grouped and squeezed.
Stored fields are not compressed one document at a time. Doing so would waste effort, since a single small document has little repetition to exploit. Instead, the format collects many consecutive documents into a block, compresses the whole block at once, and writes it out. Repeated names, repeated field labels and similar values across neighbouring documents all help the compressor find patterns.
flowchart TD
A["Documents arrive at the segment writer"] --> B["Collect documents into a block"]
B --> C{"Block full or size limit reached?"}
C -->|No| B
C -->|Yes| D["Compress the whole block"]
D --> E["Write compressed block to the stored fields file"]
E --> F["Record block position in a small index"]
F --> G["Search later reads the index, then decompresses only the needed block"]
Why blocks affect speed
To return one document, the reader must find its block, decompress the entire block, and then pick out the wanted document. If the block is large and the compression method is heavy, this takes more time. Stronger compression gives smaller blocks on disk, but each read costs more processor effort. A fast method like LZ4 decompresses very quickly. A stronger method like DEFLATE decompresses more slowly, but stores more data in the same space.
Suppose a photo album is stored as zip files, each holding fifty photos. To see one photo, you unzip the whole file of fifty. A light zip opens instantly. A tightly compressed zip is smaller on the shelf, but takes more time to open. If you seldom look at the photos, the tight zip is a fine deal.
What kinds of data compress best
Data with a lot of repetition compresses very well. Logs are a perfect example, since each line repeats field names and often repeats values such as host names, status codes and message templates. Data made of random numbers or already compressed content, such as encoded images, compresses poorly, and the choice of codec will make little difference to size.
| Kind of data | How well it compresses | Expected effect of best compression |
|---|---|---|
| Application logs and events | Very well, lots of repeated text | Clear saving on stored documents |
| Product catalogues with long descriptions | Well | Good saving |
| Metrics with mostly numbers | Depends, numbers often live in doc values | Smaller effect on total index size |
| Documents with embedded encoded blobs | Poorly | Little saving |
| Very small documents | Moderately, blocks help | Modest saving |
The only reliable way to know how much you will save is to test with your own data. Load a representative sample into two test indices, one per codec, and compare their sizes. Guesses from other companies’ numbers are often wrong for your data.
8Compression Trade-offs: Disk, CPU and Speed
Every choice gives something and takes something, so let us weigh them honestly.
Choosing best compression is not a free upgrade. You pay with processor time and with slower document retrieval, and you gain disk space. Whether that is a good deal depends on how your index is used. To decide, think about three activities: writing data, fetching documents in search results, and running searches that never return stored documents.
Writing and merging
Stronger compression means more processor work whenever new segments are written, which happens at every refresh and every merge. Heavy indexing workloads feel this cost.
Fetching documents
Returning the source of matching documents requires decompressing blocks. Searches that return many documents per request feel the difference most.
Counting and aggregating
Searches that only count matches, or compute aggregations from column data, mostly avoid stored fields and see little change.
Storing and copying
Smaller files mean less disk to buy, less data to copy during recovery, faster snapshots and less network traffic when shards move.
| Your situation | Better choice | Reason |
|---|---|---|
| Hot log index with constant heavy ingestion | default | Save processor time for indexing and merging |
| Interactive product search returning many documents | default | Keep document fetching fast for users |
| Week-old logs searched a few times a day | best_compression | Rare fetches, high storage cost, so saving disk wins |
| Archive kept for compliance, rarely read | best_compression | Storage cost dominates and speed hardly matters |
| Dashboards using only aggregations | Either, small difference | Stored fields are rarely touched |
| Very tight processor budget on an indexing tier | default | Avoid extra compression work |
Vacuum-sealing clothes is worth it for winter coats stored in the attic. It is not worth it for the shirts you wear every week. Old, rarely used data is the attic, and fresh, heavily used data is the wardrobe.
The size of the benefit and the size of the slow-down both depend on your document shapes, your query mix and your hardware. Measure with realistic data and realistic queries before you change many indices.
9Changing the Codec on an Existing Index
The switch is easy, but the effect arrives later, and only if you do the last step.
You may decide that yesterday’s index should use best compression. Because the setting is static, the index must be closed while you change it. After you reopen it, new segments will use the new codec. But the segments that already exist are sealed and immutable, so they keep their original encoding. Only a merge that rewrites them will recompress them.
Confirm the index no longer needs heavy writing
Best compression is most sensible for indices that are finished or nearly finished receiving data, such as yesterday’s log index.
Check free disk space
Rewriting segments temporarily needs extra space, because old segments stay until the new ones are complete.
Close the index
A closed index cannot be searched or written to, so plan a maintenance window if users depend on it.
Update the codec setting
Set index.codec to the new value while the index is closed.
Reopen the index
After it reopens, new segments will be written with the new codec. Existing ones are untouched.
Force merge to rewrite old segments
Run a force merge so that the old segments are read and rewritten as new segments using the new codec. This is the step that actually shrinks the stored data.
Verify the result
Compare disk size before and after, and check the segment details to confirm that the stored-field mode changed.
Index lifecycle management can perform this sequence for you. Its force merge action accepts an option to apply the best compression codec, and it handles closing, updating and reopening around the merge. Check the documentation for your version for the exact behaviour and requirements.
Do not run a force merge on an index that is still receiving writes. It produces a very large segment that natural merging will neglect. Do this step only for indices that are read-only or finished.
10Codecs and Merging: Mixed Encodings in One Shard
A shard can hold segments written under different rules, and merges quietly unify them.
Because each segment records its own codec, a single shard can contain a mixture. Suppose you changed an index from default to best compression yesterday. Some segments were written before the change, and some after. Both kinds sit side by side in the same shard and are read correctly, each with its own rules.
graph LR
A["Old segment written with default"] --> M["Merge"]
B["Old segment written with default"] --> M
C["New segment written with best compression"] --> M
M --> N["New merged segment written with the current codec setting"]
Merging is the great translator
A merge reads all live documents from its input segments, no matter which codec wrote them, and writes them into a new segment using whichever codec is currently configured. So every merge gradually converts the shard towards the current setting. This is why a codec change slowly takes effect on active indices even without a force merge, but only for the segments that happen to be merged.
Why large old segments linger
Natural merging tends to leave very large segments alone for a long time. A big segment written under the old codec may therefore keep its old encoding for weeks. If you want prompt conversion, use a force merge on a finished index. If you are happy to wait, natural merges will do the work over time.
A warehouse has old loosely packed boxes and new vacuum-sealed boxes side by side. Whenever the staff repack several boxes into one, they use the new vacuum method. Boxes that are never repacked stay loose until someone deliberately decides to redo them.
Mixed encodings are normal and harmless. They cost nothing in correctness. They only mean that the space saving from a codec change is partial until merges or a force merge have rewritten the old segments.
11Per-Field Formats and How Mappings Influence Encoding
The codec can behave differently for different fields, guided by your mappings.
Some formats, such as the postings format, the doc values format and the vector format, can be chosen field by field. Elasticsearch wraps the underlying Lucene codec in a layer that looks at each field’s mapping and picks a suitable format for that field. This means the codec you set for the index is a starting point, and the mapping refines it.
| Mapping choice | What it changes in the segment | Effect on space and speed |
|---|---|---|
| Field type such as keyword, text, number or date | Which structures and formats are written for the field | Different types have very different sizes |
| Index option turned off for a field | No inverted index written for it | Saves space, but the field cannot be searched |
| Doc values turned off | No column data written for the field | Saves space, but sorting and aggregating on it are lost |
| Norms turned off | No length data written for the field | Saves space, but relevance scoring on the field changes |
| Dense vector index options | Selects the vector structure and possibly quantised storage | Large differences in memory and disk use |
| Source stored or trimmed | Changes what the stored fields format must hold | Directly changes the size of the biggest part |
Mappings are your most powerful space tool
The compression choice usually changes stored data by a fraction. A careful mapping can change the total index size by far more. Not indexing fields you never search, not keeping doc values for fields you never sort or aggregate on, and choosing the right field types often save more space than any codec switch. The codec is the last polish. The mapping is the main design.
Before vacuum-sealing your belongings, the best space saving is to throw away things you do not need. Mapping decisions decide what is stored at all. The codec then decides how tightly what remains is packed.
Trying to fix a bloated index by switching codecs, when the real problem is that every field is indexed and stored in several ways although only a few are ever queried. Review the mapping first.
12Codecs, Lucene Versions and Upgrades
Each new release may bring new formats, and old ones must still be readable.
Lucene’s developers continually improve the formats. A new release may pack postings more tightly, speed up doc values or add new vector structures. When that happens, they introduce a new codec version with a new name, rather than changing the old one. New segments use the new codec. Existing segments keep the codec that wrote them, and Lucene keeps the ability to read older codecs for a limited number of releases.
A release ships with a current codec
Every Elasticsearch version uses a specific Lucene version whose default codec has a versioned name.
You upgrade the cluster
The nodes now write new segments using the newer codec. Old segments remain, still readable through older, kept-for-compatibility codecs.
Merges convert gradually
As segments merge, data is rewritten with the new codec, so the old encodings fade out over time.
Compatibility has a limit
After a few major versions, the reading support is removed. Very old indices must be rebuilt, restored differently or treated as read-only archives, depending on the version rules.
What this means for upgrades
Elasticsearch generally supports reading indices created in the previous major version, but not those from further back. This is largely a consequence of Lucene’s codec compatibility promises. Before a major upgrade, check the upgrade guide and use the upgrade assistant tools to find indices that would be too old. Reindexing such data into a fresh index writes it with the current codec and solves the issue.
Why the segment version differs between segments
If you list segments after an upgrade, you may see different Lucene versions in the version column. That is your visible sign of mixed codecs in the same shard. It is expected, and it slowly disappears as merges rewrite old segments.
Before any major upgrade, record the segment versions of your biggest indices. Afterwards you can confirm that new data is being written by the new codec, and you can plan when to rewrite the oldest indices.
Lucene allows developers to write their own codecs, but Elasticsearch only supports a small set of choices through its settings. Trying to load a homemade codec would tie your data to code that nobody else maintains, and would make upgrades and recovery risky. Treat custom codecs as an advanced research topic, not a production technique.
13Codecs and Index Lifecycle Management
The best place to use best compression is when data grows old and cool.
Data has a life. Fresh data is written constantly and searched often. As time passes, it is searched less and finally almost never. Index lifecycle management, usually shortened to ILM, moves indices through phases such as hot, warm, cold and delete. Each phase is a natural place to switch to a cheaper storage configuration.
flowchart LR
H["Hot phase: heavy writes and frequent search, default codec"] --> W["Warm phase: read-only, force merge with best compression"]
W --> C["Cold phase: rarely searched, cheaper storage"]
C --> D["Delete phase: index removed at end of retention"]
| Phase | Data behaviour | Codec approach |
|---|---|---|
| Hot | Constant writes, frequent reads of fresh data | Keep the default codec for speed |
| Warm | No more writes, moderate reads | Force merge, optionally with best compression, and possibly fewer replicas |
| Cold | Rare reads | Best compression already in place, cheaper hardware or snapshot-backed storage |
| Delete | Retention ended | Index is removed, and disk space returns |
Why the warm phase is ideal
At the warm phase the index is finished receiving data. That makes a force merge safe. The merge is the perfect moment to recompress stored fields with the stronger method. Fewer, better-compressed segments then serve the many weeks of occasional searches that follow, at a fraction of the storage cost of the hot copy.
Let ILM run the sequence rather than doing it by hand. A policy is repeatable, reviewable and applies the same rules to every new index, which removes the risk of forgetting a step.
If your users regularly fetch large numbers of documents from old data, for example exporting large result sets, slower decompression will be noticeable there. Test your real cold-data queries before applying best compression to everything.
14What Codecs Do Not Do
Several other settings involve compression or encoding, and they are easy to mix up.
Beginners often see the word “compression” in several places of Elasticsearch documentation and assume they are all the same. They are not. Only one of them is the index codec. Separating them will save you from confusing experiments.
| Setting or feature | What it compresses | Is it the index codec? |
|---|---|---|
| Index codec setting | Data inside segments on disk, mainly stored documents | Yes |
| HTTP compression | Requests and responses between clients and Elasticsearch | No |
| Transport compression | Messages between nodes in the cluster | No |
| Recovery compression | Data sent while a shard copy is rebuilt on another node | No |
| Snapshot repository compression | Metadata files written into the repository | No |
| Source filtering and source modes | Which parts of the source are stored at all | No, but they interact with size |
The codec is not encryption
Encoding and compression make data smaller or faster to read. They do not hide it. Anyone with the files and matching Lucene software can decode them. If confidentiality matters, use disk or volume encryption, and access controls, as separate layers.
The codec does not change search results
Switching between the default and best compression changes only how data is stored and how fast it is read. It should not change which documents match a query or how they score. If results differ after a codec change, the cause lies elsewhere, such as a different set of documents or a mapping change.
Vacuum-sealing clothes changes the size of the bag, not the clothes. When you unpack them, they are the same shirts. Compression at the segment level works the same way: the contents are identical, only their packing changes.
When a colleague says “we enabled compression”, ask which one. The answer might be network compression, index codec compression or snapshot settings, and each has different effects and different places to check.
15Measuring the Effect: A Fair Test
Never adopt a codec change on hope. Prove it with a small, honest experiment.
A good test compares two identical indices that differ only in codec, filled with the same realistic data and queried with the same realistic requests. If anything else differs, you will not know which change caused which effect. The steps below give you a reliable method that any team can follow.
Pick a representative sample
Take a slice of real data, large enough to show real behaviour, such as one full day of logs or ten percent of a catalogue.
Create two identical indices
Use the same mappings, shard count and replica count. Set the codec to default on one and best compression on the other.
Load the same data into both
Record indexing time and processor use for each, since compression work appears here.
Force merge both the same way
This removes differences caused by uneven segment layouts and gives a fair size comparison.
Compare sizes
Look at total store size, and use the disk usage analysis feature to see the stored-field share in each.
Replay real queries
Run the same mix of searches against each index, including ones that return many documents and ones that only aggregate, and compare latencies.
Decide with numbers
Weigh the disk saving against the change in indexing and fetch time, in terms of your real costs and your users’ patience.
Tools you can use
Index stats and cat indices
Show the store size of each index, the simplest way to compare disk use.
Segments API
Lists segments with sizes, Lucene versions and attributes such as the stored-fields compression mode.
Disk usage analysis
Breaks down space by field and by structure, showing exactly where your bytes are.
Benchmarking tools
Elastic’s benchmarking tool can replay workloads and report repeatable timings.
Comparing sizes before merging. Freshly written indices contain many uneven segments, and deleted documents may distort the numbers. Force merge both test indices identically before comparing.
16Real-World Examples
The same setting helps very different businesses, in very different ways.
Log analytics with a long retention period
A company must keep ninety days of application logs. The first two days sit on fast hot nodes with the default codec. After that, a lifecycle policy moves indices to warm nodes, force merges them with best compression and reduces replicas. The stored documents of older days take clearly less disk, letting the team keep the full ninety days on fewer machines.
Compliance archive for a bank
A bank retains transaction event records for years. These are searched rarely, mostly during audits. The team applies best compression and merges to a small number of segments, then snapshots the indices. Storage costs drop, and the occasional slower fetch is acceptable during audits.
E-commerce product search
An online shop shows search result pages with many documents per request and needs consistently quick responses. The team keeps the default codec on the live catalogue index, because fetch speed for many documents matters more than saving some disk space.
Security operations
A security team investigates alerts on recent events with heavy interactive searching, then rarely revisits older events. They keep hot data with the default codec and use best compression for indices older than a week, balancing analyst speed against the storage bill of a large event volume.
Metrics and monitoring dashboards
A monitoring system stores numeric measurements. Most queries are aggregations that read column data, not stored documents. The team finds that switching the codec barely changes query speed, but it still helps a little on disk, so they apply best compression to older indices as a low-risk saving.
A migration after a major upgrade
A team upgrades and notices a mix of Lucene versions in their segment lists. They confirm that new data uses the new codec, and schedule force merges of finished older indices so that they are rewritten in the current format, reducing the risk when a later upgrade drops support for the oldest encodings.
17Other Ways to Shrink an Index
The codec is one lever among many, and often not the strongest one.
If your goal is to reduce disk use, compare the codec setting with the other options below. Often a combination gives the best result, and several of them save more than a codec change does.
Leaner mappings
Index only fields you search. Keep doc values only for fields you sort or aggregate on. Turn off norms where scoring does not matter.
Right field types
Use compact types for numbers and exact values, and avoid mapping everything as free text.
Source handling
Trimming the stored source, or using special source modes in versions that support them, reduces the biggest stored part. Check limitations first, since some features need the full source.
Index sorting
Sorting documents within segments can make similar documents neighbours, improving how well they compress. It has costs at indexing time, so test it.
Merging to fewer segments
Fewer segments carry less per-segment overhead and drop deleted documents, on finished indices only.
Fewer replicas on older data
Each replica is a full copy. Reducing replicas on cold data cuts storage directly, at some cost to resilience.
Right-sized shards
Avoid many tiny shards, which multiply per-shard and per-segment overhead.
Storage tiers and snapshot-backed storage
Move old data to cheaper tiers or to storage backed by snapshots, where the cost per byte is lower.
Reducing storage in a house is a mix of decluttering, buying smaller furniture, using vacuum bags and renting a cheaper storage unit for old items. Vacuum bags alone rarely solve a cluttered house. The codec is your vacuum bag.
Start with the mapping, then the lifecycle and replica settings, and use the codec as a finishing step for cold data. This order usually gives the biggest savings for the least risk.
18Security and Compliance Considerations
A codec changes the shape of files, not who can read them.
Encoding is not protection
The codec only defines a layout. The files written by any supported codec can be decoded by anyone who has the files and compatible Lucene software. Do not treat compression as a way to hide data. Protect the data directory with disk or volume encryption, strict file permissions and access controls on the machines, and protect snapshots in their repositories too.
Encryption at rest
Encrypt volumes that hold shard data, so that a stolen disk reveals nothing regardless of codec.
Restricted file access
Allow only the service account and named administrators to read the data directory.
Integrity checks
The checksums in codec files help detect corruption or tampering. Do not disable startup and recovery checks without a strong reason.
Change control
Treat codec changes like other production changes: reviewed, tested and recorded, since they involve closing indices and heavy merges.
Deleting data still needs merges
Changing the codec and force merging rewrites segments, and that drops deleted documents from the new files. This can be helpful when regulations require data to disappear from primary storage. But older snapshots and other copies still contain the earlier files, so retention rules for snapshots must be handled separately.
Closing an index, forcing large merges and reopening are heavy operations. A mistake during them, such as too little disk space, can affect availability. Rehearse the procedure on a copy of an index first, and make sure a recent snapshot exists before starting.
19Monitoring, Logging and Metrics
You should see the effect of a codec choice on your dashboards, not just in a one-off test.
After any codec change, watch a small set of numbers for a few weeks. They tell you whether the saving is real and whether any side effects have appeared. Keep a record of the values before the change, so that the comparison is fair.
| Signal | Where to see it | What it tells you |
|---|---|---|
| Store size per index | Cat indices or index stats | Whether disk use is falling as expected after merges |
| Stored fields size | Disk usage analysis | How much the stored-fields part shrank |
| Segment versions and attributes | Segments API | How much of the shard still uses the old encoding |
| Fetch phase time | Search stats and slow logs | Whether slower document retrieval is noticeable to users |
| Indexing time and merge time | Index and node stats | The extra processor cost of stronger compression |
| CPU use on the affected tier | Host metrics | Whether nodes have headroom for the extra work |
| Free disk space during merges | Cat allocation or host metrics | Whether rewriting has enough temporary room |
| Query latency percentiles | Application and cluster metrics | The final experience of real users |
Useful alerts
Slow fetch on compressed tiers
Warn when fetch-phase latency on cold indices rises well above its normal level.
Low disk during maintenance
Warn when free space drops below what a planned force merge will need.
Lifecycle step failing
Warn when an ILM policy is stuck at a force merge or close step.
Unexpected size growth
Warn when an index grows well beyond forecasts, which may show a merge or mapping problem.
Record a “before” snapshot of key numbers in your runbook when you change any storage setting. Without it, six weeks later nobody can say whether the change really helped.
20A Closer Look at LZ4 and DEFLATE
Understanding the two compression methods explains why the trade-off looks the way it does.
Both LZ4 and DEFLATE belong to a family of methods that find repeated pieces of data and replace them with short references. If the phrase “status=200 method=GET” appears many times, the compressor stores the phrase once and points back to it afterwards. The difference lies in how hard each method searches for repeats and how it encodes the result.
LZ4: fast and light
LZ4 is designed for speed. It looks for repeats in a simple, quick way and writes references in a very compact but simple form. It does not spend effort on a second stage of clever encoding. The result is a modest reduction in size, but decompression is extremely fast, often close to the speed of memory copying. That is why it is a good default for data that is fetched often.
DEFLATE: patient and thorough
DEFLATE adds a second stage. After finding repeats, it also encodes the leftover symbols using shorter codes for common symbols and longer codes for rare ones. This entropy coding squeezes out more redundancy. The result is smaller output, but both compression and decompression need more work. That extra work is what you pay when you fetch documents from a best-compression index.
| Property | LZ4 | DEFLATE |
|---|---|---|
| Main goal | Speed | Smaller size |
| Compression effort | Low | Higher |
| Decompression speed | Very fast | Slower |
| Output size | Larger | Smaller |
| Where Elasticsearch uses it | Default stored fields | Best compression stored fields |
LZ4 is like folding your clothes quickly and stacking them. DEFLATE is like folding carefully, rolling tightly and packing every gap. The careful way fits more in the suitcase, but it takes more time at both ends of the journey.
Compression methods used by Elasticsearch can evolve. Newer releases may introduce other methods with different balances of speed and size. The principle stays the same: find the point on the speed versus size curve that suits the age and use of your data.
21Design Patterns and Anti-Patterns
Habits that give codec choices a good result, and traps that waste effort.
Patterns worth copying
Fast when hot, small when cold
Keep the default codec for active indices and apply best compression when data reaches the warm or cold phase.
Compress during the force merge
Combine the codec change with the one force merge you would run anyway on a finished index, so data is rewritten only once.
Policy, not manual steps
Encode the sequence in an index lifecycle policy so every index follows it consistently.
Mappings first
Reduce indexed and stored data through careful mappings before turning to compression.
Test with a twin index
Compare two identical indices under a realistic workload before rolling out a change widely.
Record before and after
Keep numbers for size and latency in the change record so that value is demonstrated, not assumed.
Anti-patterns to avoid
Problem
The team sets best compression on a busy hot index that constantly receives writes and serves interactive searches with many documents per page.
Why It’s Harmful
Every refresh and merge spends extra processor time compressing harder, and every page of results decompresses heavier blocks. Users notice slower pages and indexing throughput falls, all to save disk on data that is short-lived.
Correct Approach
Keep the default codec on hot indices, and apply best compression only when the index becomes read-only and cool.
Problem
An administrator changes the codec on an existing index and reports the work finished, without any merge afterwards.
Why It’s Harmful
Existing segments keep the old encoding. The disk saving never appears for old data, and the team believes compression is active when it only affects new segments.
Correct Approach
After changing the setting on a finished index, run a force merge and then verify the sizes and the segment attributes.
Problem
The team forces a merge on an index that is still being written to, in order to apply the new codec quickly.
Why It’s Harmful
The merge creates a huge segment that natural merging will ignore, while new writes keep adding small segments. Deleted documents pile up inside the giant segment, and disk use rises.
Correct Approach
Use rollover so write indices stay small, and perform force merges and codec changes only after an index stops receiving writes.
Problem
A bloated index is blamed on the codec, and repeated experiments with codec settings are run while every field is indexed and stored in several ways.
Why It’s Harmful
The codec only changes packing. The real waste lies in what is stored. Effort goes into the smallest lever while the largest one is untouched.
Correct Approach
Analyse disk usage by field, trim mappings and source handling, then consider the codec as a finishing step.
22Advantages and Trade-offs
An honest balance sheet for choosing best compression.
Advantages
- Smaller stored data means lower disk cost across nodes, replicas and snapshots.
- Smaller shards recover and move faster, since less data travels over the network.
- The change is one setting, easy to apply through lifecycle policies.
- Search results and scoring are unaffected, only packing and speed change.
- Mixed encodings inside a shard are safe, so rollout can be gradual.
- The format is versioned and checksummed, which supports safe upgrades and corruption detection.
Disadvantages / Trade-offs
- Fetching documents is slower, especially when many documents are returned.
- Indexing and merging use more processor time.
- The setting is static and needs the index closed to change.
- Savings only appear after segments are rewritten by merges.
- Effect size depends heavily on your data and is hard to predict without tests.
- Force merging to apply it requires spare disk and careful timing.
23Best Practices and Common Mistakes
A checklist to keep beside your lifecycle policy.
Best practices
Know your data’s age profile
Find out how quickly your data cools. The codec choice should follow the point where reads become rare.
Start with the default
Use the default codec unless you have a clear storage reason and a measured result to support a change.
Apply best compression on finished indices
Do it as part of the warm phase, together with the force merge, through an automated policy.
Keep disk headroom
Ensure enough free space for the rewrite, since old and new segments coexist for a while.
Snapshot first
Take a recent snapshot before closing or heavily rewriting an important index.
Verify and document
Check sizes and segment attributes afterwards, and record the outcome for the next person.
Common mistakes
Confusing different compression settings
Enabling network compression and believing disk usage will fall, or the reverse. Each setting has its own scope.
Changing the setting on an open index
The static setting cannot be updated while the index is open. Plan for closing it, or set it at creation using a template.
Applying it everywhere at once
Rolling a change to every index in one step leaves no room to notice problems. Try one index and one tier first.
Ignoring query patterns
Compressing an index that users export in large batches every day may make those exports noticeably slower.
Forgetting templates
Changing existing indices but not the index template means every new index starts with the old codec, so the work repeats every day.
24Troubleshooting Codec Problems
Match the symptom to its likely cause and check the simplest explanation first.
| Symptom | Likely cause | What to check |
|---|---|---|
| The disk size did not change after switching codec | Old segments were not rewritten | Whether a force merge ran, and the segment attributes |
| Setting update is rejected | The index is open and the setting is static | Close the index, apply the setting, then reopen |
| Search results return more slowly after change | Heavier decompression for fetched documents | Fetch-phase time and how many documents each request returns |
| Indexing slowed down after change | More compression work at refresh and merge | Processor use and merge time on the tier |
| New daily indices still use the old codec | The index template was not updated | Template settings and matching patterns |
| Lifecycle policy stuck at the merge step | Not enough disk, or the index is still receiving writes | Lifecycle explain output and free space |
| Segment list shows several Lucene versions | Segments written by different releases or codecs | Normal after upgrades, gradually fades with merges |
| Corruption errors after copying files manually | Codec headers or checksums no longer match | Restore from a snapshot rather than repairing by hand |
A calm routine
First, confirm the current codec setting of the index and of its template. Second, list the segments and look at their versions and attributes to see which encoding they actually use. Third, compare disk size with what you expected after a full force merge. Fourth, check the performance numbers for the affected phase of searching or indexing. Most codec puzzles turn out to be one of three things: an unmerged old segment, a template that was never updated, or a static setting that needed the index closed.
If a shard reports corruption, use Elasticsearch’s recovery from another copy or a snapshot. Editing files, or removing checksum footers, can turn a small problem into permanent data loss.
25Frequently Asked Questions
Short answers to doubts that come up again and again.
A codec is the bundle of rules that defines how every part of a segment is written to disk and read back, and its name is stored inside each segment.
The well-known values are default, which uses LZ4 for stored fields, and best_compression, which uses DEFLATE. Newer versions may add more, so read the documentation for your release.
No. It changes how stored data is packed and how quickly it can be read. The matching and scoring of documents are the same.
Not while it is open. The setting is static, so you set it at creation or close the index, change it and reopen it.
Existing segments keep the codec that wrote them. Only new segments and merged segments use the new one. Run a force merge on a finished index to rewrite the old data.
Yes. Each segment records its own codec and is read accordingly. Merges gradually convert everything to the current setting.
Mostly those that return many documents, since they decompress stored blocks. Searches that only count or aggregate use column data and are far less affected.
It depends on your data. Repetitive data such as logs compresses well, while already compressed or random content does not. Test with a sample of your own documents and compare sizes after a force merge.
No. Those compress data in transit between clients and nodes or between nodes. The index codec controls how data is stored in segment files on disk.
No. Encoding and compression are not security controls. Use disk encryption, permissions and access controls to protect data.
Lucene allows it, but Elasticsearch supports only its documented choices. A custom codec would create serious risks for upgrades and recovery, so it is not a normal production practice.
On finished, read-only indices that are searched rarely, typically in a warm or cold lifecycle phase, applied together with a force merge.
26Summary and Key Takeaways
A codec is the set of rules that turns every part of a Lucene segment into bytes on disk and back again. It is a bundle of formats for postings, stored documents, column values, norms, points and bookkeeping, and its name is stamped on each segment so that old data stays readable. In Elasticsearch you mainly influence it through the static index codec setting, choosing between fast LZ4 for stored documents and stronger DEFLATE for smaller storage. Changes reach old data only when merges rewrite it, so the best practice is to keep the default while data is hot and apply best compression with a force merge when data is finished and cool, guided by measurements from your own workload.
Key Takeaways
- A codec is a bundle of formats — it defines how each structure inside a segment is encoded and decoded, and every segment records the codec that wrote it.
- The main choice is about stored fields — default uses fast LZ4, best_compression uses stronger DEFLATE, trading disk space against fetch and indexing speed.
- The setting is static — set it at creation or while the index is closed, and remember that it does not encrypt or change search results.
- Merges do the conversion — old segments keep their old codec until rewritten, so use a force merge on finished indices to realise the saving.
- Mixed encodings are safe — one shard can hold several codecs at once, and upgrades bring newer codecs while keeping older ones readable for a limited time.
- Mappings come first — trimming indexed and stored data usually saves more than any codec change, so use the codec as a finishing step.
- Automate and measure — let lifecycle policies apply the change to cool data, test with a twin index, and record sizes and latencies before and after.