Doc values are an on-disk structure, built when a document is indexed, that stores the value of a field for every document in a column-like layout.

Elasticsearch stores your data as documents. A document is a bundle of fields, such as a product with a name, a price, a category, and a date. When you index that document, Elasticsearch does not just save it once. It builds several helper structures so that different kinds of questions can be answered quickly. Doc values are one of these helpers.

For each field, doc values keep a simple table: for document number one the value is this, for document number two the value is that, and so on. Because all values of one field are stored close together, the engine can read a whole field for many documents very fast. This is exactly what you need when you want to sort results by price, count documents per category, or calculate an average.

Doc values are switched on by default for almost every field type. Most people use them every day without knowing, because every time Kibana draws a bar chart or you sort search results, doc values are doing the work behind the scenes.

Search and analytics look at the same data from opposite directions. One structure cannot be great at both.

The classic search structure is the inverted index. It works like the index at the back of a textbook. You look up a word and you find the pages where it appears. In Elasticsearch, you look up a term and find the list of documents that contain it. This is wonderful for questions such as “find all documents that mention laptop”.

Now suppose you ask a different question: “For the documents that matched, what is the price of each one?” The inverted index is bad at this. It would have to walk through every term of the field to find which terms belong to each document. This is like reading the whole textbook index just to learn which words are on page 57. That is slow.

flowchart LR
  subgraph INV["Inverted index: term to documents"]
    T1["laptop"] --> D1["docs 1, 4, 9"]
    T2["phone"] --> D2["docs 2, 3, 7"]
  end
  subgraph DV["Doc values: document to value"]
    A1["doc 1"] --> V1["price 800"]
    A2["doc 2"] --> V2["price 500"]
    A3["doc 3"] --> V3["price 450"]
  end
        

Doc values fix this by storing the opposite direction. Given a document number, you can find its value in one step. And because they are laid out by field, reading the value of one field for a million documents means reading one continuous stretch of data, which computers are very good at.

To understand why doc values are fast, it helps to understand the difference between row storage and column storage.

In a row-oriented layout, all the fields of one document are stored together. This is what the original document, called _source, looks like. It is great when you want the whole document back, for example when showing a search result. But it is poor for analysis. To add up all the prices, you would have to open each document, skip past the name, the description, and the date, and pick out the price.

In a column-oriented layout, all values of one field are stored together. To add up all the prices, you read one tight column of numbers and ignore everything else. Data warehouses use this idea for the same reason. Doc values bring the same benefit inside Elasticsearch.

The similarity of values in one column also helps compression. A column of ages contains only small numbers, so it can be squeezed into very few bits per value. A pile of mixed documents cannot be squeezed as tightly.

Elasticsearch and its engine Lucene keep several structures for each field. Doc values are only one member of the family.

Because each structure takes disk space and index time, Elasticsearch lets you decide per field which ones to keep. A field that you only ever search and never sort or aggregate does not need doc values. A field that you only ever aggregate and never search does not need the inverted index. Understanding this table is the key to designing lean mappings.

Whenever Elasticsearch needs the value of a field for many documents at once, it reaches for doc values.

Almost all field types come with doc values switched on. There are a few important exceptions.

The most important exception is the text field. Text is broken into many small terms by an analyzer, such as splitting a sentence into words. A document’s text field therefore does not have one simple value, but many pieces. Sorting or aggregating on those pieces is rarely what anyone wants. So text fields do not build doc values.

Doc values are not made when you search. They are made when you index, which is why they cost disk space and a little indexing time.

When a document arrives, Elasticsearch reads each field, and for every field with doc values enabled, it adds the value to an in-memory buffer for that field. The document also goes into the buffers for the inverted index and the other structures. These buffers are held by the shard, which is one Lucene index.

Periodically, the shard performs a refresh. During a refresh, the buffers are written to disk as a new segment. A segment is a small, complete, read-only mini index. The doc values of the fields for the documents in that segment are written as part of it. Once written, a segment is never changed.

sequenceDiagram
  participant C as Client
  participant S as Shard
  participant B as In-memory Buffers
  participant G as New Segment on Disk
  C->>S: Index a document
  S->>B: Add terms to inverted index buffer
  S->>B: Add value to doc values buffer
  S->>B: Keep original document
  Note over S,B: More documents arrive
  S->>G: Refresh writes buffers as a segment
  G->>C: New documents become searchable
        

Merges rewrite columns

Over time a shard collects many small segments. A background process called a merge combines several small segments into one larger segment and then removes the old ones. During a merge, the doc values columns of the old segments are read and rewritten as one bigger column. This keeps the number of segments low, and it also physically removes values of deleted documents.

Under the hood, Lucene knows a small number of doc values shapes. Every Elasticsearch field maps onto one of them.

You do not have to remember these names to use Elasticsearch. But they explain many behaviors. Numbers of all kinds are turned into whole numbers. A price stored as a decimal is converted into a number pattern that keeps its order. Dates become a count of milliseconds since a starting date. Strings become positions in a sorted list of unique strings, which is the ordinal trick that we explore next.

Strings are big and repeated. Ordinals let doc values store them cheaply and compare them quickly.

Imagine a field called country with a million documents. Only about two hundred different countries exist, but the same names repeat again and again. Storing “United States” a hundred thousand times would waste space. So doc values do something clever. They first build a sorted list of the unique values in the segment. Then, for every document, they store only the ordinal, which is the position of its value in that list.

flowchart LR
  subgraph L["Sorted unique list in the segment"]
    O0["0: Brazil"]
    O1["1: India"]
    O2["2: Japan"]
  end
  D1["doc 1"] --> O1
  D2["doc 2"] --> O0
  D3["doc 3"] --> O1
  D4["doc 4"] --> O2
        

This has three big benefits. First, space: a small number replaces a long string. Second, speed of comparison: to sort documents by country, you compare small numbers, not strings, because the list is sorted, so smaller ordinals always mean earlier alphabetical order. Third, speed of grouping: to count documents per country you can count how many times each ordinal appears, and only look up the actual string names at the end for the few groups that survive.

Ordinals are local to a segment. To combine results across segments, Elasticsearch needs one shared numbering.

Each segment has its own sorted list of unique values, and the ordinal numbers mean different things in different segments. Ordinal 1 might be “India” in one segment and “Japan” in another. When a terms aggregation runs across a whole shard, it needs to add up counts from all segments. Comparing segment ordinals directly would be wrong.

To fix this, Elasticsearch builds global ordinals. This is a mapping that translates every segment’s local ordinals into one numbering that is valid across the whole shard. With global ordinals, a terms aggregation can count in a single simple array, one slot per unique value, no matter how many segments exist.

flowchart TD
  S1["Segment A local ordinals"] --> G["Global ordinals map"]
  S2["Segment B local ordinals"] --> G
  S3["Segment C local ordinals"] --> G
  G --> AGG["One counting array for the whole shard"]
        

The catch: they are built at query time and stored on the heap

Global ordinals are not saved in the segment files. They are built on demand, the first time an aggregation needs them, and then kept in memory on the JVM heap. When a refresh creates new segments, the old global ordinals are no longer valid and must be rebuilt at the next aggregation. On a busy index that refreshes every second, and with a field that has many unique values, this rebuilding can make the first aggregation after each refresh noticeably slow.

A field can hold a list, such as several tags on one article. Doc values treat lists in a special way.

When a document has many values in one field, the doc values store all of them for that document, but not exactly as you sent them. The values are stored in sorted order, not in the order you provided. For string values, repeated copies of the same value are also merged into one. For numbers, repeated values are kept.

This is usually harmless, since aggregations and sorting do not care about the original order. But it surprises people who read values back from doc values and expect the same list they sent. The original list is still safe in the _source, which is the place to read from when order matters.

Sorting and aggregating on lists

When you sort on a field that holds many values, you must decide which value represents the document. Elasticsearch lets you choose the smallest or the largest, or for numbers, the sum, average, or median. In aggregations, a document with many values is counted once for each of its distinct values, so a document tagged with two labels appears in two buckets.

Doc values look big on paper, but smart encoding keeps them small.

A column of similar values compresses very well. Lucene uses several techniques, chosen automatically for each column, depending on what the values look like.

On disk, a segment stores doc values in two kinds of file. One holds the actual data, and a small companion file holds the metadata that says where each field’s column begins and how it is encoded. You rarely touch these directly, but they show up when you analyze disk usage per field.

This is the most famous advantage of doc values. They live on disk and rely on the operating system, not on the Java heap.

Elasticsearch runs inside the Java Virtual Machine, which has a limited memory area called the heap. If too much data lives there, garbage collection pauses grow and the node can even crash with an out of memory error. Doc values avoid this problem. The files are memory-mapped, which means the operating system treats them as if they were part of memory, loading pieces from disk when needed and keeping the popular pieces in its page cache.

flowchart TD
  Q["Aggregation or sort needs a column"] --> P{"Is the needed part in the OS page cache?"}
  P -->|"Yes"| F["Read at memory speed"]
  P -->|"No"| L["Operating system loads it from disk"]
  L --> F
  F --> R["Result returned, no big heap allocation"]
        

This is why guides say to leave about half of a machine’s memory free for the operating system instead of giving everything to the heap. That free memory becomes the page cache, and a bigger page cache means more of your doc values sit in fast memory. The machine can handle far more data than its heap alone could hold.

Fielddata is the old alternative that still exists for text fields. Knowing the difference saves many outages.

Text fields have no doc values. If you insist on sorting or aggregating on a text field, Elasticsearch can build something called fielddata. It does this by reading the entire inverted index of that field in reverse, turning terms back into document-to-term data, and holding the result on the Java heap. This is slow to build and uses a lot of memory, since it must hold every term of every document in that field.

For this reason, fielddata is turned off by default on text fields. It is a safety feature. If you really need it, you can switch it on, but the better route is almost always to add a keyword sub-field and use its doc values instead.

Let us follow a simple aggregation, counting products per category, from request to answer.

sequenceDiagram
  participant C as Client
  participant Co as Coordinating Node
  participant Sh as Each Shard
  participant Seg as Segments
  C->>Co: Search with terms aggregation on category
  Co->>Sh: Run the query and the aggregation on your shards
  Sh->>Seg: Find matching documents using the inverted index
  Seg->>Sh: Read category ordinals for those documents from doc values
  Sh->>Sh: Count per ordinal, translate to names for top groups
  Sh->>Co: Send partial counts
  Co->>Co: Merge counts from all shards
  Co->>C: Final buckets
        

Step by step

Sorting looks simple, but it depends on doc values in a specific way.

By default, search results are sorted by relevance score, which is calculated from the inverted index and norms. When you ask for a sort by a field, such as price or date, each shard must obtain that field’s value for the candidate documents. It reads them from doc values, keeps the best ones in a small priority list, and sends only that list to the coordinating node. The coordinating node merges the lists from all shards and returns the final page.

Sorting on a keyword field compares ordinals inside each segment, which is fast. Sorting across segments and shards uses the real values for the final comparison. Numeric and date sorting compares plain whole numbers, which is the fastest of all.

Scripts let you compute new values on the fly. Doc values decide how fast that computation runs.

A script is a small program that runs inside Elasticsearch, often written in the built-in Painless language. Scripts are used to calculate custom scores, build values for sorting, or define runtime fields, which are fields computed at query time instead of at index time.

To read a field’s value inside a script, you have two options. The first reads from doc values, which is fast because the value is right there in a column. The second reads from the original document source, which is slow because the document must be fetched and its structure parsed for every match. Modern scripting interfaces steer you towards doc values wherever possible, and they fall back to the source only when a field has no doc values.

There are several ways to get a field back in a search response. They read from different places and behave differently.

Returning fields from doc values can be handy when you do not want to fetch and parse the whole source, or when you want the value as the engine sees it, for example to check what actually got indexed. The trade-off is that the values are normalized. A date returns in the format you request, and a list returns sorted, without duplicates for strings.

Doc values are mainly for sorting and aggregations, but they can quietly help queries too.

Numeric and date range filters are normally answered by the points structure, which is a tree built for range lookups. Points are excellent when the range is selective and the rest of the query is broad. But there is a case where they are not the best choice. If the query already has another very selective condition that leaves only a handful of documents, it is cheaper to check just those few documents’ values in doc values than to walk the whole tree.

Lucene has a smart query type for this. It looks at the cost of each approach for the current situation and chooses between the points structure and doc values on the fly. Elasticsearch uses it for range queries on suitable fields, so you get the better of the two without doing anything.

flowchart TD
  RQ["Range filter on price"] --> C{"Is another clause already very selective?"}
  C -->|"No, range does most filtering"| PT["Use points tree"]
  C -->|"Yes, few candidates remain"| DV["Check candidates with doc values"]
  PT --> M["Matching documents"]
  DV --> M
        

Doc values only fields

You can switch off the searchable index of some field types while keeping doc values. Such a field is stored only for sorting and aggregations. In modern versions, some of these fields can still be searched through doc values, but slowly, because every document’s value must be checked one by one. This is useful for fields that you rarely filter on but that you do want to chart.

You can ask Elasticsearch to arrange the documents inside each segment in a chosen order. Doc values are what make this possible.

Normally, documents inside a segment are stored in the order they arrived. With index sorting, you choose one or more fields, and documents are reordered by those fields when segments are written and merged. The sort fields must have doc values, because that is where the engine reads the values to sort by.

The benefit shows up when your searches sort on the same fields. Since the documents are already in order, a search can stop after reading the first few matches in each segment, because everything after them would sort later. This is called early termination, and it can save a lot of time for queries that want only the top results. Sorted data also tends to compress better, because similar values sit next to each other.

Doc values cost disk space and a little indexing time. For some fields, the cost buys nothing.

You can set the doc values option to false for a field in the mapping. The field then builds no column. It saves disk space, and it shortens indexing slightly. But the field can no longer be sorted, aggregated, or used in scripts through doc values. This is a trade you should make only on purpose.

The type you pick decides which structures a field gets. That makes mapping design a doc values decision.

A common trap is storing identifiers, such as order numbers, as numbers just because they look like numbers. If you never do math or ranges on them, a keyword field is usually a better fit. Exact match lookups on keyword are simple and fast, and you avoid paying for the range-oriented points structure that you do not use.

Every design choice has a price. Here is what doc values cost you, and where those costs appear.

Disk. Doc values add to the size of every segment. The amount depends on how many fields you have, their types, and how much the values compress. Low-cardinality fields with repeated values are extremely small. High-cardinality fields, such as unique identifiers, are much bigger, since every value is different and ordinals do not help.

Indexing speed. Building the columns takes some processor time, and writing them makes segments larger, which makes flushing and merging slightly heavier. For most workloads this cost is small compared with the cost of the inverted index, but for extremely write-heavy pipelines with hundreds of fields, it adds up.

Memory. The column data itself is served by the page cache, so it competes for free operating system memory rather than the heap. The heap holds only global ordinals and temporary structures used during aggregations.

Follow these steps in order when aggregations or sorting feel slow or when disk use grows too fast.

You cannot manage what you cannot see. Elasticsearch gives several views into doc values and their side effects.

Warning signs

A sudden rise in disk reads on nodes that answer aggregations suggests that the page cache is too small for the working set of columns. A growing fielddata size on a text field suggests that someone switched on fielddata and should be redirected to a keyword sub-field. Repeated slow first aggregations after refresh suggest global ordinals being rebuilt too often.

Doc values hold real data. Hiding a field in one place does not hide it everywhere.

Many teams try to protect sensitive fields by leaving them out of the returned source. But if that field has doc values, its values still exist in columns and can be revealed by aggregations, sorting, or scripts unless access is restricted in a proper way. For example, someone who cannot see a salary field in results might still learn the salary range through a histogram.

Most doc values problems show up as a small set of familiar messages and symptoms.

As Elasticsearch grows, more features lean on doc values. Knowing this helps you plan ahead.

The direction is clear. Elasticsearch is moving from being a pure search engine towards being a combined search and analytics engine, and the column structure is at the center of that shift. Details of these features change between versions, so check the documentation for the version you run.

These habits keep mappings healthy and analytics fast.

These are the mistakes that appear in real incidents involving doc values and their relatives.

A short checklist to keep nearby when designing mappings.

Doc values are a fine design, but no design is free of trade-offs.

The message is one of balance. For most fields, the small cost of doc values is worth the power they give. For a small number of large, purely searchable fields, switching them off is a fair and useful saving.

These stories show doc values at work in typical systems.

Theory becomes useful when you apply it. Let us design the mapping of a small product catalog and decide, field by field, which structures each one needs.

Suppose we are building a search page for an online shop. Customers will type words to find products, filter by brand and price, sort by price or newest first, and see counts of products per brand and per category. We also chart daily orders for the sales team. For each field, we ask five questions: will we search it, sort it, aggregate it, script it, or only show it?

Reading the table

Notice how few decisions are hard. Most fields keep their defaults. The two interesting cases are the fields that are only shown and the fields that are only used for exact lookup. For the image address, we do not even need the searchable index, and we can skip doc values too, because the value is always taken from the source. For the stock keeping code, we keep the searchable keyword index, but we might switch doc values off if disk analysis shows the column is large and no dashboard charts it.

If you do not write a mapping, Elasticsearch guesses one. The guesses decide your doc values without you noticing.

When a document contains a field that the index has never seen, dynamic mapping decides its type from the value. A whole number becomes a long. A decimal number becomes a float. A true or false value becomes a boolean. A string that looks like a date in a common format becomes a date. Any other string becomes a text field with an added keyword sub-field.

That last rule is the most important for doc values. The text part supports full-text search and has no doc values. The keyword sub-field has doc values and is used when you sort or aggregate. To prevent giant values from causing problems, the automatic keyword sub-field skips strings longer than a fixed limit, so very long strings have no keyword form and no doc values.

Real documents are rarely flat. Doc values follow the shape of your data in ways that are worth understanding.

Object fields

When a document contains an object, for example an address with a city and a postal code, Elasticsearch flattens it into dotted field names such as address city and address postal code. Each of these leaf fields is a normal field with its own doc values column. There is no special structure for the object itself.

This has a subtle effect on lists of objects. If a document has a list of two products, each with a color and a size, the colors go into one column and the sizes into another. The link that says this color belongs to this size is lost, because doc values store each field independently.

Nested fields

The nested type solves the lost link. Every object in the list is indexed as its own hidden document that sits next to the parent. Each hidden document has its own doc values, so the color and size of one product stay together. Aggregations can then be run on the nested documents with a special nested aggregation, and results can be joined back to the parent with a reverse nested step.

flowchart TD
  P["Parent order document"] --> N1["Hidden nested doc: color red, size M"]
  P --> N2["Hidden nested doc: color blue, size L"]
  N1 --> C1["Own doc values columns"]
  N2 --> C2["Own doc values columns"]
        

Flattened fields

The flattened type is for objects with an unpredictable number of keys, such as user-defined labels. The whole object is indexed as one keyword-like field, and its doc values keep the key together with the value. This avoids creating a new column per key, which protects you from mapping explosion, at the price of supporting only keyword-style behavior on the values.

Not all aggregations behave alike. Knowing how the common ones read doc values helps you predict cost.

Execution choices for terms

A terms aggregation on a string field has more than one way to run. The usual way uses global ordinals, which is excellent when many documents match and the number of unique values is moderate. The other way works directly with the string values and skips building global ordinals. It is better when very few documents match, because it avoids paying the cost of building the shared numbering for the sake of a handful of documents. You can hint which way to use, but the default is usually right.

Segments never change, so how do doc values behave when documents are updated or deleted? The answer is worth knowing.

When you delete a document, Elasticsearch does not erase it from the segment. It marks the document as deleted in a small side record. The document’s values remain in the doc values column until the next merge, but searches and aggregations skip the marked documents, so they do not appear in results.

An update is really a delete plus a new index operation. The old version is marked deleted, and the new version is written into the newest segment with fresh doc values. So a workload that updates the same documents very often produces many deleted entries and many small segments, and the merge process has to work hard to clean them up.

Snapshots and restores

Snapshots copy segment files, and doc values are simply part of those files. Restoring a snapshot brings the columns back exactly as they were, with no rebuilding step. This is also true for searchable snapshots, where the segment files stay in object storage and pieces are fetched into a local cache when needed. Aggregations on such data read the columns through that cache, so the first read of a column can be slower than later reads.

Doc values influence how you size disks, memory, and node tiers. Plan for them explicitly.

For disk, do not guess. Index a realistic sample of your data into a test index with your intended mapping, then run the disk usage analysis. It shows how many bytes each field uses for doc values, the inverted index, stored fields, and other structures. Multiply the result to your full data volume, and add room for replicas and for merges, which temporarily need extra space.

For memory, remember there are two separate budgets. The heap needs to be large enough for the normal operation of the node and for global ordinals of your busiest keyword fields. The rest of the machine memory, outside the heap, is the page cache, which decides how many columns can be served without touching the disk. If your dashboards hit only the last day of data, the working set is small and a modest cache is enough. If they scan months of data, you need a much bigger cache or faster disks.

The idea behind doc values is not unique to Elasticsearch. Seeing the same idea elsewhere makes it easier to remember.

Analytics databases and data lake formats have used column storage for many years. Files designed for analytics store each column together and compress it, so a query that needs only three columns out of a hundred reads only those three. This is exactly the advantage doc values bring to Elasticsearch. The main difference is that doc values live inside a search engine, next to the inverted index, so you get search and analytics on the same data without moving it.

This comparison also explains a design boundary. If your main need is scanning many billions of rows for simple sums, and you have no need for full-text search, a dedicated analytics store might cost less. If you need both text search and fast grouping on the same records, Elasticsearch gives you both from a single copy of the data.

Some ideas about doc values are repeated so often that people treat them as facts. Let us test the most common ones.

Yes, for nearly all field types, including keyword, numbers, dates, booleans, IP addresses, and geo points. The main exceptions are text and annotated text fields, and the binary type.

Text is analyzed into many terms, so a document has no single value to sort by, and text fields build no doc values. Use a keyword sub-field for sorting and grouping.

They are stored on disk in the segment files, and the operating system caches the active parts in its page cache. They do not take up the Java heap, except that global ordinals for string aggregations are built on the heap.

The inverted index maps values to the documents that contain them, and doc values map each document to its value. The first is for finding documents, the second is for reading values across many documents.

Not in place. Doc values are built at index time, so you need a new index with the desired mapping, and you reindex the data into it.

No. Values in a multi-valued field are stored sorted, and duplicate strings are merged. The original order stays in the source.

They are a shard-wide numbering of the unique values of a keyword field, built at query time from each segment’s local ordinals. They make terms aggregations fast but can be slow to rebuild after each refresh.

Only on fields that you are certain are never sorted, aggregated, or scripted. Measure disk use first, check dashboards and scripts, and remember that turning them back on means reindexing.

Ordinals only save space when values repeat. With unique values, every document needs its own entry in the sorted list, so the column is large.

No. Fielddata is built at query time on the heap from the inverted index, mainly for text fields, and it is disabled by default. Doc values are built at index time and live on disk.

Not directly. Relevance comes from the inverted index and norms. Doc values matter when you sort, aggregate, or use scripts, including custom scoring scripts that read field values.

Use the analyze index disk usage operation, which reports the size of each structure for each field, including doc values, the inverted index, and stored fields.

In modern versions, some field types can be searched through their doc values when the searchable index is turned off, but it is slow because values are checked one by one. Keep the normal index on fields that you filter by often.

Usually not. Doc values are served by the operating system cache. A heap that is too big steals memory from that cache and lengthens garbage collection pauses.

Because segment files are memory-mapped, the operating system limit on the number of memory maps must be high enough. Raise it according to the setup guide for your platform.

Yes. Excluding a field from the source does not remove its doc values. If the field is sensitive, use field level security to control access.

Doc values are the column-style structure that Elasticsearch builds at index time to answer one question quickly: what is the value of this field in each document? They power sorting, aggregations, scripts, and returning of normalized values. They are stored in segment files, compressed with several clever techniques, and served through the operating system page cache rather than the Java heap. Strings use ordinals, and global ordinals connect segments for fast grouping. Like the school register in our opening story, Elasticsearch works best when both helpers are present: the inverted index to find, and doc values to measure.