To understand fielddata you first need to understand the structure it is built from: the inverted index.

Documents, fields and indices

Elasticsearch stores information as documents, which are pieces of data such as a product, a blog post or a log line. Each document is made of fields, for example a title, a price and a date. A collection of similar documents is stored in an index, roughly comparable to a table in a traditional database. Elasticsearch splits each index into shards and spreads them over the nodes of a cluster, so that the work and the data are shared.

Inside each shard, the data is written into small immutable files called segments. A segment, once written, is never modified. New documents create new segments, and background merging combines small segments into larger ones. This detail will matter a great deal later, because fielddata is built one segment at a time.

The inverted index: a map from words to documents

When you store a text field such as “Quick brown fox,” Elasticsearch runs it through an analyzer, a small pipeline that breaks the text into tokens (the individual words), converts them to lowercase, and optionally removes noise words or reduces words to their roots. The result for our example might be the tokens quick, brown and fox. Elasticsearch then records, for each token, which documents contain it. This structure is the inverted index, and it is the reason full-text search is so fast.

graph LR
  subgraph Docs["Documents"]
    D1["Doc 1: quick brown fox"]
    D2["Doc 2: brown bear"]
    D3["Doc 3: quick red fox"]
  end
  subgraph Inv["Inverted index: term to documents"]
    T1["brown: Doc 1, Doc 2"]
    T2["fox: Doc 1, Doc 3"]
    T3["quick: Doc 1, Doc 3"]
    T4["bear: Doc 2"]
    T5["red: Doc 3"]
  end
  D1 --> T1
  D2 --> T1
  D1 --> T2
  D3 --> T2
  D1 --> T3
  D3 --> T3
  D2 --> T4
  D3 --> T5
        

The question the inverted index cannot answer well

The inverted index is brilliant for searching, but it is awkward for a whole different family of questions. Sorting results by a field, counting how many documents fall into each category, computing an average, or running a script that reads a field’s value for each matching document all need the opposite direction: given a document, what value does it have in this field? Going from a document to its terms with the inverted index would mean scanning the whole term dictionary for every single document, which is far too slow.

Search engines therefore keep a second, separate view of the data organised by document. Elasticsearch has two ways to provide that view. One is stored on disk and built when documents are indexed; it is called doc values. The other is built in memory on demand by flipping the inverted index upside down; that is fielddata.

Fielddata is a temporary, in-memory structure that maps each document to the terms it contains for a particular field.

A precise definition

Fielddata is a data structure that Elasticsearch builds in the heap memory of a node by reading the inverted index of a field and turning it around, so that for each document in a segment it can quickly answer “which terms does this document have in this field?” The process of flipping the structure is called uninverting. The finished result is stored in a cache on the node and reused for later requests that need the same field.

Three words in that definition deserve attention. It is in memory, which means it competes with everything else in the heap for space. It is built at query time, which means the first request that needs it pays a noticeable cost. And it is built from the inverted index, which means that for an analyzed text field it contains the individual tokens, not the original sentence.

What operations need it?

Fielddata is used whenever Elasticsearch has to look at a field from the document’s point of view. The main cases are sorting results by a field, running aggregations such as terms counts or cardinality estimates, and reading field values inside scripts. For most field types this job is done by doc values instead, and fielddata is never involved. It matters mostly for one field type, the analyzed text field, which cannot use doc values at all.

graph TD
  Q["Request: sort, aggregate or script on a field"] --> C{"What type is the field?"}
  C -->|"keyword, number, date, boolean, ip"| DV["Read doc values from disk, cached by the operating system"]
  C -->|"text"| F{"Is fielddata enabled?"}
  F -->|"No, the default"| ERR["Request fails with a clear error message"]
  F -->|"Yes"| FD["Uninvert the inverted index into heap memory"]
        

The default setting is a safety feature learned through many painful production incidents.

The default and the error message

On a text field, fielddata is disabled by default. If you try to sort, aggregate or script on such a field, Elasticsearch refuses and returns an error explaining that text fields are not optimised for operations that need per-document values. The message suggests using a keyword field instead, and mentions that you could enable fielddata on the field, but with a warning that this can use significant memory. That warning is not decoration; it reflects real experience.

This default was introduced with Elasticsearch 5.0, when the old single “string” field type was split into two distinct types, text for full-text search and keyword for exact values. Before that, string fields were analyzed by default and their fielddata could be loaded quietly on the first sort or aggregation. Many clusters ran out of memory without anyone understanding why. Splitting the types and switching fielddata off for text made the cost explicit and turned a hidden trap into a deliberate choice.

Why text fields are especially expensive

A keyword field stores one whole value per document, such as “Delhi” or “Blue.” A text field, by contrast, produces many tokens per document, and across millions of documents produces a very large dictionary of distinct terms. Uninverting it means creating structures that hold every term and connect each document to all of its terms. The more documents, the longer the text and the more varied the vocabulary, the bigger the structure becomes.

Moreover, this memory cost is paid on top of what the index already needs. The inverted index still sits on disk and in the file system cache. The fielddata copy is an extra, in-heap duplicate of the same information in a different shape. And because the heap is shared with everything else the node is doing, a single careless aggregation can push a node toward trouble.

These two structures do the same job in very different ways, and the difference explains almost every practical decision.

Doc values: built early, stored on disk

Doc values are a column-oriented structure written to disk at index time, alongside the inverted index. For each field, the values are stored together, one after another, in document order. Reading the value for a document is therefore a quick, predictable lookup. Because the structure lives in files, the operating system’s file system cache keeps the hot parts in memory automatically, and the Java heap is not burdened with them. Doc values are enabled by default for almost every field type: keyword, numbers, dates, booleans, IP addresses, and more. The main exception is the analyzed text field.

Fielddata: built late, stored in memory

Fielddata takes the opposite approach. Nothing is prepared at index time. When a request first needs the structure, Elasticsearch reads the inverted index, builds the document-to-terms view, and keeps it in heap memory. The upside is flexibility, since it works on analyzed text without any extra index-time preparation. The downside is cost: a delay on first use, memory pressure, and extra pressure on garbage collection.

graph LR
  subgraph Index["At index time"]
    W["Document is written"] --> INV["Inverted index on disk"]
    W --> DVS["Doc values on disk (keyword, numbers, dates)"]
  end
  subgraph Query["At query time"]
    DVS --> OSC["Operating system file cache"]
    INV -->|"only if fielddata is enabled"| UNINV["Uninvert into heap"]
    UNINV --> HEAP["Fielddata cache in JVM heap"]
  end
        

The cleanest way to avoid fielddata is to store the same value in two shapes, one for searching and one for sorting.

Two field types for two jobs

A text field is analyzed, broken into tokens, and designed for full-text search: you can find a document by searching for a single word in a long paragraph. A keyword field is stored as one exact value, not analyzed, and is designed for filtering, sorting and aggregating. If a product name is “Red Running Shoes,” a text field lets people find it by typing “running,” while a keyword field treats the whole phrase as one unit that can be counted or alphabetised.

Multi-fields: the everyday answer

A multi-field lets you index one value in more than one way. The typical pattern is a main field of type text, plus a sub-field of type keyword. You search on the main field and you sort or aggregate on the sub-field. When you let Elasticsearch create mappings dynamically for string values, it does this for you by default: a text field with a keyword sub-field, where the keyword version ignores values longer than a set number of characters, 256 by default. That is why many users see fields called something like name and name.keyword.

The sub-field costs some disk space, because the doc values are written to disk, but it costs almost nothing on the heap. That trade is nearly always worth it. You pay a little at index time and a little in storage, and you avoid the runtime memory hazards of fielddata.

What about case-insensitive sorting?

A plain keyword field is case-sensitive, so “apple” and “Apple” sort in different places and count as different values. If you want them treated as one, you can attach a normalizer that lowercases the value before it is stored. The value remains a single token, so it still has doc values and needs no fielddata. This is far more predictable than sorting on analyzed tokens, which would split “New York” into two separate words.

Understanding the lifecycle explains why the first request is slow and why memory can creep upward.

Lazy loading

By default, fielddata is loaded lazily. Nothing happens when a text field with fielddata enabled is first indexed. The first time a sort, aggregation or script needs it, each shard reads the relevant terms from the inverted index of each segment and builds the structure. That work uses processor time and disk reads, and the memory needed is reserved in the heap. The request that triggers it can be far slower than later ones.

Segments make it a moving target

Because segments are immutable and new ones appear continuously as you index, fielddata is tracked per segment. When new documents arrive and a refresh creates a new segment, the next request must load fielddata for that new segment, while the older segments’ entries are still valid. When segments are merged, the old segments are retired and their fielddata is released, and the merged segment needs a fresh load the next time it is used. On a cluster with heavy indexing, this creates a steady background rhythm of loading and freeing.

sequenceDiagram
  participant U as User
  participant S as Shard
  participant B as Circuit breaker
  participant H as Fielddata cache in heap
  U->>S: Aggregate on text field
  S->>H: Is fielddata loaded for each segment?
  H-->>S: No
  S->>B: Estimate memory needed
  B-->>S: Within limit
  S->>S: Uninvert inverted index
  S->>H: Store structure
  S-->>U: Aggregation result (slow first time)
  U->>S: Same aggregation again
  S->>H: Loaded already
  S-->>U: Aggregation result (fast)
        

Eager versus lazy

Loading on demand means users can feel the cost. Some setups prefer to pay the cost in advance, so that no user request has to wait. Elasticsearch offers a setting to build structures eagerly, when new segments become visible after a refresh, rather than waiting for the first query. Eager loading moves the cost from user requests to the refresh step. It gives smoother query latency but slows down refreshes and uses memory whether or not anyone runs the query. It suits fields that are certain to be used frequently, and it is a poor choice for fields that are rarely queried.

A closer look at the loading step shows exactly why the cost grows with your data.

Step one: the term dictionary

Every segment stores, for each indexed field, a sorted list of all the distinct terms found in that field, called the term dictionary. Next to each term, the segment stores a postings list, which is simply the list of document numbers in that segment that contain the term. The term dictionary and postings lists together are the inverted index. When fielddata is built, Elasticsearch walks through this dictionary from the first term to the last, one term at a time.

Step two: filling in the document rows

For each term, the loader reads its postings list and, for every document in the list, records that this document has this term. By the end of the walk, each document has a small list of the terms it contains, the exact opposite view of the one stored on disk. Terms are kept as compact numbers pointing into a shared dictionary of the actual text, so the same word is not stored again and again for every document that has it.

What decides how big the result is?

flowchart LR
  A["Term dictionary of a segment"] --> B["Read next term"]
  B --> C["Read postings list for that term"]
  C --> D["For each document in the list, add the term to its row"]
  D --> E{"More terms?"}
  E -->|"Yes"| B
  E -->|"No"| F["Document-to-terms structure complete and cached"]
        

Why this work cannot be skipped

Some people ask whether Elasticsearch could load only the parts needed for a particular query. In practice, an aggregation or sort usually touches a large share of the matching documents, and those documents are spread across the whole segment. Loading only a slice would mean repeating the expensive term walk many times. Doing it once for the whole segment and caching the result is the sensible engineering choice, which is why the cost is front-loaded and heavy.

Even if you never enable fielddata on a text field, a related in-memory structure may appear when you aggregate on keyword fields.

What is an ordinal?

Comparing long strings again and again is slow. To speed up counting, Elasticsearch gives every distinct term in a segment a small whole number, an ordinal, based on its alphabetical position. The aggregation then works with these small numbers, which are cheap to compare and count, and converts them back to text only at the end when building the result.

Why “global”?

Ordinals in one segment do not match ordinals in another, because each segment has its own list of terms. To combine results across the segments of a shard, Elasticsearch builds a global ordinals structure that gives every distinct term one number valid across all segments of the shard. Building it needs to read through the terms of all segments, so it can be expensive for fields with a large number of distinct values, and it must be rebuilt when the set of segments changes.

Global ordinals are stored in the heap and count toward the same memory accounting and circuit breaker as fielddata. That is why an aggregation on a high-cardinality keyword field, one with millions of different values, can show up in your fielddata memory statistics even though you never enabled fielddata on it.

Sometimes you really do need to aggregate on analyzed tokens. Here is what that means and what to watch for.

What you actually get

When fielddata is enabled on a text field, sorting and aggregating work on the analyzed tokens, not on the original sentence. If a document’s field contains “New York City,” an aggregation on that field will produce separate buckets for new, york and city, all lowercase, and probably not what most people expect when they think of counting cities. Analysis steps such as stemming, if configured, would also change the words that appear in the results.

There are legitimate uses. If you want to find the most common words in a body of text, for example the most frequent words in customer reviews, aggregating over analyzed tokens is exactly the point. But for names, categories, statuses and identifiers, tokens are the wrong unit, and a keyword field is the correct tool.

Controlling the size with a frequency filter

Elasticsearch lets you limit which terms are loaded into fielddata using a frequency filter. You can tell it to load only terms that appear in a certain range of documents, either as absolute counts or as fractions of the segment, and to skip small segments altogether. Very rare terms, such as typos and one-off tokens, and extremely common terms, such as filler words, are often useless for analysis yet take up a lot of memory. Filtering them out can shrink fielddata dramatically. The trade-off is that the filtered terms will not appear in your results.

Changing the setting on an existing field

The fielddata option on a text field is one of the few mapping properties that can be changed on an existing field, because it does not affect how data is indexed, only what happens at query time. That makes it easy to turn on, and also easy to turn on carelessly. Turning it back off does not immediately free memory already used; the cached structures stay until they are evicted or cleared.

Fielddata is dangerous not because it is slow, but because it takes memory that every other part of the node also needs.

Why heap memory is precious

Elasticsearch runs on the Java Virtual Machine, which manages memory in a fixed-size area called the heap. Everything the node keeps as live objects, including caches, request buffers, the cluster state and fielddata, lives there. A well-run node keeps its heap a comfortable distance from full, because the JVM periodically runs garbage collection to reclaim unused memory. When the heap is nearly full, garbage collection runs more often and for longer, and the node can appear frozen while it is busy cleaning up.

Good practice is to set the heap to no more than about half of the machine’s memory, and below the size at which the JVM loses its compact-pointer optimisation, generally around thirty gigabytes. The rest of the memory is intentionally left to the operating system, which uses it as a file system cache. This is exactly the memory that makes doc values fast. So the two structures compete in opposite ways: doc values lean on the operating system’s cache, while fielddata eats the very heap that has been carefully limited.

The chain reaction

Why memory pressure hurts the whole cluster

This chain reaction shows why a single misjudged aggregation is not only that user’s problem. A node that is struggling with memory can be evicted from the cluster, and if it is also serving as the elected master, the cluster must run a new election. That is why teams often separate the master role from busy data nodes. Fielddata-heavy workloads belong on data nodes with generous heap headroom, never on the nodes that coordinate the cluster.

Elasticsearch includes automatic protections that stop runaway memory use before a node falls over.

What is a circuit breaker?

In an electrical system, a circuit breaker trips when too much current flows, cutting the power to prevent a fire. In Elasticsearch, a circuit breaker estimates how much memory an operation will need, and if that would push usage above a limit, it rejects the operation with an error instead of allowing an out-of-memory failure. The rejected request fails, but the node stays healthy and continues serving everyone else.

The relevant breakers

The fielddata circuit breaker estimates the memory needed to load fielddata and rejects the request if loading would exceed its limit, which by default is a large fraction of the heap, around forty percent. It also applies a small overhead multiplier to estimates to stay on the safe side. Above it sits the parent circuit breaker, which watches total memory use across all the breakers and, in modern versions, can look at real heap usage; its default limit is much higher, around ninety-five percent of the heap. There are further breakers for request memory and in-flight data, but the fielddata breaker is the one that trips when fielddata loading is too greedy.

When a breaker trips, the error message says the data was too large, and it reports the estimated size and the limit. Seeing this message is a clear sign that a request is asking for more memory than the node can safely provide. The right response is to change the query or the mapping, not simply to raise the limit.

The cache size setting

By default the fielddata cache has no size limit; entries stay until their segment goes away or someone clears the cache. You can set a maximum size, in which case the least recently used entries are evicted when the cache is full. This may sound like a cure, but it hides a subtle trap. If the limit is smaller than the fielddata that queries actually need, entries are evicted and then reloaded again and again, an expensive cycle sometimes called thrashing. It is also important that any cache size be lower than the circuit breaker limit, or the breaker will trip before eviction can help.

Guessing is dangerous. A little measuring and a little multiplication go a long way.

There is no magic formula

The memory needed by fielddata depends on the vocabulary, the number of documents, the analyzer, the number of segments and the frequency filter. Because these factors interact, no simple formula gives a precise answer. The reliable method is empirical: load fielddata for the field in a realistic test environment with realistic data and read the memory figure from the statistics. Then scale that figure thoughtfully to production volumes.

Where the memory lands

Fielddata is held on the nodes that hold the shards, not on the node that received the request. If a field’s data is concentrated in a few large shards, the nodes holding them carry more of the load. If shards are spread evenly, the load is spread too. Uneven shard sizes are a common cause of one node hitting its breaker while its neighbours are calm.

Elasticsearch has several caches, and mixing them up is a very common source of confusion.

A tour of the caches

People often use the word “cache” for any memory-based speed-up, but Elasticsearch has distinct caches with different purposes, different owners and different lifetimes. Understanding which is which helps you avoid clearing the wrong one or blaming the wrong one for a slow query.

graph TD
  subgraph HeapArea["JVM heap: limited and precious"]
    FD["Fielddata cache"]
    QC["Node query cache"]
    RC["Shard request cache"]
    OT["Request buffers, cluster state, indexing buffers"]
  end
  subgraph OSArea["Operating system memory: outside the heap"]
    FS["File system cache: inverted index files and doc values"]
  end
        

Why the difference matters for tuning

Clearing the fielddata cache does not clear the request cache, and clearing the request cache does not free fielddata. If a dashboard is slow, the cause may be a cold fielddata load, a cold file system cache after a restart, or simply a heavy query. Diagnosing correctly saves hours. A cold restart, for example, empties every cache at once, so the first queries after startup are slow for several reasons together, not only because of fielddata.

The refresh interval and eager loading

Elasticsearch makes new documents searchable by refreshing, which by default happens about once a second on active indices. Each refresh may create a new segment, and new segments need fresh structures if a query uses them. If you have chosen eager loading for global ordinals, each refresh triggers that build work. On an index receiving constant writes, that can become a heavy tax. A common compromise is to lengthen the refresh interval for indices that do not need near-real-time visibility, which reduces the number of new segments and the associated rebuild work.

Merging and read-only indices

Because fielddata and ordinals are built per segment, having fewer, larger segments reduces overhead. For indices that are no longer written to, such as yesterday’s logs, merging down to fewer segments can make later aggregations cheaper and more predictable. Combined with moving such indices to nodes with suitable resources, this is a standard pattern for time-based data.

You cannot manage what you cannot see, and Elasticsearch offers several simple views into fielddata usage.

Where to look

The cat API for fielddata shows, for each node, how much memory each field’s fielddata is using. The node statistics API reports fielddata memory in bytes and the number of evictions, both per node and optionally per field. The index statistics API gives the same picture per index. The breaker statistics section of node stats reports each breaker’s limit, its estimated current usage, and how many times it has tripped. Together these tell you which fields are large, which nodes are affected, and whether protections are being triggered.

What healthy looks like

On a well-designed cluster that uses keyword fields and doc values for sorting and aggregations, fielddata memory is small or close to zero for text fields, and modest for global ordinals. Evictions are rare or absent. Breaker trip counts stay at zero. Heap usage follows a steady sawtooth pattern, rising as memory is used and falling as garbage collection reclaims it, without creeping upward over days.

flowchart TD
  A["Alert: heap high or breaker tripped"] --> B["Check fielddata memory by field"]
  B --> C{"Is one field dominant?"}
  C -->|"Yes"| D["Find queries using that field"]
  C -->|"No"| E["Check other heap users: request buffers, shard count"]
  D --> F{"Is it text with fielddata enabled?"}
  F -->|"Yes"| G["Move to keyword sub-field or add frequency filter"]
  F -->|"No, keyword"| H["Check cardinality and global ordinals"]
        

When memory trouble strikes, work from the gentlest fix to the most drastic one.

The cache-clearing step in context

Clearing the fielddata cache is a quick, targeted tool: you can clear it for one index, several indices, or specific fields. It is a good emergency measure and a good cleanup step after a mapping change. But clearing only frees memory now. If queries continue to ask for the same fielddata, the cache will simply refill. Treat cache clearing as a bandage, and treat the mapping or query change as the cure.

Why reindexing is sometimes unavoidable

Adding a new sub-field to an existing mapping affects only documents indexed after the change. Older documents will not have values in the new sub-field until they are re-processed. Reindexing to a new index with the improved mapping, then switching an alias over to it, is the cleanest approach because the switch can be almost instant for users, and the old index remains available for rollback until you are confident.

Let us follow a realistic story from the first error to the final fix.

The scenario

A team runs a support-ticket search application. Each ticket has a title, a long description, a customer company name, a status and a creation date. The application shows a table of tickets that users can sort by company name, and a chart of the ten most common companies. The team maps every string as a text field, because it seemed the natural choice, and relies on dynamic mapping for the rest.

Act one: the error

A developer adds the “sort by company name” feature. The first test fails with the message saying that text fields are not optimised for sorting and that fielddata could be enabled. A colleague suggests enabling fielddata on the company field, because it “just works.” The developer applies the change to the live mapping, and the feature suddenly works. Everyone is happy, and the change is shipped.

Act two: the slow burn

Over the following weeks, ticket volume grows. The company field contains names such as “Acme Trading Private Limited,” which the analyzer splits into several lowercase tokens. The chart of top companies now shows entries such as “private,” “limited” and “trading” as the biggest “companies,” which confuses the business team. At the same time, heap usage on the data nodes creeps higher, and the monitoring dashboard shows garbage-collection pauses stretching to several seconds during busy hours.

sequenceDiagram
  participant Dev as Developer
  participant ES as Elasticsearch
  participant Ops as Operations
  Dev->>ES: Sort by company (text field)
  ES-->>Dev: Error: fielddata disabled on text
  Dev->>ES: Enable fielddata on company
  ES-->>Dev: Sort works
  Note over ES: Weeks pass, data grows, heap fills
  Ops->>ES: Alerts: long GC pauses
  ES-->>Ops: Breaker trips on some aggregations
  Ops->>ES: Investigate fielddata memory by field
  ES-->>Ops: Company field is the largest consumer
        

Act three: the diagnosis

The operations engineer opens the cat fielddata view and sees that the company field dominates the fielddata memory on every data node. Node statistics show growing memory and the breaker statistics show trips. The engineer cross-checks the slow log and confirms that the sort and the top-companies chart are the queries causing the loads. The weird buckets in the chart make the modelling error obvious: the field is being aggregated as tokens rather than as whole company names.

Act four: the fix

Changing how a field is modelled touches data, queries and people, so a checklist keeps it calm.

Before you start

The migration path

flowchart LR
  APP["Application uses alias"] --> ALIAS["Alias: tickets"]
  ALIAS -->|"before switch"| OLD["Old index: text with fielddata"]
  ALIAS -.->|"after atomic switch"| NEW["New index: keyword sub-fields"]
  OLD -->|"reindex copies data"| NEW
        

When something looks wrong, match the symptom to the most likely cause, then apply the gentlest fix.

A calm mental checklist

When faced with a memory alarm, resist the urge to change settings at random. First identify what is using memory, using the fielddata views. Second, identify which queries cause it, using logs. Third, decide whether the queries or the mapping should change. Fourth, apply the change in a test environment and measure. Only then apply it to production. Most incidents resolve at step three, when someone realises a field is modelled for the wrong job.

The best fielddata strategy is a data model that never needs it.

flowchart TD
  S["Need to sort or aggregate on a string"] --> Q1{"Is the value a whole label such as a name, tag or status?"}
  Q1 -->|"Yes"| K["Use a keyword field or keyword sub-field with doc values"]
  Q1 -->|"No, it is long free text"| Q2{"Need the most frequent or unusual words?"}
  Q2 -->|"Yes"| SA["Use a text-aware aggregation, no fielddata needed"]
  Q2 -->|"No"| Q3{"Need a derived value occasionally?"}
  Q3 -->|"Yes"| RF["Use a runtime field"]
  Q3 -->|"No"| FD["Last resort: fielddata with a frequency filter and monitoring"]
        

These habits show up again and again in troubled clusters.

Memory-heavy features are also an availability risk, and availability is part of security.

Resource abuse as a risk

Anyone who can run aggregations on a field with heavy memory needs can, deliberately or accidentally, force nodes to load large structures. In a shared cluster, this makes fielddata-enabled fields a route for one team’s queries to degrade everybody else’s service. Restricting which roles can query which indices, and which fields are exposed, reduces the surface area. Circuit breakers help, but they are a last line of defence rather than a substitute for access control and good modelling.

Exposure of token-level content

Enabling fielddata on a text field makes it possible to list the individual tokens found in that field, along with counts. Depending on the content, such tokens could reveal sensitive words, names or fragments. Field-level and document-level security rules still apply in a secured cluster, but you should think about whether allowing token-level aggregation on a field is appropriate before enabling it, especially for fields containing personal or confidential text.

Operational discipline

Running Elasticsearch as a service or on containers changes who presses the buttons, not the physics of memory.

Managed services

Managed offerings, whether from Elastic or from cloud providers, take care of provisioning nodes, configuring the heap and applying upgrades. They do not, however, change how a mapping behaves. A text field with fielddata enabled will consume heap on a managed cluster exactly as it would on your own hardware. Managed services often include monitoring dashboards that show heap use, garbage collection and circuit breaker trips, which are precisely the signals that reveal fielddata trouble, so it pays to learn to read them.

A managed cluster’s instance sizes are often tied to heap sizes, which means that solving a memory problem by moving to a bigger instance has a direct and recurring cost. Since remodelling a field is a one-time effort, while larger instances are billed month after month, fixing the mapping is usually the cheaper path.

Containers and Kubernetes

When Elasticsearch runs in containers, the heap must fit inside the container’s memory limit, with room left over for the operating system’s file cache and other overheads. If the limit is too tight, the container may be killed by the platform when memory rises, which looks like a sudden crash. Because doc values rely on the file system cache, starving the container of non-heap memory quietly slows every aggregation even when fielddata is not in use.

Tiered storage and node roles

Many clusters use tiers: hot nodes for fresh, busy data, warm and cold nodes for older data. Putting heavy aggregation workloads on the tier whose nodes have the most heap headroom, and keeping master-eligible nodes free of such work, is a sound structural choice. Data that is rarely queried should not be forced into memory-hungry structures at all, and the simplest way to ensure that is to avoid fielddata on it.

Fielddata is a tool with a narrow sweet spot. Here is the balanced view.

Comparing the ways to get a document-to-value view

Here is how the ideas play out in typical production settings.

The lesson across all these examples

In each case, the healthy design separates the two jobs of a string: being found, and being grouped or ordered. Text fields do the first job, and keyword fields do the second. Whenever a system tries to make one field do both jobs through fielddata, it pays in memory. Whenever it models both jobs explicitly, it pays a small, predictable disk cost instead.

A few persistent misunderstandings cause a surprising share of the trouble, so let us clear them up.

A short history explains why so much older advice online looks different from current practice.

Short answers to the questions people ask most often about fielddata.

It is an in-memory, document-to-terms structure that Elasticsearch builds by flipping the inverted index of a field, so that sorting, aggregations and scripts can read each document’s values quickly.

Because analyzed text fields have no doc values, so the only way to sort or aggregate on them is fielddata, which lives in the heap and can consume a lot of memory. The message nudges you toward a keyword field.

Keyword fields use doc values for sorting and aggregations, not classic fielddata. However, aggregations on keyword fields can build global ordinals in the heap, which are accounted for alongside fielddata in memory statistics.

Usually not. First try a keyword sub-field, which is the intended solution for whole-value sorting and grouping. Enable fielddata only when you really need token-level aggregation and have planned for the memory.

Because fielddata on a text field contains analyzed tokens. The analyzer split and lowercased your text, so the buckets are tokens rather than original phrases. Aggregate on a keyword field to get whole values.

Use the cat fielddata API for a quick table, the node or index statistics for exact numbers and evictions, and the breaker statistics to see limits and trips. Combine these with heap and garbage-collection metrics.

No. Already loaded structures stay in the cache until they are evicted, their segments are removed, or you clear the cache. After a mapping change, clear the fielddata cache to release memory promptly.

It estimates the memory a fielddata load would need and rejects the request if the total would exceed its limit, protecting the node from running out of heap. The failed request gets an error, but the node stays up.

It can help bound memory, but if the limit is smaller than the working set, entries are evicted and reloaded repeatedly, hurting performance. Any limit should also sit below the circuit breaker limit so eviction can act before the breaker trips.

Doc values are built at index time and stored on disk for most field types, while fielddata is built at query time in memory, mainly for analyzed text fields.

They are different structures, but they live in the same in-memory accounting. Global ordinals speed up aggregations on keyword-like fields, and their size depends on how many distinct values the field has.

Sorting on a text field with fielddata orders documents by tokens, which rarely matches what people expect for names or titles. A keyword sub-field, possibly with a lowercase normalizer, gives predictable whole-value ordering.

Use an aggregation designed for significant terms in text. It analyses the text of matching documents on the fly and highlights unusually frequent words, without requiring fielddata on the field.

Fielddata is the in-memory, document-oriented view that Elasticsearch can build by flipping the inverted index of an analyzed text field. It exists because the inverted index answers “which documents contain this word” brilliantly but cannot answer “what does this document contain” without help. Since version 5.0 it has been disabled on text fields by default, because loading it consumes heap, delays the first request, produces token-level results that surprise users, and can push a node into long garbage-collection pauses and even out of the cluster. The dependable way to sort and aggregate on strings is to give them a keyword sub-field with doc values, leaving fielddata as a rare, monitored fallback. Understand the alternatives, watch the memory, respect the circuit breakers, and your cluster will spend its heap on serving users, not on flipping indexes.