Three plain ideas, put together: a group, a comparison, and surprise. Once each is clear, the rest follows naturally.

An aggregation in Elasticsearch is a way of summarizing data instead of listing it. Rather than returning the documents that match a query, an aggregation returns counts, averages, groups, and other summaries of them. A bucket aggregation sorts documents into groups called buckets, and reports something about each bucket. The familiar terms aggregation, for example, produces one bucket for each distinct value of a field, and counts the documents in each. It answers the question “what are the most common values?”

The significant terms aggregation is a bucket aggregation that answers a different question: “which values are unusually common in this group of documents?” It works with two sets. The foreground set is the group you are interested in, usually the documents that match your query. The background set is the reference population, by default all the documents of the index. For each term, the aggregation measures how often it appears in the foreground and how often in the background. Terms that appear far more often in the foreground than the background are called significant, and they are returned, ranked by a score.

This is a shift from “popular” to “distinctive.” A term can be extremely popular in the foreground and still not be significant, because it is equally popular everywhere. Words like “the” and “and” are in nearly every document, so they are never distinctive. A term that appears in a modest share of the foreground, but hardly anywhere else, can be highly significant. The aggregation is a tool for finding the fingerprints of a group.

What it is not

A significant terms aggregation is not a full-text search. It does not find documents. It reads the documents that your query has already found and summarizes the terms in a chosen field. It is not a proof of cause: a term being unusually common in a group shows a statistical link, not a reason. And it is not the same as a relevance score, although both compare frequencies. Relevance ranks documents for a query. Significance ranks terms for a group of documents.

The one-sentence version

If you remember one line from this chapter, make it this: a significant terms aggregation compares the terms in a foreground group of documents with a background population, and returns the terms that are unusually frequent in the group, ranked by a statistical score.

Every feature answers a frustration. Here the frustration is that counting the most common things nearly always tells you what you already know.

Suppose your support team has a thousand tickets that mention “checkout failing.” An analyst wants to know what these tickets have in common. She runs a terms aggregation on the “product” field. The result: the most common product is the main web store, with 900 tickets. That is true, and useless, because the main web store also accounts for 85 percent of all tickets in the whole system. The count reflects the size of the store, not anything about checkout failures.

What she really wants is the product, browser, region, or error code that is over-represented among the failing tickets. It might be a payment provider that accounts for 45 percent of the failing tickets but only 2 percent of all tickets. That single fact, invisible in a plain count, is probably the cause of the problem. Finding such needles by comparing many fields by hand, and computing percentages in a spreadsheet, is slow and error-prone. The significant terms aggregation does the comparison for every term in a field, at once, inside the search engine.

A short history

The exact release for each step is worth double-checking in the release notes of your version. The direction of travel is what matters: from “count everything” to “find what is different,” with better statistics and cheaper ways to run it.

Problems it solves

Why not just use a terms aggregation and compare by hand?

You can, and for one or two candidate values it is fine. But when a field has thousands of distinct values, comparing each one’s share in the group and in the whole index is laborious. The significant terms aggregation does the arithmetic for every value, and ranks them, so that the interesting ones rise to the top. It also uses statistics that guard against being fooled by very rare values that happen to be concentrated in a small group.

A dozen words explain nearly everything about this aggregation. Learn them here, and later chapters will read like common sense.

The four numbers behind every term

For each term, the aggregation needs exactly four numbers. The foreground count is how many foreground documents contain the term. The foreground size is how many documents the foreground has. The background count is how many background documents contain the term. The background size is how many documents the background has. Every heuristic is a formula on these four numbers. Nothing else about the documents matters. Grasp this, and the mathematics becomes simple.

Frequency of a term versus frequency of documents

A subtle but important point: the counts here are counts of documents that contain the term, not the total number of times the term occurs. If a word appears ten times in one document, it still adds one to the count. This matches how Lucene stores information, since the number of documents that contain a term is stored with each term in the index, as the document frequency, and can be read without touching the documents.

Why the background matters most

Change the background and the answer changes. If the background is the whole index, a term such as “checkout” may look unusual for tickets about payment. If the background is only the tickets about billing in general, “checkout” may be entirely ordinary. Choosing the background is the most important decision you make with this aggregation, and Chapter 7 is devoted to it.

Quick glossary

Where does the work happen, and which parts of the cluster take part when a significant terms aggregation runs?

A significant terms aggregation is part of a search request. The request carries a query, which defines the foreground, and an aggregation, which describes what to compute. The request goes to every shard involved. Each shard computes its local view of the answer, using only the documents it holds. The coordinating node then combines the shards’ partial answers into the final list.

flowchart TB
    Req["Search with a query and a significant terms aggregation"] --> Coord["Coordinating node"]
    Coord --> S1["Shard 1: local foreground and background counts, top candidate terms"]
    Coord --> S2["Shard 2: local foreground and background counts, top candidate terms"]
    S1 --> M["Coordinating node adds the counts per term and rescores"]
    S2 --> M
    M --> R["Top significant terms returned"]
    

Read the picture from top to bottom. The coordinating node forwards the request to a copy of every shard. Each shard finds the documents matching the query, counts the terms among them, looks up the background statistics for its own documents, scores the terms, and sends back its best candidates. The coordinating node adds up the foreground and background counts of each term across shards, computes the score again on the totals, and returns the best ones.

The main components

Why the shards cannot simply send everything

A field can have millions of distinct terms. If each shard sent all of them, the network and the coordinating node would drown. So each shard sends only its top candidates, a number controlled by the shard size setting. This is efficient, but it makes the answer approximate. A term that scores modestly on every shard, but very well when the counts are added, might not be among any single shard’s top candidates and could be missed. Later chapters explain how to reduce this risk.

Walking one aggregation through the machine

Now we open the request and the response and look at each part. A handful of settings control almost everything.

A significant terms aggregation is defined inside the aggregations section of a search request. You give it a name, name the field to analyze, and optionally set the controls below. The query of the request defines the foreground, and the aggregation reads that foreground. It is common to ask for zero ordinary hits, since the aggregation is the point of the request.

The shape of the response

The response begins with two numbers about the sets. The first is the number of documents in the foreground. The second is the number of documents in the background. Then comes the list of buckets, one for each significant term. Each bucket contains four things.

Reading a result by hand

Suppose the response says that the foreground has 2,000 documents and the background has 10 million. A bucket says: key “payment-gateway-x,” document count 900, background count 200,000, score 9.7. You can read this in plain words: the term appears in 900 of the 2,000 foreground documents, which is 45 percent, and in 200,000 of the 10 million background documents, which is 2 percent. The term is about 22 times as frequent in the group as in the whole index. That is a strong signal. The score condenses this ratio and the absolute difference into one number.

The order of results

Buckets are returned in order of decreasing score. This is different from the ordinary terms aggregation, which orders by count. A term with a small document count can top the list, if its background count is tiny. This is exactly the point, but it also explains why the minimum document count matters: without a floor, terms that appear in two foreground documents and nowhere else could dominate the list with impressive-looking ratios that mean nothing.

Nesting inside other aggregations

A significant terms aggregation can be placed inside another bucket aggregation, so that the foreground is the documents of each parent bucket. For example, group tickets by month, and within each month find the significant error codes. Then the foreground of each nested aggregation is that month’s tickets, while the background stays the whole index, unless you say otherwise. This is a powerful pattern, discussed in Chapter 11.

Ask for hits or not

Most requests of this kind ask for no ordinary search hits, only the aggregation. The query still defines the foreground, but the hits list is set to size zero, saving time and bandwidth. When results are needed as well, keep in mind that the aggregation still summarizes all matching documents, not only the returned page.

The score is a short formula on four numbers. Once you see the formula, the results stop being mysterious.

The default: a balance of two ideas

The default heuristic looks at two things at once. The first is the absolute difference between how common the term is in the foreground and how common it is in the background, measured as percentages. The second is the relative difference: how many times more common it is. The score is the product of the two. A term needs both a large enough difference and a large enough ratio to score well. It scores zero if the term is not more common in the foreground than in the background.

A numbers walkthrough

Let us use invented numbers. The background has 10 million documents. The foreground has 2,000, and they are the tickets that mention “checkout failing.” We examine three terms and one suspicious rare term.

For the first term, the absolute difference is 45 minus 2, or 43 percentage points, and the ratio is 22.5, so the product is about 9.7. For “checkout,” the difference is 65 points and the ratio is about 3.2, giving about 2.1. It is very common in the foreground, but also common in the whole index, so it scores modestly. For “us-east,” the difference is 10 points and the ratio is 1.25, giving about 0.13, which is barely interesting. The rare code has a tiny absolute difference but an enormous ratio, and comes out at about 7.5. It appears in three foreground documents and three background documents in total, so all of its occurrences are in the foreground. This may be a real signal, or pure chance. That is why the minimum document count exists: raise it to ten, and this term is dropped, while the gateway remains.

The available heuristics

flowchart TD
    A["Choose a heuristic"] --> B{"Need a safe default?"}
    B -- Yes --> C["Default: balances absolute and relative change"]
    B -- No --> D{"Want a stronger statistical footing?"}
    D -- Yes --> E["Chi-square or mutual information"]
    D -- No --> F{"Want a simple ratio and accept noise on rare terms?"}
    F -- Yes --> G["Percentage"]
    F -- No --> H["Normalized distance or a custom script"]
    

Two options that change how heuristics behave

Scores are for ranking, not for meaning

A score of 9.7 does not mean “97 percent sure.” The scale of scores depends on the heuristic, the size of the foreground, and the size of the background. Scores are good for ordering terms within one result, and for a rough sense of how far the top is from the rest. They are not comparable across different queries, different heuristics, or different indexes. Do not set a fixed threshold on a score and expect it to mean the same thing next month.

Trying several heuristics

The result of the aggregation depends on two choices: who is in the group, and who is in the comparison population. The second is easy to overlook.

flowchart LR
    A["Whole index: the background set"] --> B["The query selects the foreground set"]
    B --> C["Count each term in the foreground"]
    A --> D["Count each term in the background"]
    C --> E["Compare percentages with a heuristic"]
    D --> E
    E --> F["Rank terms by significance, not by popularity"]
    

The foreground: the question you ask

The foreground is whatever your query matches, or, inside another aggregation, the documents of the parent bucket. It is the group whose character you want to understand. A tight, coherent foreground gives clear answers: “tickets mentioning checkout failing,” “requests with status 500,” “customers who cancelled in March.” A loose, mixed foreground gives muddy answers, because its terms reflect many different stories at once.

The default background

By default, the background is all the documents in the indexes that the request searches, taken from the statistics that each index keeps for each term. It does not depend on the query at all. This makes it fast, because these counts are already stored in the index. It also means that the background contains the foreground: every foreground document is also part of the whole index. That is the situation that the default settings assume.

Setting a custom background

Sometimes the whole index is the wrong comparison. If the index contains tickets from ten products, and you study tickets about one product’s checkout, the terms that distinguish that product from the other nine will dominate the results, and they will hide what distinguishes checkout problems from other problems of that product. A background filter lets you restrict the comparison population: for example, only tickets of that product. Then the question becomes “what is unusual about checkout problems, compared with other tickets of this product?”

Useful comparisons

When the background does not contain the foreground

If you set a background filter that excludes some of the foreground documents, for example, a foreground of the last hour and a background of the previous week, then the assumption that the foreground is contained in the background no longer holds. The statistics behave differently. Several heuristics have an option that tells them so, and it must be set correctly, or the scores will be distorted. Read the documentation for your chosen heuristic before using a background that only partly overlaps.

The cost of a custom background

The default background counts are cheap, because they are read from term statistics. A background filter is not stored anywhere in the index. To count a term in the filtered background, Elasticsearch must run the filter together with a term query, for each candidate term. If there are many candidates, this is many extra queries. It works, and is often worth it, but it makes the aggregation slower, sometimes much slower. Keep the filter simple, keep the candidate list modest, and measure.

Subtleties of the default background

Now we open the box on a single shard. The work splits neatly in two: counting the foreground, and looking up the background.

flowchart TD
    A["Documents matching the query on this shard"] --> B["Read the field values of each matching document"]
    B --> C["Count each term in the foreground"]
    C --> D["For each candidate term, look up its document frequency in the index: the background count"]
    D --> E["Score with the chosen heuristic"]
    E --> F["Keep the best shard-size candidates"]
    

Counting the foreground

As the search collects matching documents, the aggregator reads, for each one, the value or values of the chosen field. For keyword fields, this uses a compact per-field structure that maps each document to a small number standing for its term, and the aggregator increments a counter for that number. Counting in numbers is much faster than comparing text. At the end of the collection, the counters hold, for each term, how many matching documents contained it. A document with a repeated value still counts once for that term.

Looking up the background

The background count of a term is just the number of documents in the shard that contain it. That number is the length of the term’s postings list, and it is stored with the term in the term dictionary, as its document frequency. The aggregator reads it directly, without touching a single document. This is why the default background is fast: the index has already done the counting, when it was built. The size of the background is the number of documents in the shard.

Choosing candidates

The shard cannot score every term of a huge field cheaply, and cannot send every term to the coordinating node. It scores its terms, keeps the best in a small priority structure, and sends only those. The number kept is the shard size. Terms that fail the shard-level minimum are dropped early. This step is where the approximation enters, because the shard scores using only its own counts.

Merging on the coordinating node

The coordinating node receives, from each shard, a list of candidate terms with foreground counts, background counts, and the sizes of the sets. For each term, it adds up the counts across shards, adds up the set sizes, and computes the score again on the totals. Then it sorts by score, applies the minimum document count, and returns the top terms. A term that some shards did not send is treated as not seen there, so its total may be a little lower than the true value.

Two ways to count

Memory: where the aggregation can hurt

Aggregations use memory in proportion to the number of distinct terms they must track. For a field with millions of distinct values, and a large foreground, the counters can be big. Elasticsearch protects itself with circuit breakers, which stop a request that would use too much memory, and with a limit on the number of buckets that a request may create. If you hit these limits, the fix is not to raise them, but to reduce the foreground, choose a lower-cardinality field, use sampling, or exclude noise.

Why the postings connection matters

This aggregation is a nice example of the value of the inverted index. The background count of a term is the length of its postings list, which the index computed once, when documents were indexed. The aggregation gets the background nearly free. It is also why the default background always reflects the whole index and never the query, because the index knows nothing about your query when it stores document frequencies.

The aggregation works on terms, so the kind of field decides what a term is. This choice determines whether the results are useful.

Keyword fields: the easy case

For fields such as product, region, error code, browser, tag, or user identifier, each value is a term, and the aggregation works directly. If a document has several values in the same field, such as a list of tags, each value is a term for that document. This is how “people who liked this item also liked” recommendations are built: each user is a document with a list of liked items in one keyword field.

Text fields: the harder case

A text field is analyzed: split into words, lowercased, and perhaps stemmed. The inverted index of a text field contains the words. The plain significant terms aggregation would need to read those words back per document, which is not available cheaply, since the index goes from words to documents, and not from documents to words. The original way is to enable a memory structure called fielddata on the field, which loads every word of every document into memory. This is very costly on large fields, and is discouraged.

The significant text aggregation

The significant text aggregation solves this. Instead of using fielddata, it takes the matching documents, reads their original text, analyzes it again on the fly with the field’s analyzer, and counts the words. The background counts still come from the inverted index, since the words were indexed with the same analyzer. No extra memory structure is needed. The price is that it does analysis work at query time, in proportion to the amount of text in the foreground, so it is best used with a limited sample.

flowchart LR
    Q["Query finds the matching documents"] --> S["Sampler keeps the best-scoring documents per shard"]
    S --> A["Analyze the text of each sampled document"]
    A --> F["Filter duplicate text"]
    F --> C["Count the words in the sample"]
    C --> H["Compare with background statistics from the index"]
    H --> R["Significant words"]
    

Details that matter for free text

Choosing the field

Analyzing millions of documents is slow, and analyzing weakly related ones is noisy. Sampling helps with both.

Why sample

A query on a big index may match millions of documents, many only loosely related to what you asked. The aggregation would count terms in all of them, spending time and memory, and diluting the signal with the weak matches. A sampler keeps only a limited number of the best-matching documents on each shard, and runs the significant terms aggregation over those. The result is faster, and often clearer, since the sample is the core of the foreground, not its fringe.

The kinds of sampler

A numbers walkthrough

Let us use invented numbers. A query for “outage” matches 4 million documents across 10 shards, taking a large share of the index. Analyzing the text of 4 million documents might take tens of seconds and a lot of memory. A sampler that keeps the top 200 documents on each shard reduces the foreground to 2,000 documents, and the analysis takes a fraction of a second. The terms found are those of the documents that best match the query, so they describe the “core” of outage documents. The background remains the whole index, so the comparison is still against the general population.

Effects of sampling on the answer

Diversity: a defense against one loud voice

Imagine a foreground in which a single account posted five thousand nearly identical messages. Without care, its unusual vocabulary would dominate the significant terms, and you would learn about one spammer, not about the topic. A diversified sampler limits the number of documents per account, so that no single source contributes too many. The remaining sample reflects many voices, and the significant terms describe the topic.

When not to sample

• Small foregrounds. When the foreground has a few thousand documents, sampling loses information without saving much.
• Counts must be exact. Sampling gives approximate counts, so it is unsuitable when exact percentages are the deliverable.
• Rare but important terms. A term present in a small share of the foreground might not appear in a small sample.
• Filter-only foregrounds without a random sampler. The choice among equally scored documents is arbitrary.

On its own, the aggregation describes one group. Nested inside others, it describes many groups at once, and can show how a story changes across time or place.

Inside a terms aggregation: one story per group

Place a significant terms aggregation inside a terms aggregation, and the foreground of the inner one becomes the documents of each outer bucket. For example, group support tickets by product, and within each product find the significant error codes. You now get one list per product, each showing what stands out for that product compared with the background. This is a compact way to profile many groups at once.

Inside a date histogram: a story over time

Place it inside a date histogram, and you get a list of significant terms for each period. This shows how the character of a group changes: which error codes stand out this week, which topics stand out this month. When a new term suddenly appears in the list, it points to something that began at that time, such as a release or an incident. Because the background is the whole index, long-standing patterns are suppressed, and newly unusual terms rise to the top.

Inside a filters aggregation: comparing named groups

A filters aggregation lets you define several named groups with queries, such as “failed requests,” “slow requests,” and “all requests from mobile.” Placing the significant terms aggregation inside gives one list per named group, from a single request. This is useful for side-by-side comparisons.

What stays the same: the background

In all these nestings, the background is by default still the whole index, not the parent bucket. So a term that is unusual for a product may simply be unusual because that product differs from the rest, which may or may not be what you want. If you want to compare each group with a narrower population, set a background filter. Be aware that a background filter is applied to the entire aggregation, not automatically per parent bucket.

Adding companions

The cost multiplies

Nesting multiplies work. If the outer aggregation has a hundred buckets and each contains a significant terms aggregation over a field with thousands of candidate terms, the total memory and time grow quickly, and the request may hit the bucket limit or a circuit breaker. Keep outer aggregations small, keep the size of the inner one modest, and restrict the time window. Test on a small range before running on months of data.

The same mechanism supports very different jobs. Each has its own choice of foreground, background, and field.

Pattern 1: root-cause hunting

flowchart LR
    A["Query: failed requests in the last hour"] --> B["Significant terms on host, version, region, and error code"]
    B --> C["Terms over-represented among the failures"]
    C --> D["Investigate: one host, one release, or one region"]
    

The foreground is the set of failing requests. The background is all requests. The fields are attributes that might explain the failure: host, software version, region, client type, endpoint. If one host appears in 70 percent of failures and only 5 percent of all requests, it is a strong lead. This is the same idea that some Elastic tools use to explain sudden changes in log rates, by comparing a period of change with a baseline period.

Pattern 2: recommendations

flowchart TD
    A["Find users who liked item X"] --> B["Foreground: all items liked by those users"]
    B --> C["Background: all items liked by all users"]
    C --> D["Items unusually popular among the X group"]
    D --> E["Recommend those items"]
    

Each user is a document with a list of liked items in a keyword field. The foreground is the users who liked item X. The aggregation over the list of liked items finds items that these users like much more than users in general. This avoids the trap of recommending the most popular items overall, which everyone likes and which tell you nothing about the taste of the X group. It is the same idea as the doctor’s potato salad: the recommendation is the item that is distinctive, not merely common.

Pattern 3: fraud and abuse leads

Take confirmed chargebacks as the foreground, and all transactions as the background. Significant merchant identifiers, device types, issuing banks, or shipping countries are candidates for investigation. Because fraud is rare and concentrated, the aggregation is well suited. It generates leads for analysts, not automatic decisions, since legitimate customers may share the same attributes.

Pattern 4: keyword and topic discovery

Use articles that match a subject as the foreground, and all articles as the background. The significant text aggregation returns words that characterize the subject: for “electric vehicles” it might return “charging,” “battery,” and “range,” and not the ubiquitous “car.” These words can label topics, populate tag suggestions, or seed a synonym list.

Pattern 5: query expansion

To help users who search with too few words, run their query, take the significant words from the top documents, and offer them as suggested refinements, or add them quietly to the search with a low weight. This is a classic technique for improving recall, and the aggregation provides it in one request. Keep the added terms modest, and test whether they help on judged queries, because expansion can also drift away from the user’s intent.

Pattern 6: profiling customer groups

The foreground is a group, such as customers who cancelled, or customers who bought a premium plan. The aggregation over plan features, acquisition channel, country, or support topics shows what is over-represented in that group. It summarizes a group without needing a statistician, and prompts hypotheses for further study.

The aggregation always returns something. Whether that something is meaningful is your job to judge. Here are the classic ways to be fooled, and the defenses.

Correlation is not causation

A term being unusually common in a group shows that the two go together. It does not show that the term causes the group. A particular server may be over-represented among failures because it serves a busy region, or because it received a new release, or because it is where the failing customers happen to live. The term is a lead, and the causal story must be checked by looking at documents, at timelines, and at experiments.

Small numbers

A term that appears in three foreground documents and three background documents in total has a perfect ratio and no evidence. Chance alone can produce such patterns. Always look at the document count, and require a sensible minimum. Treat any finding built on a handful of documents as a curiosity, not a conclusion.

Many terms, many chances

If a field has a hundred thousand distinct values, some of them will look unusual by pure chance, just as some coin flips out of a huge number will look lucky. The more terms examined, the more likely it is that some apparent finding is an accident. Defenses include a higher minimum count, focusing on fields with a moderate number of values, requiring that findings persist across time windows, and confirming with a second, independent sample.

Redundant evidence

Several top terms may describe the same underlying fact. A host name, its IP address, and its rack number are three names for one machine. A product and its version are related. The list of significant terms can therefore look like many findings while containing one. Group related terms, and count the distinct stories, not the distinct terms.

Mixed groups and Simpson’s paradox

When a foreground blends several different populations, the significant terms may describe the mixture, not any single population, and may even point in the wrong direction for each part. Splitting the foreground into meaningful subgroups, with nested aggregations, often reveals that the story is different in each.

Effect size versus significance

A very large foreground can make even tiny differences statistically clear, while being practically meaningless. A term that is 51 percent of the foreground and 50 percent of the background is a real difference in a huge sample, and a useless one. Always read the percentages, not only the score, and ask whether the difference is big enough to matter.

Validating a finding with a follow-up query

Significant terms aggregations are more expensive than plain counts. Knowing where the cost comes from tells you how to keep them cheap.

Where the cost comes from

A numbers walkthrough

Let us use invented numbers. A foreground of 5 million documents is aggregated over a keyword field with 200,000 distinct values, across 10 shards. Each shard sees, on average, 500,000 documents, and counts values in an array of 200,000 counters: about 800 kilobytes per shard for the counters, which is small. The time is dominated by the reading of 500,000 values per shard, which takes tens of milliseconds to a few hundred, depending on caching. Adding a sampler that keeps 500 documents per shard reduces the reading to 5,000 values per shard, a hundredfold saving. Now imagine a background filter: with a shard size of 100, each shard would run 100 extra filtered queries to get background counts, which may take longer than the entire counting. This shows which choices matter most: the size of the foreground, the presence of a background filter, and nesting.

Tuning levers

Caching and repeated requests

Aggregation results for requests that return no hits can be remembered in the shard request cache, and served again as long as the shard has not changed. On busy indexes, refreshes invalidate it often, so the benefit is greatest on data that no longer changes, such as older time-based indexes. Dashboards that run the same aggregation repeatedly on frozen periods benefit most. Queries that use the current time, in relative form, change with every request and defeat the cache, unless the time is rounded.

Limits that protect the cluster

• The bucket limit. A single request may create only a limited number of buckets in total. Deep nesting can reach it.
• Circuit breakers. A request that would use too much memory is stopped, with an error, before it can harm the node.
• Timeouts. A search can be given a time limit, after which partial results are returned. For aggregations, partial results may be misleading.
• Asynchronous search. Long analyses can be submitted, and their results fetched later, so that clients do not hold connections open.

Every design carries a price. Here is the honest balance sheet of the significant terms aggregation.

How ambitious should you be?

A good analysis gives the same answer tomorrow. Here is why a significant terms aggregation might not, and what to do about it.

Approximation across shards

As Chapter 8 explained, each shard proposes only its best candidates, using its local counts. The coordinating node adds up the counts of the candidates it received. A term that no shard proposed is invisible, even if its combined counts would have ranked it high, and a term proposed by some shards but not others has undercounted totals. The result is an approximation. The error is usually small for strong signals, and larger for borderline ones. Raising the shard size reduces the error at some cost, and a small number of shards reduces it too.

Replicas and segments differ

An aggregation runs on one copy of each shard, chosen by the coordinating node. Different copies of the same shard have different segments, and so can have slightly different term statistics, especially the background counts, which may include deleted documents until merges remove them. Two runs of the same request, served by different copies, can return slightly different scores, and, rarely, a different order for borderline terms. Using a stable preference so that the same user is served by the same copies makes repeated analyses more consistent.

Data changes between runs

Indexes are alive. New documents arrive, old ones are updated or deleted, and merges reshape segments. A foreground defined by “the last hour” changes continuously. For a reproducible analysis, fix the window with absolute times, run it on data that has stopped changing, and record the date, the version of Elasticsearch, and the exact request.

Partial failures

If a shard fails or times out during the request, the response may still arrive, reporting that some shards were unsuccessful. The aggregation then covers only the shards that answered, which changes both the foreground and the background. Always check the summary of shard successes and failures in the response, and treat an analysis with failed shards as incomplete. In automated systems, refuse to act on such a result.

Availability of the feature

The aggregation runs on the data nodes and the coordinating node like any search. If nodes fail, replicas take over, and the aggregation continues on them, with possible slight differences. During heavy indexing, or after restarts with cold caches, it may be slower. Plan capacity so that analytical requests do not compete with user-facing search: run them on a schedule, off-peak, or against dedicated resources.

An aggregation returns summaries, not documents. That feels safe, and is not always so.

Aggregations can reveal what documents cannot

A user who is not allowed to read individual documents may still be allowed to run aggregations, on the theory that summaries are harmless. But a significant terms aggregation can reveal a lot. If the foreground is narrow enough, for example the documents about one named person, the returned terms, with their counts, describe that person: their address, employer, or health topics. Summaries over small groups are effectively the data itself.

Protecting against small-group disclosure

Document-level security and statistics

Elasticsearch can restrict each user’s search to the documents they are allowed to see. For this aggregation, an important question is whether the background statistics reflect only the allowed documents, or the whole index. The default background comes from index-wide term statistics, and term statistics are a well-known way in which information about hidden documents can leak: a background count that includes documents the user cannot see reveals how many such documents contain a term. Test how your version and your security configuration behave, with a restricted user, and consider using an explicit background filter that enforces the same restriction, at some cost in speed.

Free text and sensitive words

The significant text aggregation returns words from the documents. If those documents contain personal data, the returned words may include names, addresses, or identifiers. Restrict this feature on sensitive text, use stop lists or exclusion patterns to hide categories of terms, and review the output before it is shown to a wide audience.

Abuse and cost

• Expensive requests. Deep nesting, high-cardinality fields, and background filters can consume a lot of memory and time. Limit sizes and depths for untrusted users.
• Denial of service by aggregation. A few heavy requests can slow the cluster for everyone. Use timeouts, quotas, and separate resources for analytics.
• Exploring the data through summaries. Automated scripts can map the contents of an index with many aggregations. Use rate limits and audit logs.

Audit and accountability

Analytical queries deserve logging, like any access to sensitive data. Record who ran which aggregation, over which data, when, and with what result size. Where your subscription supports audit logging, turn it on for sensitive indexes. When findings are used for decisions about people, keep the evidence and the request that produced them.

You need to watch two things: whether the aggregation is healthy for the cluster, and whether its answers are healthy for the people who rely on them.

Windows onto the cost

Metrics to chart

flowchart TD
    A["Run the aggregation"] --> B{"Did all shards answer?"}
    B -- No --> R["Rerun, or treat the result as incomplete"]
    B -- Yes --> C["Repeat on another window or another sample"]
    C --> D{"Do the top terms persist?"}
    D -- No --> E["Treat as noise: raise the minimum count or narrow the fields"]
    D -- Yes --> F["Check counts, percentages, and example documents"]
    F --> G["Follow up with targeted queries before acting"]
    

Monitoring the quality of the answers

Cluster metrics tell you whether the aggregation is affordable. They do not tell you whether it is right. For analyses that feed decisions or products, add checks on the results themselves. Measure the overlap of the top terms across repeated runs, as a stability score. Track how often the top term changes from day to day. Keep a small set of known cases, such as a past incident with a known cause, and rerun them after upgrades or configuration changes, to confirm that the aggregation still surfaces the known culprit.

Logging that helps

A troubleshooting playbook

Good habits make significant terms trustworthy and affordable. Bad habits produce confident nonsense.

Patterns that work

Anti-patterns to avoid

A short list you could hand to a new teammate on their first day.

Best practices

Common mistakes

• Reading the bucket count as the overall count. The bucket’s document count is the foreground count. The overall count is separate.
• Forgetting that the default background is the whole index. It may contain unrelated data that distorts the comparison.
• Treating association as cause. The term is a lead, not a verdict.
• Trusting rare terms. Ratios on tiny counts mean little.
• Using the plain aggregation on analyzed text. Use the text variant, and sample.
• Ignoring failed shards. An incomplete run is not a result.
• Comparing scores across heuristics or queries. They live on different scales.
• Setting a background filter that excludes foreground documents without telling the heuristic. The statistics are then distorted.
• Running huge nested analyses interactively. Use asynchronous search, samples, and short windows.

Myths versus facts

Where does the significant terms aggregation quietly earn its keep?

Many well-known organizations have said publicly that Elasticsearch is part of their systems, including Wikipedia’s search, Netflix, Uber, and Stack Overflow. They rarely publish how they use aggregations like this one, so the scenarios below are typical patterns for such workloads, not descriptions of anyone’s private configuration.

Short answers to the questions that come up most often.

If you skim only one part of this guide, make it this one.

The significant terms aggregation finds what is different about a group. It compares the terms in a foreground of documents with a background population, scores each term by how surprising its frequency is, and returns the most surprising ones. It uses the index’s own statistics to make the comparison cheap, and it rewards care: a well-chosen background, sensible evidence floors, sampling, and a habit of verifying every lead.