The shortest possible answer: it is a metric aggregation that computes descriptive statistics for several numeric fields together, including covariance and correlation between every pair of fields.

Let us build the definition slowly. Elasticsearch is a search and analytics engine. Companies use it to search products, explore logs, count events, and draw charts. Data lives in indices, which are like big labeled cabinets of documents, and each index is cut into pieces called shards so that many servers can share the work.

An aggregation is a way of asking Elasticsearch to summarize many documents into a few numbers or groups, instead of listing the documents themselves. A metric aggregation is a kind of aggregation that calculates numbers from field values: the average price, the highest temperature, the total number of bytes. The matrix stats aggregation is a metric aggregation with a special talent. It works on several fields at the same time, and it reports how those fields relate to each other.

It reports two kinds of things. For each field alone, it reports the count of values, the mean (the average), the variance (how spread out the values are), the skewness (whether the values lean to one side), and the kurtosis (whether the values have heavy tails). For each pair of fields, it reports the covariance and the correlation, which say how the two fields vary together. Because the pairwise numbers form a square table, with one row and one column for every field, the aggregation is called a matrix.

What it is not

The matrix stats aggregation does not find the highest or lowest value, does not compute a sum, and does not give percentiles. It does not draw charts, and it does not prove that one thing causes another. It also does not work on text. It gives a compact statistical portrait of numeric fields, and it leaves the interpretation to you.

Where you will meet it

You will meet the matrix stats aggregation when an analyst wants to know whether price, rating, and sales move together across a product catalog, when an operations team wants to see whether CPU load, memory use, and response time rise and fall in step, when a factory studies which sensor readings are linked to defects, and when a data scientist wants a quick first look at which numeric fields might be useful for a model. In each case, it answers “which of these move together?” in one request.

Most aggregations look at one field at a time. But many real questions are about relationships between fields, and answering them by other means is slow and clumsy.

Elasticsearch offers many single-field summaries. The average of a price field tells you the typical price. The stats aggregation adds the minimum, maximum, and sum. The extended stats aggregation adds spread. These are valuable, but each of them looks at one field in isolation. They cannot tell you whether expensive products also tend to get high ratings, or whether busy servers also tend to answer slowly.

The old ways of studying relationships

• Export and analyze elsewhere. Pull millions of documents out of Elasticsearch into a statistics tool. This is slow, moves a lot of data, and creates copies that must be protected.
• Many separate requests. Ask for the average of each field, then compute the relationships by hand. This gets the averages but not the co-movement, which needs values from the same document at the same time.
• Custom scripts. Write a scripted metric aggregation that accumulates sums of products. This works, but the script is long, slow, and easy to get wrong.

The matrix stats answer

The matrix stats aggregation moves the calculation to where the data already lives. Every shard computes its share of the statistics while reading its own documents, and the coordinating node merges the shares into an exact final answer. Only a tiny table of numbers travels back to you, no matter how many millions of documents were analyzed.

An illustrative story

A short history

The matrix stats aggregation arrived in Elasticsearch 5.0 as a module of its own, next to the core aggregations. It is bundled with the standard distribution, so no extra plugin is needed in normal installations. Its design uses running totals that can be merged across shards, which is why the result is exact rather than approximate. The details of parameters and defaults have stayed stable, though it is wise to confirm them in the documentation for the version you run.

To follow the rest of this guide you need eight simple ideas: aggregation, metric, bucket, doc values, numeric field, shard, doc count, and statistic.

Query and aggregation work together

A search request can contain a query, which chooses which documents to look at, and an aggregation, which summarizes them. The aggregation only sees the documents that match the query. So a request can ask for “the correlation between price and rating for products in the shoes category that are in stock.” The query narrows down the documents, and the aggregation studies them.

The path of an aggregation request

The diagram shows how a request travels. The client usually reaches the cluster through a load balancer or an API gateway, which passes the request to a coordinating node. The coordinating node asks every shard to compute partial results and then merges them.

flowchart LR
    A["Client application"] --> B["Load balancer or API gateway"]
    B --> C["Coordinating node"]
    C --> D1["Shard 1: partial statistics"]
    C --> D2["Shard 2: partial statistics"]
    C --> D3["Shard 3: partial statistics"]
    D1 --> E["Coordinating node merges the partial statistics"]
    D2 --> E
    D3 --> E
    E --> F["Small result table returned to the client"]

The first statistics describe each field on its own: where its values sit, and how widely they are scattered.

The count

The count is simply how many values of a field entered the calculation. In the matrix stats aggregation, it is important to check the count, because documents that lack a value can be left out, as Chapter 11 explains. A count much smaller than you expected is a warning that something is being skipped.

The mean

The mean is the ordinary average: add all values and divide by how many there are. It tells you where the values are centered. If the temperatures of a week are 10, 12, 14, 12, and 12 degrees, the mean is 12.

The variance

The variance measures how far values typically lie from the mean. To compute it, take each value’s distance from the mean, square that distance, add the squares, and divide by one less than the count. Squaring makes all distances positive and gives extra weight to large ones. A small variance means the values huddle near the mean. A large variance means they are scattered.

The squared distances add up to 8. Dividing by the count minus one, which is four, gives a variance of 2. The units of variance are the square of the original units, so this variance is in “degrees squared,” which is awkward to picture. That is why people often take its square root, called the standard deviation. Here, the standard deviation is about 1.41 degrees. The matrix stats aggregation reports the variance, and you can take the square root yourself.

Sample versus population

Statisticians divide by the count minus one, not the count, when the values are a sample of a bigger population. This makes the estimate of the spread slightly larger and more honest for small samples. The matrix stats aggregation reports the sample form, so with very few documents, the numbers include this small correction. With millions of documents, the difference is negligible.

What the aggregation does not report

The matrix stats aggregation does not report the minimum, the maximum, the sum, or the standard deviation. If you need them, use the stats or extended stats aggregations on single fields, or compute the standard deviation from the variance. Chapter 14 compares these aggregations.

A caution about the mean

The mean is sensitive to extreme values. One huge outlier can pull it far away from where most values lie. The variance is even more sensitive, because it squares the distances. Keep this in mind when the data contains rare giant values, such as an order worth a million among orders worth fifty.

Two more statistics describe the shape of a field’s values: whether they lean to one side, and whether they have unusually heavy tails.

Skewness: leaning to one side

Skewness measures the lopsidedness of the values. If the values are balanced around the mean, like a symmetric hill, the skewness is close to zero. If a long tail stretches to the right, with a few very large values, the skewness is positive. If a long tail stretches to the left, with a few very small values, it is negative.

Why skewness matters

Many real measures, such as response times, prices, and file sizes, are strongly skewed to the right. For such fields, the mean is pulled toward the tail and can be misleading, and the variance is dominated by rare large values. A large positive skewness is a hint to look at percentiles or to consider a transformation, such as taking logarithms, before comparing fields.

Kurtosis: how heavy the tails are

Kurtosis measures how much of the spread comes from rare, extreme values in the tails, compared with a standard bell-shaped curve. A field with high kurtosis has “heavy tails”: most values cluster near the mean, but occasionally extreme values appear. A field with low kurtosis has “light tails”: the values stay within a narrow band, with few surprises.

The matrix stats aggregation reports the plain kurtosis, for which a perfect bell-shaped, or normal, distribution gives a value of about three. Some tools subtract three to report the “excess” kurtosis. Check the documentation of your version to be sure which form is reported, and compare like with like.

Use the shape statistics with care

• They need lots of data. With few documents, skewness and kurtosis jump around wildly. Trust them only with a large count.
• They are sensitive to outliers. Kurtosis uses fourth powers, so a single extreme value can dominate it.
• They are descriptive. They tell you the shape, not the reason for it.

Why the aggregation reports them together

The shape of each field matters for interpreting the relationships between fields. The correlation that comes later in the response is most meaningful when the fields are not wildly skewed or heavy-tailed. By returning shape statistics next to the correlations, the aggregation lets you check this without a second request.

These two statistics are the reason the aggregation exists. They measure how two fields move together, one in raw units and one on a friendly scale from minus one to plus one.

Covariance: do they move together?

The covariance of two fields looks at each document and asks, “Is the first field above its mean while the second is also above its mean, or is one above and the other below?” For each document, it multiplies the two distances from the means. If both are above or both are below, the product is positive. If one is above and the other below, the product is negative. The covariance is the average of these products, again dividing by the count minus one.

• A positive covariance means that when one field is high, the other tends to be high too.
• A negative covariance means that when one field is high, the other tends to be low.
• A covariance near zero means that no straight-line pattern of moving together shows up.

The trouble with covariance is that its size depends on the units. The covariance between height in centimeters and weight in kilograms is a different number from the covariance between height in meters and weight in grams, even though the relationship is the same. So the raw number is hard to judge by itself. The sign is meaningful, the size is not easily comparable.

Correlation: the friendly scale

The correlation fixes the units problem. It divides the covariance by the product of the two standard deviations, which strips away the units and squeezes the result into a range from minus one to plus one. This is the Pearson correlation coefficient, the most widely used measure of straight-line relationships.

These bands are rules of thumb. What counts as “strong” depends on your field of work: in physics, 0.9 may be weak, and in the study of human behavior, 0.3 may be interesting.

The matrix of relationships

When you study several fields, you can ask about every pair. With three fields, there are three distinct pairs, and the results can be arranged in a square table with one row and one column for each field. The diagonal of this table compares each field with itself, which always gives a correlation of exactly one. The table is symmetric: the value for the first field against the second equals the value for the second against the first.

The numbers in this table are invented to illustrate how to read it. Warm days go with fewer hot drinks and more cold drinks, and hot and cold sales trade off against each other. One glance at the table tells the whole story.

Why the number of pairs grows quickly

With two fields there is one pair, with three fields three pairs, with four fields six, with ten fields forty-five, and with twenty fields one hundred and ninety. The number of pairs grows with the square of the number of fields. This is the main reason why the cost of the aggregation grows quickly as you add fields, which Chapter 16 discusses.

What the covariance and correlation of a field with itself mean

The response includes each field’s relationship with itself. The covariance of a field with itself is its variance, and the correlation of a field with itself is one. These entries are not new information, but they make the tables complete and easy to read as matrices.

The request has only a few parts. Knowing each one, and the rules it must follow, prevents almost every beginner error.

A matrix stats request is an ordinary search request with an aggregation section. Inside the aggregation section, you give the aggregation a name of your choice, then choose the type, matrix stats, and list its settings. The most common pattern is to set the number of returned documents to zero, since you want only the statistics and not the documents themselves.

Choosing the fields

List only the fields you really want to compare. Because the number of pairs grows with the square of the number of fields, a long list makes the request heavier and the response bigger. A good habit is to start with a small group of fields that you have a reason to compare, such as price, rating, and sales, and not to throw every numeric field into the list.

The narrowing query

The query controls which documents are analyzed, and therefore what the statistics mean. A correlation between price and rating over all products is a different fact from the same correlation over shoes only, or over the last thirty days. Always be clear about the population you are describing. Filtering by time, region, category, or status is very common.

Placement within a bigger aggregation

The matrix stats aggregation can be placed at the top level, or nested inside a bucket aggregation, so that separate statistics are computed for each group. This is a powerful way to compare relationships across groups, and Chapter 13 covers it in detail.

What is not available

• No scripts. The aggregation works on real, indexed field values, not on values computed by scripts. If you need a calculated value, store it in a field at indexing time, or use a runtime field where your version allows it, and consider the extra cost.
• No text fields. Only numeric fields are accepted.
• No per-field options beyond the missing value. The mode applies to all listed fields.

The module behind it

The aggregation lives in a module of Elasticsearch called the matrix stats module. In standard distributions, this module is installed and enabled, so you do not add anything. In customized or minimal builds, it may be absent, and the request would then fail with an error saying that the aggregation type is unknown.

The response is a compact report. Each field gets its own entry, and inside each entry, pairwise results are listed by the names of the other fields.

The result appears under the name that you gave the aggregation. It contains a top-level count of documents, followed by a list with one entry per field.

Building the matrix yourself

The response lists correlations inside each field entry, keyed by the partner’s name. To show a familiar matrix table, take the correlation map of each field and arrange them as rows. The rows and columns should list the fields in the same order. Because the table is symmetric, the entry for the first field against the second equals the entry for the second against the first, which gives a handy self-check: if the two disagree, something is wrong with your assembly, not with the data.

When numbers cannot be computed

Some statistics are undefined in certain situations. With fewer than two documents, the variance cannot be computed, because it divides by the count minus one. If a field has the same value in every document, its variance is zero, and any correlation involving it is undefined, since it would divide by zero. In such cases, the response may show null or a special not-a-number marker for those entries. Treat them as “not enough information,” not as zeros.

The number of decimals

Values are returned as floating point numbers with many digits. Round them for display, and remember that tiny differences in the last digits are normal, as Chapter 17 explains. Do not compare two results for exact equality; compare them with a small tolerance.

Checking the document count

The single most useful sanity check is to compare the document count in the response with the number of documents that your query matches. You can obtain that number with an ordinary count of the same query. If the aggregation used far fewer documents, some of them lack a value in one of the listed fields, and the statistics describe only the subset that has all of them.

Numbers make the method concrete. Here is a complete calculation with five documents and two fields, so that you can see where every reported number comes from.

Imagine five documents, each with two numeric fields. The first field is the number of hours that a customer spent on a tutorial, and the second is the score that the customer received on a quiz. The data are small on purpose, so that the arithmetic is easy to follow.

Step one: the means

The hours add up to 15, so the mean is 3. The scores add up to 20, so the mean is 4.

Step two: the distances from the means

Step three: the variances and the covariance

Divide each total by the count minus one, which is four. The variance of the hours is 10 divided by 4, which is 2.5. The variance of the scores is 6 divided by 4, which is 1.5. The covariance is the total of the products, 6, divided by 4, which is 1.5.

Step four: the correlation

Divide the covariance by the square root of the product of the two variances. The product of the variances is 2.5 times 1.5, which is 3.75, and its square root is about 1.936. So the correlation is 1.5 divided by 1.936, which is about 0.775.

Reading the results

• The relationship is positive. More hours went with higher scores in this small group.
• The relationship is strong but not perfect. A value of 0.775 is well above zero and well below one.
• The sample is tiny. With only five documents, this number could change a lot with one more student. Statistics from tiny samples are fragile.
• The self-entries are consistent. The covariance of each field with itself equals its variance, and each correlation with itself is one.

What Elasticsearch would return

The response would contain a document count of five and two field entries, named hours and score. Each entry would list its mean and variance, its skewness and kurtosis, and the two maps of covariance and correlation with the numbers above, apart from tiny rounding differences in the last digits. You can verify a real response on a small test index in a couple of minutes, which is a very good way to build trust in the aggregation.

The aggregation never needs to gather all values in one place. Each shard keeps small running totals, and the totals can be merged exactly.

To compute a correlation directly, you would need every pair of values in one place, which is impossible at large scale. The trick is that all the statistics can be built from a few running totals that are easy to update one document at a time and easy to combine across shards.

sequenceDiagram
    participant C as Client
    participant N as Coordinating node
    participant S1 as Shard 1
    participant S2 as Shard 2
    C->>N: Search with a matrix stats aggregation
    N->>S1: Run query and aggregation
    N->>S2: Run query and aggregation
    S1->>S1: Read doc values, update running totals
    S2->>S2: Read doc values, update running totals
    S1-->>N: Partial running totals
    S2-->>N: Partial running totals
    N->>N: Merge totals and compute final statistics
    N-->>C: Count, means, variances, shapes, covariance and correlation maps

Exact, not approximate

Some aggregations, such as counting distinct values or computing percentiles, are approximate, because their exact versions would need too much memory. The matrix stats aggregation is different: its running totals contain everything needed for the exact answer, so the result is exact, apart from tiny floating point rounding effects. This is a strong point in its favor for analyses where you want the true relationship in the selected documents.

Numerically stable merging

The running totals are kept in a form, based on distances from running means, that avoids the large rounding errors of the naive method of summing squares of raw values. The merge formula is designed to combine two such states without loss of accuracy. This matters especially for fields with large values and small differences, where naive methods can lose all their precision.

Replicas and repeatability

Each shard is queried on one of its copies, and all copies hold the same documents, so the statistics are the same. However, the order in which shard states are merged can differ from run to run, and floating point addition is not perfectly associative, so the last few digits of a result may differ slightly between two identical requests. This is normal and harmless.

What the shard does not keep

The shard does not keep the individual values, only the totals. That is why memory use per shard depends on the number of fields and pairs, not on the number of documents, and why the aggregation can run over hundreds of millions of documents without needing to hold them in memory.

Real data is messy: fields hold several values, or none at all. The mode and missing settings decide how such documents are treated, and the choices change the answers.

Fields with several values

A field in Elasticsearch may hold an array of numbers in a single document, for example several measurements taken during one visit. The statistics need exactly one number per document per field, so the mode setting chooses how to reduce the array.

One mode for all fields

The mode applies to every listed field in the aggregation. If you need a different rule for different fields, for example the maximum for one and the average for another, store separate single-valued fields at indexing time, or run separate aggregations.

Documents with missing values

By default, a document that lacks a value in any of the listed fields is skipped entirely for the whole calculation. It does not contribute to the statistics of the fields that it does have. This keeps the pairwise comparisons honest, since every statistic is based on the same set of documents, but it can shrink the sample dramatically if the fields are often missing.

flowchart TD
    A["A matching document is read"] --> B{"Does it have a value in every listed field?"}
    B -->|Yes| C["Reduce multi-valued fields with the mode and add the document to the statistics"]
    B -->|No| D{"Is a default given for each missing field?"}
    D -->|Yes| E["Fill in the defaults and add the document"]
    D -->|No| F["Skip the document entirely"]

Supplying default values

The missing setting lets you give a default value for each field, so that documents lacking it are included as if they had that value. This can be useful, for example, to treat a missing “discount” as zero. But it can also distort the results. Filling in values pulls the mean toward the default, shrinks the variance, and can create false relationships. Use defaults only when the default is a truthful value, such as zero for “none,” and not as a way to keep the sample large.

Check what was skipped

To see how many documents were skipped, compare the document count in the response with the number of documents matched by the same query. You can also count documents that lack each field with a separate query. Knowing the share of skipped documents is part of interpreting any correlation.

Missing is not always random

If documents lack a value for a reason connected to the thing you study, skipping them biases the result. For example, if premium customers rarely leave ratings, a correlation between price and rating computed only from rated products describes the customers who bother to rate, not everyone. Think about why values are missing before trusting the numbers.

The aggregation is picky about field types. Knowing the rules saves time, and a few modeling choices at indexing time make the results far more useful.

Doc values must be present

The aggregation reads values from doc values, the column-like store of field values. Numeric fields have doc values by default. If you turned them off for a field to save space, that field cannot be aggregated. Check the mapping when a field is rejected.

Searching several indices

A request can target several indices, such as daily indices of the same data. If a field is mapped as a number in all of them, the aggregation works across them. If a field is a number in some and a keyword in others, the request fails or misbehaves. If a field exists in only some indices, documents from the others lack it and are skipped. Keep mappings consistent across indices that you analyze together.

Choose measures with a meaningful scale

Correlation is unaffected by changing units, but the other statistics are not. Prefer fields whose values are real quantities, such as amounts, durations, and counts. Be careful with codes stored as numbers, such as postal codes, product identifiers, or category numbers. Their arithmetic has no meaning: the average of two postal codes is not a place. A correlation involving such a field is nonsense even if the aggregation happily computes it.

Design fields for analysis

• Store measures as numbers, not text. A price stored as a text field with a currency symbol cannot be analyzed.
• Store one clear number per field. Avoid mixing units in one field.
• Precompute meaningful ratios. A field for “revenue per visit” can be more useful than two separate fields if that ratio is the thing you study.
• Keep mappings stable. A field whose type changes between indices makes multi-index analysis fragile.

Units and scale

The correlation does not care whether a field is in meters or centimeters. The covariance and the variance do. If you compare covariances between different pairs, make sure the fields use consistent units, or prefer correlations for comparison.

Placing the aggregation inside groups lets you ask whether a relationship holds everywhere, or changes from group to group, day to day, or region to region.

A bucket aggregation sorts documents into groups. The terms aggregation makes one bucket for each distinct value of a field, such as each product category. The date histogram aggregation makes one bucket for each time interval, such as each day. The filters aggregation makes buckets from named queries. When you nest a matrix stats aggregation inside a bucket aggregation, Elasticsearch computes a separate set of statistics for each bucket.

flowchart TD
    A["Query selects the documents"] --> B["Bucket aggregation: one group per category, day, or region"]
    B --> C1["Group 1: matrix stats of the chosen fields"]
    B --> C2["Group 2: matrix stats of the chosen fields"]
    B --> C3["Group 3: matrix stats of the chosen fields"]
    C1 --> D["Compare the relationships across groups"]
    C2 --> D
    C3 --> D

Useful ways to group

• By category. Does price relate to rating differently for shoes and for electronics?
• By time. Did the link between load and latency get stronger this week? A date histogram with one bucket per day gives a daily correlation, which you can chart to spot drift.
• By region or customer segment. Are the relationships the same in every market?
• By version or experiment group. Did a new release change how metrics move together?

The danger of the overall number: an example

Overall correlations can be badly misleading. Consider two groups of documents. In group A, the field x takes the values 1, 2, and 3, and the field y takes the values 5, 4, and 3. Inside group A, y falls as x rises, with a correlation of exactly minus one. In group B, x takes the values 6, 7, and 8, and y takes the values 10, 9, and 8. Inside group B, y also falls as x rises, again with a correlation of minus one. Now put all six documents together. The group B documents have both higher x and higher y than group A, so across the whole set, y tends to rise with x, and the overall correlation is about plus 0.81.

This reversal is known as Simpson’s paradox. The overall positive number says that x and y rise together, while inside each group, they move in opposite directions. A hidden group difference created the overall pattern. If you only looked at the overall number, you would draw the opposite conclusion from the truth in each group.

Practical cautions for nested use

• Small groups give unstable statistics. A correlation from five documents can swing wildly. Use the minimum document count setting of the bucket aggregation to hide tiny groups.
• The cost multiplies. Each bucket needs its own running totals, so many buckets with many fields add up. Keep the number of buckets and fields reasonable.
• Bucket limits apply. Elasticsearch limits the total number of buckets in a response. A very fine grouping may hit this limit.
• Use a composite aggregation for many groups. If you must analyze thousands of groups, page through them in batches with a composite aggregation instead of asking for them all at once.
• Compare with the overall result. Run the top-level aggregation next to the grouped one, and look for differences in sign or size.

Elasticsearch offers several numeric aggregations. Each answers a different question, and choosing the right one saves effort and avoids wrong conclusions.

What matrix stats has and lacks

flowchart TD
    A["What do you want to know?"] --> B{"One field or several?"}
    B -->|One field| C{"Need the tails?"}
    C -->|Yes| D["Percentiles"]
    C -->|No| E["Stats or extended stats"]
    B -->|Several| F{"Need relationships between them?"}
    F -->|Yes| G["Matrix stats"]
    F -->|No| H["Separate single-field aggregations"]
    G --> I{"Need regression, tests, or non-linear measures?"}
    I -->|Yes| J["Export or use a dedicated analysis tool"]
    I -->|No| K["Matrix stats is enough for a first look"]

A cross-check with extended stats

To trust your setup, run the extended stats aggregation on one of the fields, and compare its variance with the variance from the matrix stats. They should agree up to the difference between the population form and the sample form of the variance, which is a factor of the count divided by the count minus one. For large counts, they are almost identical. Checking this once builds confidence that you are reading the fields you think you are.

When to leave Elasticsearch

If you need to fit a regression line, test whether a correlation is statistically significant, measure rank correlations that are robust to outliers, or study relationships that are not straight lines, the aggregation is not enough. You can export a sample of the data to a statistics environment and do the deeper analysis there. Use the aggregation to find the promising pairs, and the specialized tool to study them.

A related feature with a similar name

Elasticsearch also has a pipeline aggregation for correlating the shape of a metric across buckets with a reference shape. It answers a different question, such as which terms show a spike in the same time window as an event. Do not confuse it with matrix stats, which correlates several fields within the same documents.

The numbers from the aggregation are exact, but they can still mislead. Most wrong conclusions come from reading correct numbers the wrong way.

An outlier example with numbers

Take ten documents where x runs from 1 to 10 and y is almost the same, with small ups and downs: 2, 1, 4, 3, 6, 5, 8, 7, 10, and 9. The correlation is about 0.94, a very strong relationship. Now change the last y value from 9 to 60, as if a data error had crept in. The correlation drops to about 0.64, even though nine of the ten documents are exactly as before. One bad document damaged the whole picture.

A curve example

Take seven documents where x runs from minus three to three, and y equals x multiplied by itself: 9, 4, 1, 0, 1, 4, 9. The relationship is perfect and predictable, since y is exactly the square of x. Yet the correlation is exactly zero, because the left half of the curve falls while the right half rises, and the two cancel out. The aggregation reports no straight-line relationship, which is true, but it does not say that there is no relationship at all.

Trends and the illusion of connection

Many business measures grow over time: users, revenue, storage, traffic. Two growing measures will correlate strongly simply because both rise with time, even if they have nothing to do with each other. To test a real link, compare their changes from one period to the next, or compute the correlation within short periods where the trend is small.

The multiple comparisons problem

If you put twenty fields into the aggregation, you get one hundred and ninety correlations. Even if no real relationships existed, a handful of them would look notable by pure chance. The more pairs you inspect, the more likely you are to find a striking but false one. Treat surprising discoveries as hypotheses, and check them on new data or a different time period before acting.

A checklist before you trust a correlation

1. Is the document count large enough, and close to the number of matching documents?
2. Are the shape statistics of both fields reasonable, without extreme skewness or kurtosis?
3. Does the relationship hold inside the main groups, and not only overall?
4. Is there a plausible reason for the relationship?
5. Does it hold on a different time period or a different sample?

The aggregation reads every matching document once. Its cost grows with the number of documents, and much faster with the number of fields.

The cost model

For each matching document, the shard reads one value for each listed field and updates the running totals for each field and each pair of fields. If there are k fields, the number of pairs is k times k minus one, divided by two. The work per document is therefore roughly proportional to k plus that number of pairs. The table shows how quickly this grows.

Levers to keep it fast

Scaling with data volume

Because each shard works in parallel on its own documents, the time falls as you add shards and nodes, up to the point where overhead dominates. The merged result is tiny, so the network is rarely a bottleneck. Doubling the number of documents roughly doubles the work per shard, and doubling the number of nodes roughly halves it, when the data is spread evenly.

The role of caches

Doc values are read through the operating system’s file cache. On a warm cache, the aggregation is fast. On a cold cache, it must read from disk, and the first run can be much slower than later runs. When you benchmark, run the request several times and note both the first and the later timings.

Measuring

Every response reports how long it took. Compare the time with a simple aggregation on a single field over the same documents, to see how much the extra fields cost. If the time is larger than expected, profile the request or test it with fewer fields to find the growth.

The aggregation uses very little memory, but numbers on computers have limits. Knowing them helps you spot results that look wrong for technical reasons.

Memory is small

The running state for one calculation consists of a count, a few numbers per field, and a few numbers per pair of fields. With twenty fields, that is a few hundred numbers, a few kilobytes at most. The state does not grow with the number of documents. This is why the aggregation is safe to run over huge indices, and why memory circuit breakers rarely trip because of it. Nested use with very many buckets multiplies the state by the number of buckets, which can matter.

Floating point numbers

Computers store decimal numbers as floating point values with a limited number of digits, about fifteen to seventeen significant digits for the standard double precision. This is plenty for most analyses, but it has consequences.

• Tiny rounding differences. Adding numbers in a different order can change the last digits. Two identical requests may differ in the fifteenth digit.
• Float fields are widened. A field stored as a single-precision float, holding a value such as 0.1, is converted to double precision with a slightly different value, like 0.10000000149. This is harmless for statistics but visible if you compare exactly.
• Huge values. Skewness and kurtosis involve third and fourth powers, so values with very large magnitudes can overflow or lose precision. Consider scaling such fields, for example by expressing bytes in megabytes.

Constant and near-constant fields

A field that has the same value in every document has zero variance, and its correlations with other fields are undefined. A field that varies only a tiny bit may have a huge relative rounding error. If you see null or not-a-number results for a field, check whether it is constant in the population you selected. This often happens after a narrow filter.

Very small samples

With one document, the variance cannot be computed. With two documents, any correlation is exactly plus one or minus one, since two points always lie on a straight line. With a handful of documents, skewness and kurtosis are almost meaningless. Always look at the count before you interpret the shape statistics.

Scale matters for some statistics

Multiplying a field by a constant changes its variance and its covariances with other fields, but leaves its correlations, skewness, and kurtosis unchanged. This is why correlation is the right number for comparing relationships across fields with different units.

Transforming skewed fields

For fields with strong positive skewness, such as amounts and durations, a common practice is to work with the logarithm of the values. The logarithm compresses the long tail and often makes relationships look more like straight lines. You can store the transformed value in its own field at indexing time. Be sure to remember that the correlation is then about the logarithms, not the original values.

Sanity checks that catch technical problems

• Run the aggregation on a tiny hand-checkable set, as in Chapter 9.
• Compare the variance with the one from extended stats.
• Check that the correlation of a field with itself is one, and that the matrix is symmetric.
• Confirm that all correlations lie between minus one and plus one, allowing for tiny rounding.

No tool is perfect. Here is an honest scorecard so that you can decide when to use the aggregation and when to look elsewhere.

The central trade-off: convenience against depth

The aggregation gives a very quick, cheap, in-place look at relationships. In exchange, it offers only the simplest measure of association. For a first screening, this is ideal. For research-grade analysis, it is a starting point.

The second trade-off: breadth against cost

Adding fields reveals more relationships, but the cost and the number of chance findings both grow quickly. A focused list of fields chosen for a reason usually beats a broad sweep.

The third trade-off: completeness against honesty

Skipping documents with missing values keeps every statistic based on the same complete set, but it shrinks the sample and may bias it. Filling in defaults keeps the sample large but may invent relationships. Neither is free, and the right choice depends on what missing means in your data.

The fourth trade-off: the overall view against the group view

One correlation over everything is simple to report but may hide group differences. Correlations per group are richer but multiply the cost and produce many numbers to interpret. Use both, and compare them.

When matrix stats is the wrong tool

If you need to understand a curved relationship, test significance, or fit a model, use a statistics environment. If you only need one field’s spread, use extended stats. If you need tails, use percentiles. If you need the same summary every hour for a dashboard, precompute it with a transform. Use the aggregation where its strengths apply.

Good use of the aggregation is mostly a matter of a few habits, chosen before the first request. These eight cover almost every case.

Pattern 5 in action

An analyst at an online store sees a correlation of about plus 0.4 between the price and the rating of products, and is ready to report that customers rate expensive products higher. Before doing so, she nests the aggregation inside a terms aggregation by category. In eight of the ten largest categories, the correlation is close to zero or slightly negative. The overall positive number came from the fact that categories with higher prices, such as electronics, also happen to have higher average ratings. She reports the category-level finding instead, which is more accurate and more useful.

Pattern 6 in action

An operations team wants a daily chart of the correlation between request rate, CPU load, and response time for each service. Running the aggregation over months of data for dozens of services every time someone opens the dashboard would be heavy. They create a transform that computes the daily statistics for each service once, stores them in a small summary index, and lets the dashboard read from it. Page loads drop from seconds to milliseconds, and the cluster load from the dashboard nearly disappears.

Pattern 7 in action

A factory records sixty sensor readings for every manufactured part, plus a numeric defect score. The engineers cannot examine all pairs at once, so they screen in batches of six fields, always including the defect score. They keep the pairs with the strongest and most stable correlations, then export samples of those and study them with scatter plots and regression tools. The aggregation narrowed the search from thousands of pairs to a handful.

A pattern for documentation

For each analysis that you share, record the query, the fields, the mode and missing settings, the document count, the date, and the version of the data. A correlation without this context cannot be reproduced or challenged, and it tends to be quoted long after its context is forgotten.

These are the traps behind most disappointing analyses. Each is common and easy to avoid once you know it.

Other frequent mistakes

• Ignoring skewness and kurtosis when a few extreme values drive the result.
• Comparing covariances between fields measured in different units.
• Assuming that the missing setting is applied per field in the order of the list, when it is a map by name.
• Forgetting that one mode applies to all fields.
• Using the aggregation on fields that were mapped without doc values.
• Treating tiny differences in the last digits as meaningful.

Aggregated statistics look harmless, but they can reveal individuals when groups are small, and they can expose business secrets. A little design keeps them safe.

The matrix stats aggregation returns only summary numbers, so it seems safer than returning documents. But summaries computed from very few documents can reveal the underlying values, and summaries computed from sensitive fields can leak facts that the reader was never meant to learn.

Access control applies to aggregations

Elasticsearch checks the privileges of the caller on every request, and its document level and field level security are applied to aggregations. A user restricted to some documents gets statistics computed only over those documents. A field that a user may not read behaves as if it were not mapped for that user, so it cannot be used in the aggregation. Use dedicated roles for analysts with the least access that their work needs.

An application layer as a gatekeeper

In many designs, users do not send free-form aggregation requests to the cluster. An application service accepts a limited set of choices, builds the request itself, and returns the result. This lets you enforce an allowed list of fields, a minimum group size, a maximum number of fields, and a maximum time range, before the cluster is ever involved. It also allows you to rate limit, log, and audit the analysis.

Sensitive relationships are sensitive facts

Even without individual values, a relationship can be confidential: the link between price and discount for a customer group, or between a manager’s approvals and a metric. Treat correlation tables that involve sensitive fields with the same care as the fields themselves.

Statistical requests can fail quietly, returning plausible numbers from the wrong documents. A few checks keep both the cluster and the analysis honest.

Partial results are the quiet danger

If a shard fails or times out, Elasticsearch may still return results from the shards that did answer, and it flags the failure in the response. For an ordinary search, missing a few hits is often tolerable. For statistics, missing a shard changes every number, in unknown directions. Make your code check the failure flags, and treat partial results as failures for anything that will be reported.

Metrics worth charting

• Response time by number of fields. Shows the growth and lets you set limits.
• Document count against matching count. Shows how much data is skipped.
• Partial and timed-out responses. Show reliability.
• Key correlations over time. Show drift in the behavior of your system or business.
• Share of missing values per field. Shows data quality trends.
• Search rejections and queue sizes. Show cluster pressure.

Alerts that help

Create a short list of alerts. First, an alert when a scheduled analysis returns partial results. Second, an alert when the document count falls far below its usual level. Third, an alert when the canary request returns unexpected numbers. Fourth, an alert when a monitored correlation changes by more than a set amount within a day. Fifth, an alert when search rejections rise. Each alert should link to a short runbook.

Statistical requests fail in two ways: loudly, with an error, and quietly, with a plausible but wrong answer. The quiet failures are the ones to fear.

flowchart TD
    A["A matrix stats result arrives"] --> B{"Any shard failures or timeout flags?"}
    B -->|Yes| C["Treat the result as unreliable: retry, and investigate the failing shards"]
    B -->|No| D{"Document count close to the matching count?"}
    D -->|No| E["Find out which fields are missing, and decide whether to filter or fill in"]
    D -->|Yes| F{"Any null or not-a-number values?"}
    F -->|Yes| G["Check for constant fields and tiny groups"]
    F -->|No| H["Read the shapes and correlations, and apply the checklist of traps"]

Scenario: a field is not numeric or not mapped

If you list a text field, or a field that does not exist, the request may fail with an error about the field type, or the field may be treated as missing so that no documents qualify. Check the mapping of every listed field, and confirm that it is a numeric type with doc values.

Scenario: the aggregation type is unknown

If the request fails because the aggregation type is not recognized, the matrix stats module may not be installed in that build. Use the standard distribution, or install the module as your deployment allows, and confirm on all nodes.

Scenario: mapping conflicts across indices

When one field is numeric in some indices and text in others, a search across all of them can fail or skip documents. Restrict the request to indices with consistent mappings, or fix the mappings by reindexing.

Scenario: partial results from failed shards

A shard that is unavailable or too slow leaves out its documents. The statistics then describe only the others, silently. Check the shard failure count in every response. Retry once, and if the failure persists, do not publish the result.

Scenario: too few documents after filtering

A narrow filter, or missing values in one field, can reduce the document count to a handful. Correlations from such a sample are unstable, and shape statistics are meaningless. Set a minimum count in your code, and report “not enough data” instead of a number.

Scenario: null or not-a-number values

A field that is constant in the selected population has zero variance, so its correlations cannot be computed. This is common in narrow groups. Treat these entries as unavailable, and do not replace them with zero.

Scenario: too many buckets

Nesting the aggregation inside a fine-grained grouping can hit the limit on the number of buckets in a response. The request fails with an error about too many buckets. Coarsen the grouping, use a minimum document count, or page through groups with a composite aggregation.

Scenario: the numbers changed and nobody knows why

If a correlation suddenly changes, first check whether the population changed: a new data source, a change in how a field is filled, a reindex, or a change in the time range. Compare the document counts and the share of missing values before and after. Most surprising changes come from the data, not from the aggregation.

Habits that make recovery easier

• A checklist of checks, run automatically in the code that consumes results.
• A canary request with known answers.
• Saved query context with every result.
• Minimum counts and clear “not enough data” answers.
• A runbook for partial results and mapping conflicts.

When something looks wrong, match the message or symptom to its likely meaning. This table covers what people meet most often.

A calm way to debug

The aggregation returns numbers. Turning them into a clear message for other people takes a few more steps, and each step protects readers from misunderstanding.

A friendly layout for the correlation table

Show the correlations as a square table with the fields on both axes, round to two decimals, and shade the cells by strength and sign, for example warm colors for positive values and cool colors for negative ones. Show the document count above the table. Because the table is symmetric, you can hide the upper half to reduce clutter. Sorting the list of pairs by the absolute value of the correlation gives a quick ranking of the most interesting relationships.

Pairing numbers with pictures

A correlation coefficient summarizes a relationship in one number, but a picture shows whether that number is a fair summary. A scatter plot of a sample of the documents reveals outliers, curves, and clusters that the number hides. A good practice is to use the aggregation to find the interesting pairs and then fetch a modest random sample of documents for those pairs to draw the scatter plots.

What about Kibana?

Kibana’s built-in visual editors are designed around the common aggregations, and, in many versions, they do not offer matrix stats as a ready-made choice. You can still run the aggregation from the developer console, from your own application, or from scheduled jobs, and display the results in a custom table or chart. Another approach is to precompute the statistics into a small summary index with a transform, and then build ordinary charts on that index. Check your version for what is available.

Alerting on drift

Because the relationship between metrics reflects how a system behaves, a change in correlation can be an early signal of a change in the system. Schedule the aggregation over fixed windows, store the correlations, and alert when a key value moves outside its normal band. Combine this with the checks on counts and partial results so that alerts are based on reliable numbers.

Keep the language honest

• Say “tends to rise together” instead of “causes.”
• Say “in this population and period” instead of “in general.”
• Say “based on N documents” so that readers can judge the sample.
• Say “straight-line relationship” when you mean the Pearson correlation.

Here are four common situations, with rough numbers, showing the aggregation at work. The companies are imaginary, and all figures are invented to show the shape of the effect.

A worked example with numbers

The matrix stats aggregation is a simple idea with a useful payoff: one request, exact statistics, and a table of how your numeric fields move together. Here is everything to remember.

Keep in mind that parameters, defaults, supported field types, and the availability of related features can change between Elasticsearch versions, so confirm exact details against the documentation of the release you run. With that habit, and the ideas in this guide, you can use the aggregation as a fast, exact first look at how your numbers relate, and pair it with the right care to turn those numbers into decisions you can trust.