Before diving into cardinality specifically, let’s understand the everyday question it’s designed to answer, and why that question is harder than it first appears.

The Question: “How Many Unique Values?”

In data analysis, “cardinality” simply refers to the number of distinct, unique values in a set. If a field called “user_id” appears across ten million log entries, the cardinality of that field tells you how many actual, distinct users generated those entries — a completely different, and often far more useful, number than the total ten million raw entries themselves.

Why This Is Genuinely Hard at Scale

The most obvious way to count unique values is to keep a running list of every distinct value seen so far, checking each new value against that entire list before deciding whether it’s new. This works fine for a small dataset, but for a field with millions or billions of distinct values, that list itself would need to be stored entirely in memory, and checking each new value against it becomes slower and more memory-hungry as the list grows. This approach simply doesn’t scale to the size of data Elasticsearch is often asked to analyze.

The Solution: A Probabilistic Algorithm

A cardinality aggregation solves this using a technique called HyperLogLog, a well-established algorithm in computer science specifically designed to estimate the number of distinct values in a very large dataset, using a tiny, fixed amount of memory, in exchange for a small, controllable amount of estimation error rather than a perfectly exact count.

The Official Definition

A cardinality aggregation is a single-value metric aggregation that returns an approximate count of the number of distinct values for a specified field across the documents matching a query, calculated using the HyperLogLog++ algorithm rather than by exhaustively tracking every individual unique value.

Let’s look at where cardinality aggregations fit inside Elasticsearch’s broader aggregation framework, and what pieces make the estimation possible.

Where Cardinality Fits Among Aggregation Types

The Core Components Involved

Precision Threshold, in Practical Terms

The precision_threshold setting lets you tune the balance between memory usage and estimate accuracy. A higher threshold means Elasticsearch will use more memory internally to track more detail, producing a more accurate estimate, especially for fields with a very large number of unique values. A lower threshold uses less memory but tolerates a slightly larger margin of error.

Let’s build an intuitive, non-mathematical understanding of how HyperLogLog manages to estimate unique counts without storing every value.

The Core Trick: Patterns in Randomness

HyperLogLog relies on a clever statistical insight: if you take many different values and convert each into a random-looking hash (a fixed-length string of bits), then the more distinct values you hash, the more likely you are to eventually see an unusually long run of leading zero-bits in at least one of those hashes purely by chance. By tracking the longest such run of leading zeros observed, the algorithm can work backward to estimate roughly how many distinct values must have been hashed to make that observation likely.

Splitting Into Buckets for Accuracy

To make the estimate more reliable, rather than relying on just one single observed streak, HyperLogLog splits incoming hashed values into many small buckets, based on part of the hash, and tracks the longest streak independently within each bucket. It then combines all these bucket-level observations together using a carefully designed averaging formula, which produces a much more stable and accurate final estimate than relying on any single observation alone. The precision_threshold setting essentially controls how many of these buckets are used.

Step by Step: What Happens When a Cardinality Aggregation Runs

flowchart TD
    A[Matching documents identified] --> B[Hash target field value per document]
    B --> C[Assign hash to one of many internal buckets]
    C --> D[Track longest leading-zero streak per bucket]
    D --> E{More matching documents?}
    E -->|Yes| B
    E -->|No| F[Combine observations across all buckets]
    F --> G[Apply HyperLogLog estimation formula]
    G --> H[Return approximate distinct count]
        

Let’s trace how a cardinality aggregation’s calculation flows across a distributed Elasticsearch cluster, since this matters for understanding its accuracy characteristics.

The Journey Across Shards

An Elasticsearch index is typically split into multiple shards, each holding a portion of the total documents, often spread across different nodes in the cluster. When a cardinality aggregation runs, each shard independently builds its own HyperLogLog structure based only on the documents it holds, without needing to communicate with other shards during this phase.

Merging Results From Multiple Shards

Once every shard has produced its own partial HyperLogLog structure, Elasticsearch merges these structures together at the coordinating node, combining them into one final structure representing the entire matching document set across the whole index. This merge step is mathematically well-defined and doesn’t require re-scanning any of the original documents.

sequenceDiagram
    participant U as User/App
    participant CO as Coordinating Node
    participant S1 as Shard 1
    participant S2 as Shard 2
    U->>CO: Cardinality aggregation request
    CO->>S1: Forward aggregation to Shard 1
    CO->>S2: Forward aggregation to Shard 2
    S1->>S1: Build local HyperLogLog structure
    S2->>S2: Build local HyperLogLog structure
    S1-->>CO: Return partial HyperLogLog structure
    S2-->>CO: Return partial HyperLogLog structure
    CO->>CO: Merge structures into one final estimate
    CO-->>U: Return approximate distinct count
        

No Persistent Storage of the Estimate

Just like other aggregation results, a cardinality aggregation’s output is calculated fresh for every request and is not stored anywhere in the index. If the underlying data changes between two identical aggregation requests, the estimate will reflect that change automatically the next time it’s calculated.

Approximation is a deliberate design choice, and understanding exactly what you gain and give up helps you use cardinality aggregations appropriately.

The Core Trade-off, in Plain Terms

An exact distinct count guarantees perfect accuracy but becomes prohibitively expensive in memory and computation as the number of unique values grows. A cardinality aggregation deliberately accepts a small, well-understood margin of error in exchange for dramatically better scalability, making it the right tool whenever “very close to correct, instantly” is more valuable than “perfectly correct, eventually.”

Cardinality aggregations are a computational feature rather than a data access mechanism, but they do carry a couple of practical considerations worth understanding.

No Direct Data Exposure of Individual Values

A cardinality aggregation only ever returns a single approximate number representing a count — it never exposes the actual underlying field values themselves as part of its result. This makes it inherently less revealing than aggregations that return actual value buckets, such as a terms aggregation, when the sensitivity of individual field values is a concern.

Understanding how cardinality aggregations behave in production helps ensure they remain both fast and sufficiently accurate for their intended purpose.

What to Watch

Validating Accuracy Where It Matters

For use cases where the acceptable error margin needs to be understood precisely, it’s worth periodically comparing a cardinality aggregation’s estimate against an exact count on a smaller, representative subset of data, confirming the observed error stays within expected bounds for the chosen precision_threshold.

Logging Practices

It’s useful to log which precision_threshold values are being used across different queries in a system, especially as data volume grows, since a setting that was sufficiently accurate at one scale may need adjustment as the underlying cardinality of a field increases over time.

Some ways of applying cardinality aggregations consistently deliver reliable, efficient results, while others lead to unnecessary cost or misleading numbers.

Good Patterns

Anti-Patterns to Avoid

A practical checklist for using cardinality aggregations effectively and appropriately.

Best Practices

Common Mistakes

Cardinality aggregations are a staple of large-scale analytics, appearing throughout dashboards and monitoring tools built on Elasticsearch.

Across every one of these examples, the shared theme is the same: these systems generate enormous volumes of data, and the specific numbers that matter most for decision-making are almost always the distinct counts hidden inside that volume, which cardinality aggregations make practical to compute quickly and at scale.

At the default precision setting, the typical error margin is generally quite small, often within a low single-digit percentage, and this margin can be tightened further by increasing the precision_threshold at the cost of additional memory.

Elasticsearch’s cardinality aggregation is inherently approximate by design. For genuinely exact counts, especially on smaller datasets, alternative approaches such as a properly deduplicated terms aggregation or an external exact-counting process may be more appropriate.

The memory used by the algorithm itself stays essentially fixed regardless of the number of unique values, though overall query performance can still be affected by how many documents must be scanned and hashed in the first place.

Yes, this is a very common and well-supported pattern, such as nesting a cardinality aggregation inside a date histogram to calculate unique counts within each individual time bucket.

Accuracy improves, but memory usage increases correspondingly, and there’s a practical upper limit beyond which further increases stop meaningfully improving accuracy while continuing to add memory overhead.

It generally requires a field type suitable for aggregation, such as a keyword field, numeric field, or date field, rather than a full-text analyzed field, since aggregations typically rely on the field’s exact, unanalyzed values.

On genuinely unchanged data, results should be consistent, since the underlying algorithm is deterministic given the same input data and configuration; any variation typically reflects the underlying data itself having changed between requests.

A cardinality aggregation solves a problem that looks simple on the surface but becomes genuinely difficult at real-world scale: counting how many distinct values exist within a massive dataset without the impossible memory demands of tracking every single one individually. By using the HyperLogLog++ algorithm to observe clever statistical patterns in hashed data, Elasticsearch delivers a fast, memory-efficient, and remarkably accurate estimate instead of an exact but impractical count. That trade-off — a small, controllable margin of error in exchange for constant memory usage and strong performance even at billions of unique values — is exactly the right choice for the vast majority of analytics use cases, from unique visitor counts to unique security event tracking, as long as it’s paired with clear labeling and reserved appropriately, away from contexts that genuinely demand exact, auditable numbers.