Before we can understand composite aggregations, we need to understand the problem they were specifically built to solve. Elasticsearch’s aggregations feature lets you compute summary statistics and group data into buckets — similar to a GROUP BY clause in SQL. A terms aggregation, for example, can group all your documents by a field’s distinct values: group log entries by status_code, group sales records by region, group user sessions by browser. This is extraordinarily useful, and for a field with a modest number of distinct values — say, a few dozen status codes, or a handful of regions — a simple terms aggregation works beautifully, returning every bucket in a single response.

But what happens when the field (or combination of fields) you want to group by has millions of distinct values? Imagine grouping web analytics data by the combination of user_id and day — across a large platform, that could easily produce tens of millions of unique buckets. A standard terms aggregation isn’t designed for this: it computes and holds its entire bucket list in memory on each node before returning results, and asking it to return millions of buckets in one response would consume an enormous, unpredictable amount of memory, risking the exact kind of resource exhaustion problem we’ve seen elsewhere in this series with uncontrolled field growth and runaway wildcard queries.

Formally, a composite aggregation is a special type of bucket aggregation designed specifically for this “paginate through every unique combination, reliably and efficiently” use case. It groups documents into buckets based on one or more sources — each source is itself a smaller aggregation, like a terms, histogram, date_histogram, or geotile_grid aggregation — and it combines the results from all of those sources into composite keys, meaning each resulting bucket represents a unique combination of values, one from each source, rather than a single field’s value alone. Crucially, it returns results in a fixed-size page (controlled by a size parameter, similar in spirit to a SQL LIMIT) along with a special token — the after key — that tells you exactly where you left off, so your very next request can pick up precisely where the previous one ended.

It’s worth being precise about what “composite” actually refers to here, since the name can be a little abstract on first encounter. It refers to the fact that a resulting bucket’s key is itself a small JSON object combining values from multiple sources at once — for example, {"region": "APAC", "day": "2026-09-15"} — rather than a single scalar value the way a plain terms aggregation’s bucket key would be. This is precisely what makes composite aggregations so useful for multi-dimensional grouping tasks: analyzing traffic by both country and hour, or sales by both product category and week, where you genuinely need every unique pairing accounted for, not just the top values of either dimension considered independently.

Each entry inside the sources array is itself a small named aggregation, and the order you list them in is not cosmetic — it directly determines the sort order composite buckets are returned in, and correspondingly, the exact shape of the after key you’ll receive back. A composite aggregation with sources [region, day] sorts and pages through results primarily by region, then by day within each region — a composite aggregation with sources [day, region] does the reverse. This ordering guarantee is actually one of composite aggregation’s most valuable, easy-to-overlook properties: unlike some other pagination-style features in Elasticsearch, composite aggregation results are returned in a stable, deterministic, fully-ordered sequence, which is exactly what makes reliable “pick up where I left off” pagination possible in the first place.

flowchart TD
    A["sources: [ terms on 'region', date_histogram on 'day' ]"] --> B["Composite key example:
{ 'region': 'APAC', 'day': '2026-09-15' }"]
    B --> C{Composite aggregation sorted
by source order}
    C --> D["Page 1: first 'size' buckets returned
+ after_key = last bucket's key"]
    D --> E["Next request sets 'after' to that key"]
    E --> F["Page 2 resumes exactly after
the previous last bucket, no gaps, no repeats"]
        

One more architectural detail matters for correctness: because a terms source inside a composite aggregation sorts alphabetically (lexicographically) by default rather than by document count, composite aggregation pagination behaves predictably and consistently across requests — a property that a plain terms aggregation’s default “most documents first” ordering does not reliably offer once you try to page deep into its results, which is precisely one of the underlying reasons composite aggregation exists as a distinct feature rather than simply being “a terms aggregation with a bigger size.”

Two smaller but genuinely useful configuration options round out the architecture. The missing_bucket setting, available on each source, controls whether documents lacking a value for that particular source field are included in their own dedicated bucket (with a null composite key component) or silently excluded from the aggregation entirely — a decision that matters a great deal for data-quality-sensitive use cases like the deduplication and auditing patterns covered later in this article, where a missing value might itself be exactly the kind of anomaly worth surfacing rather than quietly dropping. Similarly, an order setting on individual sources lets you choose ascending or descending sort direction per dimension, which affects the sequence buckets are walked through but, importantly, does not change the fundamental guarantee that every unique combination will eventually be visited exactly once across a complete pagination walk.

To understand why composite aggregation scales so much better than deep pagination through a regular terms aggregation, we need to look at what each approach actually does internally, shard by shard.

A regular terms aggregation, when asked for its top N buckets, works by having every shard independently compute its own local view of the top buckets, which the coordinating node then merges into a single, globally accurate top-N list. This works well for a reasonably small N. But if you try to simulate “pagination” with a plain terms aggregation by requesting an enormous size (say, asking for the top 500,000 buckets just so you can slice out page 5,001 client-side), every shard is forced to compute and transmit its own version of that entire 500,000-bucket list, and the coordinating node must merge all of those massive per-shard lists together — an approach whose memory and network cost grows sharply, and often unpredictably, the deeper you try to page.

A composite aggregation avoids this entirely by never attempting to compute a full ranked list at all. Instead, each shard independently builds a small, bounded structure — conceptually similar to a min-heap or a sorted, size-limited buffer — holding only the current page’s worth of the smallest (in composite-key sort order) buckets it has seen so far, discarding candidates that fall outside that current window as it processes documents. Because the after parameter tells every shard exactly which composite key to start strictly after, each shard can immediately skip past everything it already knows is “before” that point, rather than recomputing or re-transmitting buckets that were already returned in a previous page. The coordinating node then merges each shard’s small, bounded candidate set into the final page of results — a merge operation whose cost stays proportional to the requested size, not to the total number of unique combinations that might exist across the entire dataset.

This is the fundamental architectural insight worth internalizing: a composite aggregation’s per-request cost is governed almost entirely by your chosen size (how big each page is) and the number of shards involved — not by the total cardinality of the underlying data. Whether your dataset has ten thousand or ten billion unique combinations, a single composite aggregation request asking for a page of 1,000 buckets does roughly the same, bounded amount of work per shard. This predictability, more than raw speed on any single request, is precisely why composite aggregation is the tool of choice whenever you genuinely need to walk through every bucket in a high-cardinality dataset, rather than just the top handful.

It’s worth visualizing the cost curves of these two approaches side by side to see exactly why this matters in practice. Deep pagination through a plain terms aggregation has a cost curve that climbs steeply and continuously as you request more buckets — page 1,000 costs meaningfully more than page 1, because the underlying mechanism has to compute and hold an ever-larger prefix of the ranked list regardless of how far into that list you actually care about. Composite aggregation’s cost curve, by contrast, stays essentially flat across every page of a walk — page 1,000 costs roughly the same as page 1, because each shard only ever needs to track a fixed-size window of candidates relative to the current after key, never a growing prefix of the whole ranked sequence. This flatness is not a minor implementation detail; it’s the entire reason composite aggregation can be used confidently on datasets whose total cardinality is unknown or unbounded in advance, a scenario where deep terms pagination would become impractical well before the walk ever finished.

Let’s trace the full lifecycle of a typical composite aggregation job — the pattern almost every real-world usage of this feature follows, whether it’s exporting data, powering a dashboard, or feeding an ETL pipeline.

This lifecycle pattern is deliberately stateless on the server side — unlike Elasticsearch’s scroll API, which historically kept a server-side search context open and consuming resources between requests, a composite aggregation’s after_key is simply a piece of data the client carries forward and resends. This means there’s no server-side session to expire, no background resource being held open while your client is busy processing a page, and no risk of a long-idle client causing cluster-side cleanup problems — a meaningful operational advantage for pipelines that might pause for minutes or hours between pages while performing slow downstream work.

This statelessness also gives composite aggregation a useful idempotency property that many other pagination mechanisms lack: if a client crashes, retries, or simply re-sends the exact same request with the same after value, it receives the exact same page of results again (assuming the underlying data hasn’t changed in the meantime), rather than silently advancing past data it never actually finished processing. This makes it substantially easier to build genuinely reliable, retry-safe batch pipelines on top of composite aggregation than on top of a stateful mechanism, where a lost or duplicated request can leave the server-side session in an ambiguous state that’s hard for the client to reason about after the fact.

The trade-off worth naming explicitly is the same space-versus-time and flexibility-versus-predictability trade-off seen elsewhere in this series, applied here to pagination itself: a random-access, jump-to-any-page interface is more flexible for a human casually browsing results, but fundamentally harder to make memory-safe at very large scale, since supporting “jump to page 500,000” essentially requires being able to cheaply compute what comes before it. A sequential, bookmark-based interface gives up that flexibility in exchange for a guarantee that no matter how large the underlying dataset grows, the cost of the next page never depends on how many pages came before it — precisely the guarantee a batch pipeline processing an unbounded, ever-growing dataset actually needs far more than a human casually browsing ever would.

Composite aggregation’s security profile centers, much like the other features covered in this series, on resource consumption rather than data exposure — the feature exists precisely to make large-scale aggregation safer, but it can still be misused or misconfigured in ways that strain a cluster.

The most direct risk is an oversized size parameter. While composite aggregation is far more memory-bounded than deep pagination through a plain terms aggregation, its per-request cost still scales with whatever size you request — asking for a page of 100,000 or 1,000,000 buckets in a single call, especially with several nested sub-aggregations computing additional metrics per bucket, can still meaningfully strain heap memory and response payload size on a single request, even though the feature’s design keeps that cost bounded and predictable rather than unbounded and runaway. This becomes a genuine concern specifically when composite aggregation requests are triggered by user-controllable input — for example, an internal dashboard or reporting tool that lets a user configure their own page size without a sensible upper limit enforced server-side.

A second, more operational concern is runaway pagination loops in client code — a bug in a pipeline’s request/process/repeat cycle that fails to correctly advance the after parameter, or that retries the same page indefinitely after a transient failure without proper backoff, can generate a sustained stream of redundant load against a cluster. Because composite aggregation queries can involve real computational work per shard (especially with nested metric sub-aggregations), a client-side bug of this kind is a more subtle, harder-to-spot resource risk than an obviously malformed request — it looks like legitimate, well-formed traffic, just needlessly repeated, which is precisely why the monitoring practices in the next chapter matter as much for composite aggregation as for any other potentially expensive query pattern.

Because composite aggregation is so often used inside long-running batch jobs — exporting a full dataset, feeding a nightly ETL run — it’s worth pairing cluster-side monitoring with application-side observability: logging the current after key, the number of pages processed so far, and the elapsed wall-clock time at reasonable intervals lets an operator quickly tell whether a stalled-looking job is genuinely stuck, merely slow due to expected data volume, or has silently crashed partway through without properly reporting failure. This kind of lightweight, application-level checkpoint logging is standard practice across nearly every production system that leans on composite aggregation for bulk data processing.

Proven Patterns

Anti-patterns to Avoid

A composite aggregation is Elasticsearch’s purpose-built solution for reliably walking through every unique combination of one or more grouping dimensions — combinations that could number in the millions or billions — without ever needing to hold the full result set in memory at once. It achieves this through a deliberately simple, deterministic mechanism: sorted, bounded bucket sets on each shard, and a stateless after key the client carries forward to resume exactly where the previous page left off. This design trades the convenience of jumping to an arbitrary page for the guarantee of predictable, bounded resource use regardless of how large the underlying dataset grows.

None of this requires memorizing Elasticsearch’s internal heap-merging implementation details. What it requires is the mental model this article has built from the start: think of the after key as a bookmark in a very long book, think of each page as a manageable stack you can process and set aside, and remember that the whole point of this design is that neither the book’s length nor the size of any single stack ever has to be known in advance for the approach to work reliably. Configure your sources deliberately, choose a sensible page size, checkpoint your progress, and a composite aggregation becomes exactly what it should be — a dependable way to touch every last record in a dataset too large to hold in memory all at once.

Taken together with the other techniques covered across this series — dynamic templates for controlling how new data gets typed, wildcard queries for ad-hoc pattern matching, and edge n-grams for instant, keystroke-driven search — composite aggregation completes a recurring theme: Elasticsearch consistently offers a deliberate, more disciplined alternative wherever a naive, convenience-first approach would eventually break down at real production scale. Learning to recognize which situation you’re actually in — small and simple, or large and unbounded — is what lets you reach for the right tool the first time, rather than discovering the limits of the convenient one the hard way, in production, under load.