Pipeline Aggregations in Elasticsearch: Doing Math on Top of Your Math
A complete, zero-prior-knowledge walkthrough of what a pipeline aggregation is, why raw aggregation results aren't always enough, how it works internally as a second computation pass, and how real companies use it to spot trends, totals, and anomalies.
Imagine a store manager who already has a report showing “total sales per day” for the past month. That report alone is useful, but the manager’s real question is often one step removed from it: “which day had the biggest jump compared to the day before?” or “what is the running total across the whole month?” or “what is the average of those daily totals?” Answering these follow-up questions doesn’t require going back to the original cash register receipts — it only requires doing a second round of arithmetic on the daily totals that were already calculated. In Elasticsearch, the tool built specifically to perform this “second round of arithmetic on already-calculated results” is called a pipeline aggregation, and this guide explains, from first principles, what it is, why it exists, and how it actually works.
Before pipeline aggregations make sense, we need to recall how ordinary aggregations work, and where their results alone fall short.
A quick reminder: what an aggregation does
An aggregation, at its core, is Elasticsearch’s mechanism for computing summary statistics across a set of documents — counting how many documents fall into each category, summing a numeric field, calculating an average, or grouping documents into buckets based on a date range, a term value, or a numeric interval. A typical aggregation, sometimes called a “bucket” or “metric” aggregation, reads directly from the underlying documents to produce its results, such as “total revenue per month” broken into twelve monthly buckets.
So what is a pipeline aggregation?
A pipeline aggregation is a special category of aggregation that does not read from the raw documents at all. Instead, it takes the output of another aggregation that has already run — the buckets and metrics that were already calculated — and performs a further calculation on top of those results. This is why it is called a “pipeline”: the output of one aggregation stage flows into the input of a second computation stage, much like the output of one step in an assembly line becomes the input of the next.
Think of a school that first calculates each student’s average grade for the semester (this is the ordinary, document-reading aggregation). A separate, follow-up question — “what is the average of all students’ averages, across the whole school?” — does not require re-reading every single test score from scratch. It only requires taking the list of already-computed student averages and calculating one more average on top of them. A pipeline aggregation is Elasticsearch performing exactly this kind of “average of averages” style follow-up calculation, using already-computed results as its input rather than the original raw data.
Two broad families of pipeline aggregation
Pipeline aggregations generally fall into two broad families. A parent pipeline aggregation is nested directly inside a bucketed aggregation and computes a new value for each existing bucket, adding new information to buckets that already exist (for example, calculating the derivative, or rate of change, between each consecutive bucket). A sibling pipeline aggregation sits alongside a bucketed aggregation, at the same level, and produces a single new aggregated result computed across all of that aggregation’s sibling buckets together (for example, calculating the overall average of a whole set of monthly bucket totals).
A pipeline aggregation never touches the original documents. Its entire input is the already-produced output of another aggregation, which is exactly what allows it to compute things like running totals, rates of change, and moving averages, none of which would make sense as a calculation over raw documents alone.
Adds to each bucket
Computes a new value nested inside an existing bucketed aggregation, adding information per-bucket, such as change from the previous bucket.
Summarizes across buckets
Sits alongside a bucketed aggregation and produces one combined result computed across all of its buckets, such as their overall average.
Let’s look at the pieces involved in how a pipeline aggregation is structured and executed relative to the aggregation it depends on.
The “buckets_path”: pointing at already-computed results
A pipeline aggregation is configured with something called a buckets path — essentially an address describing exactly which already-computed metric, from which already-computed aggregation, it should use as its input. This path acts like a set of directions pointing to a specific value nested somewhere inside the results structure that has already been built by the earlier, ordinary aggregation.
Common categories of pipeline aggregation
Several well-known pipeline aggregations solve recurring analytical needs. A derivative calculates the change between consecutive buckets, useful for understanding rate of change over time. A cumulative sum produces a running total that accumulates across buckets in order, useful for showing growth over a period. A moving function (including common patterns like a moving average) smooths out short-term fluctuations by calculating a rolling statistic over a sliding window of recent buckets, useful for revealing an underlying trend obscured by noisy, day-to-day variation. A bucket sort and related sibling aggregations like max bucket, min bucket, and avg bucket find or summarize interesting buckets, such as identifying which single bucket had the single highest value across an entire set.
flowchart TB
A["Raw Documents
(sales transactions)"] --> B["Stage 1: Date Histogram Aggregation
bucket per day, sum of sales"]
B --> C["Day 1: $1,200"]
B --> D["Day 2: $1,450"]
B --> E["Day 3: $1,100"]
B --> F["Day 4: $1,900"]
C --> G["Stage 2: Pipeline Aggregation
(cumulative sum, reads Stage 1's output)"]
D --> G
E --> G
F --> G
G --> H["Day 1: $1,200
Day 2: $2,650
Day 3: $3,750
Day 4: $5,650"]
“Why can’t you compute a running total or a moving average using a normal metric aggregation alone?” — A strong answer explains that ordinary metric aggregations operate independently per bucket, with no built-in awareness of neighboring buckets’ values, whereas a pipeline aggregation is specifically designed to consume a whole sequence of already-computed bucket values together, which is required for anything involving order, accumulation, or comparison across buckets.
Let’s trace, step by step, exactly what Elasticsearch does internally when a request containing a pipeline aggregation is executed.
The underlying aggregation runs first, across all relevant shards
The ordinary bucketed or metric aggregation that the pipeline depends on executes normally, reading and scanning the actual documents distributed across the target shards.
Shard-level results are gathered and combined
The coordinating node collects partial results from every shard and merges them into the complete, final set of buckets and metrics for the underlying aggregation.
The pipeline aggregation’s buckets path is resolved
Elasticsearch follows the configured buckets path to locate the exact already-computed values it needs to use as its input, now that the underlying aggregation’s full, combined results exist.
The pipeline’s own calculation is performed on those values
Whatever calculation the specific pipeline aggregation type performs — a running total, a difference between consecutive values, a rolling average across a sliding window — is computed using only the already-produced bucket values as its input.
The pipeline’s results are merged into the response
The newly computed pipeline values are added either into each corresponding bucket (for a parent pipeline) or as a single additional result alongside the whole bucketed set (for a sibling pipeline).
The complete, combined response is returned to the caller
The final response includes both the original aggregation’s results and the pipeline aggregation’s derived results together, in a single reply.
Because a pipeline aggregation only runs after the underlying aggregation’s results have already been fully assembled, it happens as a distinct, additional computation phase, not as part of the initial per-shard document scanning. This is precisely why pipeline aggregations are able to reason about sequences and relationships between buckets that a single-pass, per-document aggregation cannot.
Understanding how data flows from raw documents through to a final pipeline result clarifies both what pipeline aggregations can and cannot express.
Two distinct computation phases within one request
A search request containing a pipeline aggregation effectively involves two conceptually distinct phases occurring within the same overall request-response cycle: an initial phase where the underlying aggregation genuinely scans and summarizes documents, and a subsequent phase where the pipeline aggregation performs pure arithmetic over those already-summarized results. From the caller’s perspective, this all happens transparently within a single request, but understanding the two-phase nature helps explain both the capability and the limits of what a pipeline aggregation can compute.
Ordering matters for many pipeline aggregations
Certain pipeline aggregations, particularly those involving concepts like “previous bucket” or “running total,” depend meaningfully on the order in which the underlying buckets are arranged, most commonly a chronological date-based order. Ensuring the underlying aggregation’s buckets are sorted in the intended, meaningful order is therefore an important part of correctly using order-sensitive pipeline aggregations like cumulative sums or derivatives.
Live, continuously refreshing data
Just as with any other aggregation, a pipeline aggregation is recalculated fresh on every request, reflecting whatever data currently exists in the index as of the most recent refresh. This means a cumulative sum or moving average shown on a live dashboard naturally updates to reflect new documents as they are indexed over time, without any separate maintenance process needed to keep the pipeline calculation current.
Per Request
Parent & Sibling
Re-Scanned
Pipeline aggregations unlock genuinely powerful analytics, but they come with real limitations worth understanding clearly.
Advantages
- Enables analytical calculations (running totals, rates of change, moving averages) that are simply impossible with document-scanning aggregations alone.
- Reuses already-computed aggregation results, avoiding the cost of a second, separate document scan.
- Supports both per-bucket enrichment (parent pipelines) and whole-set summarization (sibling pipelines) from a shared, consistent mechanism.
- Keeps trend-revealing analytics, like moving averages, computed server-side rather than requiring the calling application to do this math itself after fetching raw bucket data.
- Composable: multiple pipeline aggregations can be chained, with one pipeline aggregation consuming the output of another.
Disadvantages & Trade-offs
- Entirely dependent on the correctness and structure of the underlying aggregation; a misconfigured buckets path fails to resolve meaningfully.
- Order-sensitive pipeline aggregations require the underlying buckets to already be arranged in the correct, meaningful sequence.
- Cannot reach back into raw document fields directly; if a calculation genuinely needs document-level detail beyond what was already aggregated, a pipeline aggregation alone cannot provide it.
- Chaining many pipeline aggregations together can make a request’s overall logic harder to read and reason about at a glance.
Pipeline aggregations are not a security feature, but their derived-analytics nature intersects with data-sensitivity concerns worth noting.
Derived values can still reveal sensitive patterns
Even when direct access to certain raw documents or fields is restricted, a pipeline aggregation’s derived output (a running total, a rate of change, a moving average) built from permitted, already-aggregated summary data can still indirectly reveal meaningful patterns about sensitive underlying figures, such as revenue trends or headcount changes over time, even without exposing any individual document. Access-control review should consider not just raw document visibility, but also what derived trend information becomes visible through pipeline-aggregated summaries.
Buckets paths and unintended data exposure
Because a pipeline aggregation’s buckets path can reference deeply nested aggregation structures, care should be taken when allowing less-trusted callers to construct arbitrary aggregation requests, ensuring that access-control restrictions applied to the underlying data are consistently respected regardless of how many layers of aggregation and pipeline computation are stacked on top of it.
When exposing aggregation capabilities, including pipeline aggregations, to less-trusted users or external systems, consider whether derived trend data (growth rates, running totals) over a sensitive metric requires the same access restrictions as the underlying raw metric itself, rather than assuming aggregation automatically anonymizes or protects sensitive information.
Because pipeline aggregations add an extra computational phase on top of already-expensive aggregation work, keeping an eye on their real-world cost and behavior matters.
What to watch
- Overall aggregation request latency: tracking whether requests combining multiple pipeline aggregations, especially chained ones, take noticeably longer than the underlying aggregation alone would.
- Buckets path resolution errors: monitoring for requests that fail because a configured buckets path could not be resolved, often signaling a mismatch between the pipeline configuration and the actual structure of the underlying aggregation.
- Bucket count and window size for moving-style pipeline aggregations: confirming that sliding-window calculations like moving averages are configured with a genuinely meaningful number of buckets for the data’s actual granularity and volume.
- Dashboard refresh cost: for live dashboards relying heavily on pipeline-computed trends, tracking the aggregate load these recurring, pipeline-heavy requests place on the cluster over time.
Practical Habit
When building a new dashboard panel around a pipeline aggregation, verify the underlying bucket ordering and window configuration against a small, known dataset first, confirming the derived values (running totals, rates of change) match hand-calculated expectations before trusting the panel at scale.
Certain pipeline aggregation patterns show up constantly across well-designed analytics dashboards, and certain shortcuts consistently produce confusing or misleading results.
Established patterns
Running total dashboards
A cumulative sum pipeline aggregation layered over a date histogram, giving stakeholders a clear “total so far” view alongside period-by-period figures.
Trend-smoothing via moving average
A moving-function pipeline aggregation used to reveal an underlying trend obscured by noisy, day-to-day fluctuation in metrics like traffic or error rates.
Finding the standout bucket
A max-bucket or min-bucket sibling pipeline aggregation used to identify which single time period, category, or region had the single highest or lowest value.
Anti-patterns to avoid
Order-Sensitive Pipelines Over Unordered Buckets
Applying a cumulative sum or derivative pipeline aggregation over buckets that were not sorted in a meaningful, consistent order.
Why It Hurts
The resulting running total or rate-of-change values become meaningless or actively misleading, since the calculation silently assumes a sequential relationship between buckets that was never actually guaranteed.
Trying to Reach Document-Level Detail Through a Pipeline
Attempting to use a pipeline aggregation to answer a question that genuinely requires inspecting individual document fields not already captured in the underlying aggregation’s output.
Why It Hurts
A pipeline aggregation has no access to raw documents at all; if the necessary detail wasn’t already produced by the underlying aggregation, no pipeline configuration can retroactively supply it.
A short, practical checklist distilled from how experienced Elasticsearch practitioners use pipeline aggregations well.
| Best Practice | Why It Matters |
|---|---|
| Ensure the underlying bucketed aggregation is sorted meaningfully before applying order-sensitive pipelines. | Running totals and rates of change only make sense over a genuinely ordered sequence of buckets. |
| Double-check buckets path syntax carefully against the underlying aggregation’s actual structure. | A mismatched path is one of the most common sources of pipeline aggregation errors. |
| Choose sliding-window sizes for moving-style pipelines deliberately, based on real data granularity. | A window too small still shows noise; a window too large can smooth away genuinely meaningful shifts. |
| Validate pipeline results against a small, hand-checkable dataset before trusting them at scale. | Catches configuration mistakes early, before they reach a stakeholder-facing dashboard. |
| Consider derived-trend data sensitivity, not just raw document sensitivity, in access-control review. | Pipeline-computed trends can still reveal meaningful patterns about otherwise-restricted underlying data. |
Assuming a pipeline aggregation can somehow “look back” at raw documents if the underlying aggregation didn’t already capture the needed detail. If a value wasn’t produced in the first computation phase, no pipeline configuration in the second phase can retrieve it, since pipeline aggregations have no direct access to the original documents at all.
Pipeline aggregations quietly power many of the trend charts and summary figures found on real-world analytics dashboards.
Revenue and Growth Dashboards (large e-commerce and SaaS platforms)
Business dashboards commonly show both a period-by-period revenue chart and a running cumulative total for the quarter or year, with the cumulative figure powered directly by a cumulative sum pipeline aggregation layered over the underlying period-based revenue aggregation.
Infrastructure Monitoring and Anomaly Detection (Uber-scale observability platforms)
Operations teams monitoring metrics like request latency or error rate commonly rely on moving-average pipeline aggregations to smooth out normal minute-to-minute noise, making genuine, sustained shifts in system behavior easier to spot against a backdrop of naturally fluctuating raw data.
Marketing and Campaign Performance Comparison (large advertising platforms)
Marketing analytics tools frequently use max-bucket and min-bucket sibling pipeline aggregations to quickly surface which specific campaign, region, or time period performed best or worst across a broader reporting period, without requiring a separate manual scan through every individual bucket.
Key Takeaways
- A pipeline aggregation computes a further result from another aggregation’s already-produced output, never reading the original documents directly.
- It comes in two families: parent pipelines, which add a value to each existing bucket, and sibling pipelines, which summarize across a whole set of buckets.
- Common examples include cumulative sums, derivatives, and moving averages, each solving a recurring analytical need that a single-pass, document-scanning aggregation cannot express.
- Execution happens in two phases within one request: the underlying aggregation scans documents first, and the pipeline aggregation performs its calculation afterward, purely on the already-assembled results.
- Order-sensitive pipeline aggregations depend on the underlying buckets already being arranged meaningfully, most commonly in chronological order.
- A pipeline aggregation cannot retrieve document-level detail that wasn’t already captured by the underlying aggregation it depends on.
- Derived, pipeline-computed trends can still reveal sensitive patterns, so access-control review should consider derived analytics alongside raw document visibility.