The Percentiles Aggregation in Elasticsearch
Why "average response time" often lies to you, and how percentiles reveal the truth about what most of your users actually experience.
Imagine a hundred people running a race, and ninety-nine of them finish in under ten minutes, while one person, injured mid-race, limps in after two full hours. If you calculate the average finish time across all hundred runners, that one outlier drags the “average” up to something like eleven minutes — a number that doesn’t represent a single real runner’s actual experience. Percentiles solve exactly this problem by describing the distribution of values directly, rather than collapsing everything into one easily-distorted number, and the percentiles aggregation is how Elasticsearch computes this kind of distribution-aware statistic across massive datasets.
1What Is a Percentiles Aggregation?
Let’s build a clear, intuitive understanding of what a percentile actually means before looking at the aggregation itself.
A percentile is a value below which a given percentage of observations fall. Saying “the 95th percentile of response times is 400 milliseconds” means that 95 percent of all recorded response times were 400 milliseconds or faster, and only the slowest 5 percent took longer. The percentiles aggregation in Elasticsearch is a built-in tool that calculates several of these percentile values — commonly the 1st, 5th, 25th, 50th, 75th, 95th, and 99th — across a numeric field for all the documents matching a query, all in a single, efficient request.
Think of standing all your website’s visitors in a single line, ordered from the one who experienced the fastest page load to the one who experienced the slowest. The 50th percentile is simply the person standing exactly in the middle of that line. The 95th percentile is the person standing 95 percent of the way along it — meaning 95 out of every 100 visitors had a faster experience than that one person did.
Median (50th Percentile)
The value exactly in the middle of a sorted dataset — half of all values are below it, half are above it.
Tail Latency
The slower end of a distribution, typically described using high percentiles like the 95th or 99th, representing the worst experiences a meaningful share of users actually have.
Bucket Aggregation
An Elasticsearch aggregation that groups documents (such as by day or by category), often used together with percentiles to see distributions per group.
Metric Aggregation
The broader family of aggregations, including percentiles, that compute a numeric summary statistic from a field, rather than grouping documents.
A practical example: imagine an e-commerce site monitoring how long checkout takes. If the average checkout duration looks perfectly reasonable, that number alone could still be hiding a real problem: perhaps the median checkout is genuinely fast, but the 95th percentile reveals that a meaningful chunk of customers are experiencing checkout delays several times longer than typical. Only the percentile view exposes this kind of hidden inequality in user experience — precisely the reason this aggregation exists and is used so widely for performance and quality monitoring.
The percentiles aggregation returns multiple percentile values in one response by default, so you get a fuller picture of the distribution’s shape rather than needing to run several separate queries for each percentile you’re interested in.
2Architecture and Components
Computing accurate percentiles across a huge, distributed dataset is a genuinely hard problem — let’s look at how Elasticsearch’s architecture handles it.
graph TD
A["Percentiles Aggregation Request"] --> B["Target Numeric Field"]
A --> C["Requested Percentile Values (e.g. 50, 95, 99)"]
A --> D["Algorithm: TDigest (default) or HDR Histogram"]
D --> E["Per-Shard Approximate Sketch"]
E --> F["Merged Cluster-Wide Sketch"]
F --> G["Final Percentile Values Returned"]
Component: The Target Field
This is the numeric field whose distribution you want to understand, such as a response time, a transaction amount, or a page load duration.
Component: The Percentile List
The specific percentile values you want computed, which can be customized freely — some use cases care mostly about the median and the 99th percentile, others want a fuller spread across many percentile points.
Component: The Underlying Algorithm
Because exact percentile calculation across billions of values spread over many shards is prohibitively expensive, Elasticsearch uses approximate algorithms — TDigest by default, with HDR Histogram available as an alternative — each with different trade-offs covered in the next chapter.
This architecture explains an important practical reality: percentile results in Elasticsearch are approximate, not mathematically exact, by design. This is a deliberate trade-off, sacrificing a small, usually negligible amount of precision in exchange for being able to compute meaningful percentile statistics across enormous datasets in a fraction of a second, without ever needing to sort or fully scan every single matching document.
3Internal Working — How Approximate Percentiles Are Calculated
Understanding the two underlying algorithms explains both why percentiles are approximate and how to configure them for the accuracy your use case actually needs.
The default algorithm, TDigest, works by building a compact, sketch-like summary of a data distribution rather than storing every single value. It does this cleverly by keeping finer-grained detail near the extreme ends of the distribution (the very low and very high values) and coarser detail in the middle, since the tails — the 95th, 99th percentiles, and beyond — are usually what people care most about getting right, while the middle of the distribution can tolerate slightly more approximation without much practical impact.
flowchart LR
A["Thousands of raw values"] --> B["TDigest Sketch: fine detail at the tails"]
B --> C["Compact summary, not the full dataset"]
C --> D["Percentile values estimated from the sketch"]
Think of TDigest as a museum curator organizing thousands of paintings by value. Rather than cataloguing every single piece with equal detail, the curator pays extremely close attention to the rarest, most valuable and least valuable pieces at either extreme, while grouping the vast middle range of “ordinary” paintings more loosely — because for most purposes, precisely ranking two nearly-identical mid-range paintings matters far less than correctly identifying the true extremes.
The alternative algorithm, HDR Histogram (High Dynamic Range Histogram), takes a different approach: it pre-allocates a fixed set of numeric buckets across the expected range of values and tracks counts within each bucket, offering very fast, memory-predictable performance and a guaranteed maximum error bound, at the cost of needing to know a reasonable expected value range in advance and sometimes using more memory than TDigest for very wide-ranging or unpredictable data.
| Aspect | TDigest (Default) | HDR Histogram |
|---|---|---|
| Accuracy at extreme percentiles | Very good, adaptively concentrated there | Consistent, bounded error regardless of percentile |
| Memory usage | Generally lower, scales with data shape | Fixed, based on configured precision and value range |
| Needs value range in advance | No | Yes, benefits from a known expected range |
| Typical use case | General-purpose latency and metric analysis | High-precision, performance-critical monitoring with known value ranges |
Because results are approximate, it’s tempting to assume percentile results are unreliable. In practice, for the vast majority of monitoring and analytics use cases, the approximation error is small enough to be practically invisible, especially at the tail percentiles most people actually care about.
4Data Flow and Lifecycle
Let’s trace exactly what happens across a distributed cluster when a percentiles aggregation request is executed.
Aggregation Request Submitted
A search request including a percentiles aggregation is sent, specifying the target numeric field and, optionally, the specific percentile values and algorithm to use.
Per-Shard Local Computation
Each shard holding matching documents independently builds its own local sketch (TDigest or HDR Histogram) from the relevant field’s values in the documents it holds.
Sketch Transmission
Rather than sending every raw value back, each shard sends only its compact sketch to the coordinating node, dramatically reducing the amount of data that needs to move across the network.
Sketch Merging
The coordinating node merges all the individual per-shard sketches into a single, combined cluster-wide sketch representing the entire matching dataset.
Final Percentile Extraction
The requested percentile values are calculated from this merged sketch and returned to the client as the final aggregation result.
This shard-then-merge design is precisely why percentiles aggregations remain fast even across enormous, multi-shard indices — no single node ever needs to hold or sort the entire dataset’s raw values at once. The trade-off, as covered previously, is that merging independently-built approximate sketches introduces a small amount of additional imprecision compared to a theoretical exact calculation over perfectly centralized data.
5Percentiles vs. the Average Aggregation
The average is often the very first statistic people reach for, so it’s worth a direct, honest comparison against percentiles.
| Aspect | Average | Percentiles |
|---|---|---|
| Sensitivity to outliers | High — a few extreme values can significantly shift it | Low for most percentiles — outliers mainly affect only the highest percentiles |
| Describes distribution shape | No — reduces everything to a single number | Yes — multiple points reveal the shape of the data |
| Represents a “typical” experience | Can be misleading with skewed data | The median offers a genuinely representative typical value |
| Reveals worst-case experience | No visibility into this at all | High percentiles (95th, 99th) directly reveal it |
| Computational cost | Very low, simple running sum and count | Higher, but still efficient thanks to approximate sketches |
The practical takeaway is not that averages are useless — for genuinely evenly-distributed data, an average and the median can be quite close, and averages remain computationally simpler and perfectly fine for many purposes. The key is recognizing when a metric matters specifically because of its distribution’s shape — response times, transaction sizes, page load durations — situations where a handful of slow or extreme cases genuinely matter and shouldn’t be smoothed away by a single averaged number.
6Advantages, Disadvantages, and Trade-offs
Percentiles offer a far more honest picture of a distribution than a single average, but that honesty comes with its own considerations.
Advantages
- Reveals the true shape of a distribution, including outliers, rather than hiding them inside a single averaged number.
- Scales efficiently to massive, distributed datasets using compact, mergeable approximate sketches.
- Returns multiple percentile values in a single request, giving a complete picture without repeated queries.
- Directly supports common, industry-standard performance metrics like “95th percentile latency” or “99th percentile response time.”
- Works naturally alongside bucket aggregations, enabling percentile breakdowns per category, time period, or any other grouping.
Disadvantages / Trade-offs
- Results are approximate rather than mathematically exact, which can matter in edge cases requiring guaranteed precision.
- Choosing between TDigest and HDR Histogram requires understanding your data’s characteristics and accuracy needs.
- Percentiles alone don’t reveal the total count or sum behind the distribution, often requiring complementary aggregations for full context.
- Interpreting percentile results correctly requires more statistical literacy than reading a single average value.
For the overwhelming majority of monitoring, performance analysis, and business analytics use cases, these trade-offs are well worth accepting — the small approximation error is a minor cost compared to the genuine insight gained by seeing a distribution’s actual shape instead of a single, potentially misleading average.
7Security Considerations
Percentiles aggregations are read-only statistical calculations, so their direct security surface is minimal, but a few considerations still matter in practice.
Aggregating and exposing percentile statistics over sensitive numeric fields — such as individual salary figures, personal transaction amounts, or health metrics — even without exposing raw individual documents, can sometimes still leak meaningful information about the underlying data’s distribution to an unauthorized viewer.
Mitigation: Apply the Same Access Controls to Aggregations as to Raw Data
Percentiles aggregations respect existing field-level and document-level security, so ensure sensitive fields are properly restricted at the mapping and role level, not just assumed to be safe because only a summary statistic is being returned.
Mitigation: Be Cautious With Very Small Result Sets
Percentile statistics computed over a very small number of documents can sometimes allow an observer to infer individual values with reasonable confidence, especially at extreme percentiles; consider minimum document-count thresholds for exposing aggregation results in sensitive contexts.
Mitigation: Limit Public Exposure of Business-Sensitive Aggregations
Distribution statistics about pricing, performance, or transaction volumes can sometimes reveal competitively sensitive business information, so consider who should have access to percentile-based dashboards and reports before making them broadly available.
In short, the percentiles aggregation itself introduces no new mechanism for unauthorized data access, but it’s a reminder that aggregated statistics still deserve the same thoughtful access-control consideration as the raw underlying data they’re derived from.
8Monitoring, Logging, and Metrics
Percentiles aggregations are themselves a monitoring tool, but it’s worth also monitoring their own performance and behavior at scale.
Aggregation Query Latency
Tracking how long percentiles aggregation requests take helps catch performance issues, especially on very high-cardinality fields or extremely large result sets.
Memory Usage for HDR Histogram Sketches
If using HDR Histogram, monitoring memory consumption helps ensure the configured precision and value range settings remain appropriate as data grows.
Tracked Percentile Trends Over Time
Regularly recording and graphing key percentiles (like the 95th) over time is itself one of the most valuable monitoring practices for catching gradual performance regressions.
Circuit Breaker or Slow Aggregation Warnings
Repeated warnings tied to percentile aggregation queries can indicate the need for query scoping, sampling, or algorithm tuning to keep response times manageable.
A widely adopted monitoring pattern is tracking multiple percentiles side by side over time — commonly the median alongside the 95th and 99th — since watching how the gap between them changes reveals whether a performance problem is affecting everyone equally or specifically worsening the experience of a smaller, unlucky subset of users.
sequenceDiagram
participant App
participant ES as Elasticsearch
participant Dash as Dashboard
App->>ES: percentiles aggregation on response_time
ES-->>App: p50, p95, p99 values
App-->>Dash: Log and plot percentile trends
Dash->>Dash: Alert if p95/p99 gap widens unexpectedly
9Design Patterns and Anti-patterns
A handful of recurring, effective ways teams use percentiles aggregations well, and mistakes worth avoiding.
Pattern: Combine With Date Histogram for Trend Analysis
Nesting a percentiles aggregation inside a date histogram bucket aggregation reveals how key percentiles change over time, making it easy to spot gradual performance regressions or sudden spikes.
Pattern: Track the Median Alongside a High Percentile
Reporting the median together with the 95th or 99th percentile gives both a “typical experience” number and a “worst realistic experience” number, offering much more actionable insight than either figure alone.
Problem
Relying solely on the average of a metric like response time or transaction value to judge overall system health or user experience.
Why It’s Harmful
A small number of extreme outliers can distort the average significantly, either hiding a genuine widespread problem or falsely suggesting one that doesn’t actually reflect most users’ experience.
Correct Approach
Track percentiles alongside or instead of the average, particularly for metrics where distribution shape and worst-case behavior genuinely matter.
Problem
Treating percentile results as mathematically exact figures, especially when reporting them externally or using them for precise financial or contractual purposes.
Why It’s Harmful
The underlying TDigest or HDR Histogram algorithms are approximate by design, and small discrepancies from an exact calculation are expected and normal.
Correct Approach
Use percentile aggregations for monitoring, analysis, and general reporting where small approximation is acceptable, and clearly document their approximate nature wherever precision genuinely matters.
As with the other aggregations and analyzers covered throughout this series, the underlying theme is understanding what a tool is optimized for — here, fast, scalable, distribution-aware insight — and applying it accordingly, rather than treating it as a drop-in, perfectly precise replacement for a simple average.
10Best Practices and Common Mistakes
A concentrated, practical checklist for using percentiles aggregations effectively in real dashboards and analysis.
Report percentiles alongside, not instead of, counts
A 99th percentile of 2 seconds means something very different depending on whether it’s drawn from ten requests or ten million.
Choose HDR Histogram when a known range and bounded error matter
It’s a strong fit for performance-critical monitoring where a predictable, guaranteed accuracy level is important.
Track tail percentiles for user-facing performance metrics
The 95th and 99th percentiles often reveal real problems that averages and even the median can completely miss.
Rely exclusively on the average for skewed data
Response times, transaction amounts, and similar metrics are frequently skewed enough that averages alone are misleading.
Treat percentile output as exact for critical decisions
Remember the underlying calculation is approximate, and factor that into any decision where precision genuinely matters.
Ignore how bucket size affects percentile granularity
When combining with a date histogram or other bucketing, very small buckets can produce percentiles based on too few documents to be meaningful.
When presenting performance metrics to stakeholders unfamiliar with percentiles, pairing the number with a plain-language explanation — “95 out of 100 users experienced this or better” — usually communicates the insight far more effectively than the raw percentile figure alone.
11Real-World and Industry Examples
Percentiles aggregations are a staple across performance monitoring, analytics, and business intelligence use cases built on Elasticsearch.
Application Performance Monitoring
Engineering teams routinely track 95th and 99th percentile response times as core service-level indicators, since these tail metrics directly reflect the worst experiences a meaningful share of real users actually encounter.
E-Commerce Pricing and Order Analysis
Retail analytics teams use percentiles on order values or basket sizes to understand typical customer spend without letting a handful of unusually large bulk orders distort the overall picture.
Infrastructure and Network Monitoring
Percentiles on metrics like network latency or disk I/O times help operations teams distinguish between a system that’s uniformly slow versus one that’s mostly fine but occasionally experiences severe, isolated slowdowns.
Financial and Transaction Systems
Percentile analysis on transaction processing times or amounts helps financial platforms understand normal operating ranges and flag unusually extreme values worth further investigation.
Across all these examples, the pattern is consistent: whenever “what’s typical” and “what’s the worst realistic case” are both genuinely important questions, percentiles provide a far richer and more honest answer than any single averaged number ever could.
12Frequently Asked Questions
Yes, by design, since both underlying algorithms (TDigest and HDR Histogram) trade a small amount of precision for the ability to scale efficiently across massive, distributed datasets.
A default set is returned, typically including the 1st, 5th, 25th, 50th, 75th, 95th, and 99th percentiles, though a specific custom list can always be requested instead.
The median (50th percentile) is the true middle value of a sorted dataset, unaffected by extreme outliers, while the average can be pulled significantly higher or lower by a small number of unusually large or small values.
HDR Histogram is a good fit when you know your data’s expected value range in advance and need a guaranteed, bounded accuracy level, particularly for high-precision performance monitoring.
Yes, nesting a percentiles aggregation inside a bucket aggregation like terms or date histogram computes separate percentile values for each individual group.
Small variations can occur due to the approximate nature of the underlying algorithms and how sketches are merged across shards, though these differences are typically negligible for practical purposes.
It’s generally efficient thanks to the compact, mergeable sketch design, though very high-cardinality queries or extremely fine-grained bucketing can still increase computational cost, which is worth monitoring at scale.
13Summary and Key Takeaways
The percentiles aggregation gives Elasticsearch users a far more honest window into a dataset’s true distribution than a single average ever could, revealing both what’s typical for most observations and what the worst realistic experience actually looks like for the unluckiest fraction of them. It achieves this at massive scale by relying on compact, approximate sketch algorithms — TDigest by default, or HDR Histogram when a bounded, predictable accuracy is required — that can be efficiently computed per shard and merged cluster-wide without ever needing to sort or centralize every raw value. Because the underlying calculation trades a small amount of precision for enormous scalability, using percentiles well means understanding that trade-off, choosing the right algorithm for your accuracy needs, and pairing percentile figures with document counts and complementary metrics for the fullest, most accurate picture of your data.
Key Takeaways
- Percentiles reveal distribution shape — they show what’s typical and what’s extreme, unlike a single average that hides both.
- Results are approximate by design — TDigest and HDR Histogram trade a small amount of precision for massive scalability.
- TDigest is the default, concentrating accuracy at the tails — ideal for general-purpose latency and metric analysis.
- HDR Histogram offers bounded, predictable accuracy — a strong fit when the expected value range is known in advance.
- Averages can be dangerously misleading for skewed data — a few extreme outliers can distort them significantly.
- Track high percentiles for user-facing performance — the 95th and 99th percentiles often reveal problems averages completely miss.
- Combine with bucket aggregations for deeper insight — pairing percentiles with grouping by time or category reveals trends and patterns a single overall figure cannot.