Before we meet the histogram, let us feel the problem it solves.

Software produces measurements by the million

Every time your web service answers a request, it takes some amount of time. Every time it sends a response, that response has some size. These are measurements, and a busy service creates millions of them every hour. You cannot send all of them to a monitoring tool without huge cost, and you cannot read all of them yourself. You need a summary.

The trap of the average

The most familiar summary is the average. Add everything up, divide by the count, and you get one friendly number. Averages are fine for things that behave evenly, such as the height of adults in a room. But software measurements are rarely even. Most requests are fast, a few are slow, and a very small number are extremely slow. The slow ones are exactly the ones that make customers leave, and the average hides them.

Consider ten requests. Nine take one second and one takes ninety-one seconds. The average is ten seconds, a number that matches no real request at all. Nine people were happy, one person was furious, and the average told you neither story.

The histogram is one tool in a small toolbox, so let us remember the box.

A metric in OpenTelemetry is a measurement of something about your software, tracked over time and summarised so it stays small. Your code does not write metrics directly. Instead, it uses an instrument, an object that records measurements. OpenTelemetry offers a handful of instruments, and each one summarises its measurements in its own way.

The key difference is what each instrument keeps. A counter keeps one running total. A gauge keeps one latest value. A histogram keeps a small table showing how many measurements fell into each range. That table is what lets you answer questions like “how many requests were slower than one second?” long after the individual requests are forgotten.

A plain definition first, then the reasons behind each word.

A histogram in OpenTelemetry is an instrument that records many individual measurements and summarises them as a distribution, which means a count of how many measurements fell into each of several ranges called buckets. Along with the bucket counts, it normally keeps the total number of measurements and the sum of all measured values.

That definition has three important ideas inside it.

Everything about a histogram begins with its buckets.

A bucket is a range of values with a lower edge and an upper edge. When a measurement arrives, the histogram finds the bucket whose range contains it and adds one to that bucket’s count. The measurement itself is then forgotten. Only the count remains.

A worked example

Suppose you track response times in seconds, and you choose these bucket edges: one tenth, half, one, and five seconds. That creates five buckets. The first covers everything up to one tenth of a second. The second covers up to half a second. The third covers up to one second. The fourth covers up to five seconds. The fifth, called the overflow bucket, covers everything above five seconds.

From this tiny table, you can already say a lot. Out of one thousand requests, 620 were very fast, 900 were answered within half a second, 970 within a second, and only five were painfully slow. No average could have told you that.

flowchart LR
  M["New measurement: 0.35 seconds"] --> C{"Which range contains it?"}
  C -->|"Up to 0.1"| B1["Bucket 1"]
  C -->|"0.1 to 0.5"| B2["Bucket 2: add one"]
  C -->|"0.5 to 1"| B3["Bucket 3"]
  C -->|"1 to 5"| B4["Bucket 4"]
  C -->|"Above 5"| B5["Overflow bucket"]
        

The overflow bucket

There is always one more bucket than there are edges. The last bucket catches everything bigger than the largest edge. It is easy to forget, but it matters, because if too many measurements land there, your edges are too small for your data and you are losing detail exactly where problems live.

Where do the edges belong?

By convention in OpenTelemetry, a bucket includes its upper edge but not its lower edge. A measurement exactly equal to a boundary falls into the bucket that ends at that boundary. It is a small detail, but it explains why a value of exactly half a second counts in the second bucket above, not the third.

Every histogram sends a standard bundle of information.

When OpenTelemetry sends a histogram to a Collector or a monitoring tool, each data point carries the same set of parts. Learning them helps you read histograms from any system.

What you can still calculate

Even though individual values are gone, this small bundle lets you work out several useful things. Count over time gives you the rate of events. Sum divided by count gives the average. Bucket counts give you the shape and let you estimate percentiles. Min and max, when present, give you the extremes.

Percentiles are the most common question people ask of a histogram.

A percentile answers the question “what value do this many of my measurements stay below?”. The 50th percentile, also called the median, is the value below which half of all measurements fall. The 95th percentile is the value below which 95 out of every 100 measurements fall. The 99th percentile zooms in on the slowest one in a hundred.

How a tool estimates a percentile

Because the histogram only knows how many measurements landed in each bucket, not their exact values, the tool must estimate. Imagine you want the 95th percentile of one thousand requests. You need the value below which 950 requests fall. The tool adds up bucket counts from the smallest bucket upwards until it reaches 950. It finds that the 950th request lives in a certain bucket, and then guesses its position inside that bucket by assuming the measurements are spread evenly across it.

This means the answer is an estimate, not an exact value. How good the estimate is depends on how narrow the bucket is. A wide bucket means a rough answer. A narrow bucket means a precise one.

This is the quiet reason histograms are the right tool for large systems.

Imagine you run twenty copies of the same service. Each copy measures its own response times. Now you want the 95th percentile across all twenty. You might be tempted to calculate the 95th percentile on each copy and then average those twenty numbers. This gives a wrong answer. Percentiles cannot be averaged. A busy copy and a quiet copy contribute very differently to the overall picture, and averaging their percentiles ignores that.

Histograms solve this beautifully. Because each copy reports bucket counts, you can simply add the counts together, bucket by bucket. The result is a single histogram describing all twenty copies as if they were one service. From that combined histogram, you can then estimate a correct overall percentile.

flowchart TD
  A["Service copy 1: bucket counts"] --> S["Add counts bucket by bucket"]
  B["Service copy 2: bucket counts"] --> S
  C["Service copy 3: bucket counts"] --> S
  S --> H["One combined histogram"]
  H --> P["Estimate the true overall percentile"]
        

The classic kind, where you decide the edges in advance.

In an explicit bucket histogram, the edges between buckets are a list of numbers chosen before any measurement arrives. The SDK ships with a default list that is spread across a wide range, from very small values up to ten thousand. This default is a general-purpose guess that works tolerably for many cases, but it was designed with values like milliseconds in mind.

The unit problem

Here is a common trap. OpenTelemetry’s semantic conventions recommend measuring durations in seconds, so a typical web request might take 0.045 seconds. If you record that in seconds but keep the default edges, which start at zero, five, ten and so on, nearly every measurement falls into the very first bucket, and the histogram becomes useless. The edges must match the unit and the typical size of your data.

Choosing good edges

Good edges follow three simple rules. First, cover the full realistic range of your data, including a few edges beyond the worst case you can imagine. Second, place more edges where you care most about precision. Third, put an edge exactly at any goal or threshold you care about, such as “responses should be under 300 milliseconds”, so that the count under that line is exact, not estimated.

A smarter kind that chooses its own edges.

Choosing edges by hand is a nuisance, and a wrong choice hurts. OpenTelemetry therefore offers a second kind, the exponential histogram. Instead of a list of fixed edges, it uses a rule: each bucket is a fixed percentage wider than the one before it. Small values get tiny buckets and large values get bigger buckets, which is exactly what most real data needs, because the difference between one and two milliseconds matters far more than the difference between nine thousand and nine thousand and one.

The idea of scale

The rule is controlled by a single number called the scale. A higher scale makes the buckets narrower and more numerous, giving better precision. A lower scale makes the buckets wider and fewer. The clever part is that the histogram adjusts its own scale automatically. It starts with high precision, and if measurements spread across such a wide range that it would need too many buckets, it quietly lowers its scale to keep the number of buckets under a limit. You get good accuracy without ever choosing an edge.

Other pieces

An exponential histogram also keeps a separate count for measurements that are exactly zero, since zero cannot be placed on a growth-based scale, and can keep separate buckets for positive and negative values.

A short tour of what happens between a measurement and a dashboard.

You do not need to master the internals, but knowing the steps makes the whole subject less mysterious. A histogram is a synchronous instrument, which means your code records a value at the moment something happens. For example, when a request finishes, the code notes how long it took and hands that single number to the histogram.

flowchart LR
  R["Request finishes: 0.35 s"] --> H["Histogram instrument"]
  H --> AG["SDK adds to bucket, count and sum"]
  AG --> RD["Metric reader collects each interval"]
  RD --> EX["Exporter sends over OTLP"]
  EX --> CO["Collector"]
  CO --> BE["Backend: percentiles and heatmaps"]
        

What to record in a histogram

Histograms suit any measurement that is naturally a size or an amount of time and where the spread matters. Typical examples include the duration of web requests, the duration of database queries, the size of messages or responses, the time a job waits in a queue, and the number of items processed per batch.

You can tune histograms from the outside.

In OpenTelemetry, a view is a configuration rule that changes how a metric is summarised, without touching the code that records it. Views are especially valuable for histograms, because developers who write instrumentation inside a library cannot always know the best bucket edges for your particular system.

Advice from the instrument creator

Instrument authors can also suggest sensible bucket edges when they create a histogram, as a hint. If a view exists, the view wins. If not, the hint is used. If there is neither, the SDK falls back to its general-purpose default. This layering lets library writers offer sensible defaults while still leaving final control to the application owner.

Histograms can be reported as running totals or as changes.

Like counters, histograms can be reported in two styles. A cumulative histogram reports bucket counts since the program started, so the numbers only ever grow. A delta histogram reports only what was recorded since the last report, so each report stands alone. The choice is called temporality.

Many open-source monitoring tools prefer cumulative reports, since a missed report is easily recovered. Some commercial platforms prefer delta reports, which can use less memory inside your application. Which one is right depends on your backend. A mismatch leads to strange charts, so choose the style your destination expects, or let a Collector convert between them.

Why this matters for percentiles

When you ask “what was the 95th percentile over the last five minutes?”, the tool must work on the difference between two cumulative snapshots, or on the sum of several delta reports. Either way, the underlying idea is the same: adding and subtracting bucket counts. This is possible only because histograms keep counts per bucket.

The extra detail of a histogram has a price.

A counter produces one number per combination of attributes. A histogram produces one number for each bucket, plus the count and the sum, per combination of attributes. With fifteen buckets, a single histogram series can be roughly seventeen numbers where a counter would have one. Multiply that by the number of attribute combinations, and the cost grows quickly.

The multiplication problem

Imagine a histogram of request duration with attributes for route, method, status class and region. If there are twenty routes, four methods, three status classes and five regions, you already have twelve hundred combinations. With seventeen numbers each, that is over twenty thousand numbers every collection interval. Add a high-cardinality attribute such as a user identifier, and the system can drown.

Practical ways to control cost

A histogram can point to real examples hiding in its buckets.

A histogram tells you that fifty requests last week fell in the slowest bucket. But which requests? Why were they slow? The histogram, by design, has forgotten the details. Exemplars are a clever bridge. An exemplar is a small sample attached to a histogram bucket that records one real measurement together with the identifier of the trace it came from.

With exemplars, a chart can show dots over the histogram, each dot representing a real request. Click a dot in the slow area, and you jump straight to the full trace of that request. You move from a summary to a concrete story in one step.

flowchart LR
  A["Alert: slow requests increasing"] --> B["Look at the histogram: slow bucket growing"]
  B --> C["Open an exemplar from that bucket"]
  C --> D["View the full trace of that request"]
  D --> E["Find the slow step and read its logs"]
        

Histograms are the natural building block for service level goals.

Many teams promise their customers something like “ninety-five percent of requests will finish in under 300 milliseconds”. That sentence is really a statement about a distribution, and a histogram is the perfect tool to check it. If you place a bucket edge at 300 milliseconds, the histogram can say exactly how many requests were at or under that line, and how many were over it.

Two ways to express the same goal

Alerting on user pain

Good alerts focus on what users feel. An alert such as “the share of requests slower than one second has exceeded five percent for ten minutes” is easy to understand, hard to ignore, and does not fire for harmless blips. A histogram makes this kind of alert possible, whereas an average-based alert would stay silent until things were already badly broken.

What happens after the histogram leaves your application?

Histograms travel from the application to a Collector using the OpenTelemetry Protocol, and then onward to a monitoring backend. Along the way, a few things are worth knowing.

Histograms are summaries, but their labels still deserve care.

A histogram reveals little about individual users, because it holds only counts per range. The risk comes from the attributes you attach to it, and from how widely the data is shared.

Habits that keep histograms useful and affordable.

Good habits

Mistakes to avoid

Histograms are powerful, but they are not free.

Choosing between summaries

Picture how different teams use histograms every day.

You do not need to learn everything at once.

It is an instrument that sorts many measurements, such as response times, into ranges called buckets and counts how many fell into each. It also keeps the total count and sum, so it preserves the shape of your data in a very small package.

Because averages hide outliers. A few very slow requests barely move the average, yet they are the ones that frustrate users. A histogram shows how many measurements were slow, so those problems become visible.

A bucket is a range of values with a lower and upper edge. Each measurement goes into the bucket whose range contains it, and only the bucket’s count is kept. There is always an extra overflow bucket for values above the largest edge.

The tool adds up bucket counts from the smallest bucket upwards until it reaches the desired share of the total, then estimates a position inside that bucket. The answer is therefore an estimate, and it is more accurate when buckets are narrow.

No. Percentiles cannot be averaged meaningfully. Instead, add the bucket counts of all servers together and estimate the percentile from the combined histogram.

In an explicit histogram, you choose the bucket edges in advance. In an exponential histogram, the edges follow a growth rule and adjust automatically to your data. Exponential histograms need little tuning, but not every backend supports them yet.

The most likely cause is a mismatch between your unit and your bucket edges, for example recording seconds with edges designed for milliseconds. Adjust the edges using a view, or switch to exponential histograms.

The OpenTelemetry semantic conventions recommend seconds, written as decimal numbers. Whatever unit you choose, set it clearly on the metric and make sure your bucket edges match.

A histogram stores a number per bucket plus a count and a sum for each combination of attributes, while a counter stores one. More buckets and more attribute combinations multiply the storage and query cost.

An exemplar is a small sample attached to a histogram bucket that records a real measurement together with its trace identifier. It lets you jump from a slow bucket straight to the trace of an actual slow request.

Yes. Views are configuration rules that can change bucket edges, switch the histogram type, remove attributes, or rename and drop metrics, all without editing the code that records the measurements.

No. Use histograms for the measurements where the spread truly matters, such as user-facing request times. For minor or secondary measurements, a simple counter or gauge may be enough and is cheaper to store.

A histogram in OpenTelemetry is the instrument that keeps the shape of your measurements. Instead of remembering every value, it sorts each one into a bucket and keeps the count per bucket along with the total count and the sum. That small table lets you estimate percentiles, compute averages and rates, check goals at exact thresholds, and combine data from many instances correctly by adding bucket counts. Explicit histograms use edges that you choose, so the edges must match your unit and your range of values. Exponential histograms choose their own edges and adapt automatically, as long as your backend supports them. Views let you tune histograms without touching code, exemplars connect buckets to real traces, and careful control of attributes keeps costs under control. Used well, histograms reveal the unhappy few that averages hide.