The gap between a dashboard and an investigation.

When something goes wrong in a production system, engineers usually begin with a dashboard. A chart shows that the ninety-ninth percentile of response time has jumped from 300 milliseconds to 2 seconds. This is valuable, because it tells you that a problem exists, how big it is, and when it started. But it cannot tell you which requests were slow, which service was responsible, or what those requests had in common.

To answer those questions, engineers switch to a tracing tool and start searching. They guess a time range, filter by a service name, sort by duration, and hope that the slow trace they find is related to the spike they saw on the chart. This hunt takes time, and the connection between the chart and the trace is a guess, not a fact. Under pressure during an incident, every minute of guessing is expensive.

The building blocks you need before exemplars make sense.

Why the two signals do not meet by default

Metrics and traces are produced by separate parts of the SDK and stored in separate systems. A metric data point is the result of squashing thousands of measurements together, and in that squashing, any information about individual requests disappears. The data point for “requests slower than one second: 40” does not remember which forty requests they were.

That loss is deliberate. It is what makes metrics cheap. But it creates exactly the gap described in Chapter 1, and exemplars are the standard way OpenTelemetry bridges it without giving up the savings.

A sample measurement that remembers where it came from.

In the OpenTelemetry data model, an exemplar is a recorded example of an input measurement that was folded into a metric data point. Instead of only keeping the totals, the SDK keeps a small number of real measurements, each one tagged with enough detail to find the request that produced it. Exemplars are attached directly to the data point they illustrate and travel with it to the backend.

The question every newcomer asks, and the reason exemplars exist.

At first glance there is a simpler idea. Why not put the trace identifier on the metric itself as an attribute, so every data point knows its trace? The answer is cardinality. Each unique combination of metric attributes creates a separate time series, and every request has a unique trace identifier. A system with a million requests per hour would create a million series per hour, each holding a single measurement. Memory, storage, and query costs would explode, and the entire point of metrics, which is to be a cheap summary, would be destroyed.

Exemplars avoid that trap by keeping the trace identifier outside the series identity. The series is defined only by the normal low-cardinality attributes, such as route and status class. The exemplar rides along as extra payload on a data point, and only a handful are kept for each data point, no matter how many requests were recorded.

From one measurement to a stamped sample on a data point.

flowchart LR
  A["Code records a measurement"] --> B["Aggregation updates the data point"]
  A --> C{"Exemplar filter says yes?"}
  C -->|No| D["Measurement used only for the aggregate"]
  C -->|Yes| E["Exemplar reservoir offered the measurement"]
  E --> F["Reservoir keeps or replaces a sample"]
  F --> G["Collection time: reservoir returns exemplars"]
  G --> H["Exemplars attached to the data point and exported"]
        

What happens when the measurement has no span

Plenty of measurements are recorded outside any request: a scheduled cleanup job, a background thread that refreshes a cache, or a startup routine. With the trace-based filter, such measurements are simply not eligible, because there is no sampled span in the context. Nothing breaks and no error is raised. The aggregate still counts the value, and the data point simply has fewer or no exemplars. This is a feature, not a flaw, since an exemplar with no trace behind it would be a link to nowhere.

It also means that the quality of your exemplars depends on how consistently your code records metrics inside the request path. If one team records latency inside the handler and another records it later in a callback after the span has ended, the first team gets working links and the second does not, even though both use the same SDK and the same settings.

The first gate in the pipeline, and the most important setting.

Not every measurement should become an exemplar. The exemplar filter is a rule that decides which measurements are eligible. It is the first gate, and its setting has a large effect on both cost and usefulness. The specification defines three standard filters, and SDKs usually let you choose among them with configuration or an environment variable.

Why trace based is the sensible default

An exemplar is only useful if clicking it leads to a trace. A trace exists in your backend only if its span was sampled. The trace-based filter ties the two together at the moment of recording: it admits a measurement only when the current span is sampled, which makes it likely that the link will lead somewhere. This is the reason exemplars and sampling are so closely related, a topic we return to in Chapter 12.

A tiny container with a strict rule about who stays.

Even after the filter, a busy service may record thousands of eligible measurements per collection interval, and it is impossible to keep them all. The reservoir is the component that decides which few to keep. Each time series has its own reservoir, and the reservoir has a fixed capacity, so the memory used for exemplars is bounded no matter how heavy the traffic is.

The defaults

SDKs choose a sensible reservoir for each kind of metric. Histograms with explicit buckets commonly use the bucket-aligned kind, so that every bucket can offer its own example. Other metrics use the fixed-size random kind with a small capacity. The details are SDK specific, so treat the principle as the lesson: capacity is small, and the rule for who stays is simple.

Reservoirs reset between collections

The samples held by a reservoir belong to a specific collection interval. After they are exported, the reservoir starts afresh, so the exemplars on a data point describe measurements from that period rather than from the whole life of the process. This keeps exemplars fresh and relevant to the time range you are looking at on a chart.

How big should the reservoir be

A larger reservoir keeps more examples per interval, which raises the chance that a rare bucket has at least one, but it also increases the data sent with every export. For most services the defaults are a sensible balance. Raise the capacity only when you have evidence that important buckets are regularly empty and you can afford the extra bytes. Lowering it makes sense when exemplars are rarely clicked and bandwidth to the backend is tight.

Think of it as a camera roll with a fixed number of exposures. Fewer exposures mean you must choose carefully, and more exposures mean more film to carry and develop. No setting makes the roll unlimited, and that fixed ceiling is exactly what keeps the feature cheap.

One minute of a checkout service, seen through a latency histogram.

Suppose a checkout service records the duration of each request in a histogram with buckets at 100 milliseconds, 250 milliseconds, 500 milliseconds, one second, and everything above one second. In one minute it handles ten thousand requests. The service samples ten percent of its traces, and the exemplar filter is the trace-based one. The histogram uses a bucket-aligned reservoir, so each bucket may hold one exemplar.

The surprise in the last row

The top bucket holds only ten requests, and the trace-based filter admits only those whose trace was sampled. With a ten percent sampling rate, the chance that none of the ten was sampled is about 0.9 multiplied by itself ten times, which is roughly thirty-five percent. So there is only about a sixty-five percent chance that the slowest bucket has an exemplar at all. The rarest and most interesting requests are exactly the ones most likely to be missing a link.

This is not a flaw in exemplars. It is a direct consequence of combining rare events with low sampling. It tells you that if you care about finding the worst requests, your sampling strategy must favour them, for instance by keeping all slow or failed traces, a point we develop later.

Exemplars attach to metrics that summarise individual measurements.

The data model allows exemplars on several kinds of data points. They make the most sense on metrics where individual measurements are combined into a summary, because those are the cases where you want to see a representative example. Where the metric already is a single raw reading, there is less to illustrate.

Asynchronous instruments are a special case

Observable instruments report values from a callback at collection time rather than at the moment something happens in a request. Because there is usually no active request span during that callback, there is no meaningful trace to link to. Exemplars are therefore mostly a feature of synchronous instruments, recorded inside the code path of a real request.

How exemplars travel from the SDK to your tools.

Exemplars are part of the OpenTelemetry Protocol, usually called OTLP. Each metric data point in the protocol has a place for a list of exemplars, and each exemplar has fields for the value, the time, the filtered attributes, the trace identifier, and the span identifier. When an SDK exports metrics using OTLP, the exemplars simply ride along inside each data point, and a receiver that understands them can read them back out.

flowchart LR
  A["Application SDK with exemplars"] --> B["OTLP export"]
  B --> C["Collector receiver"]
  C --> D["Processors"]
  D --> E["Metrics exporter"]
  E --> F["Metrics backend stores exemplars"]
  A --> G["Trace export"]
  G --> H["Collector then trace backend"]
  F -.->|"Trace ID link"| H
        

Metrics and traces are still stored apart

An important point that surprises beginners is that exemplars do not move traces into the metrics system. The metrics backend only receives the identifiers and a few values. The traces themselves go to a tracing backend through the trace pipeline. The link between the two works because both backends know the same trace identifier, and a viewing tool can use that identifier to open the right trace.

What the Collector does with exemplars

The OpenTelemetry Collector passes exemplars through by default as part of the metric data. Some exporters convert them into the format of the target system. For example, exporters that feed Prometheus-style systems can translate exemplars into the exemplar format of that ecosystem. Some processors that reshape or aggregate metrics may drop or alter exemplars, because merging data points can make individual samples meaningless. Whenever you add a processor that modifies metrics, check whether exemplars survive it.

Where the link becomes something an engineer can actually click.

The final piece of the story is the visualisation tool. A common arrangement uses a Prometheus-compatible store for metrics, a tracing backend for traces, and a dashboard tool that talks to both. On a latency chart, the dashboard draws small markers over the line, each representing an exemplar. Hovering over a marker shows its value and time, and clicking it opens the linked trace in the tracing backend.

sequenceDiagram
  participant E as Engineer
  participant D as Dashboard
  participant M as Metrics store
  participant T as Trace store
  E->>D: Open latency chart for the last hour
  D->>M: Query latency series with exemplars
  M-->>D: Series plus exemplar markers
  E->>D: Click a marker above the spike
  D->>T: Open trace using the exemplar trace ID
  T-->>D: Full trace of that slow request
  D-->>E: Waterfall view showing where time was spent
        

What the backend must support

Not every metrics store keeps exemplars. Prometheus, for example, can store them, but it needs the feature to be enabled, and it keeps them in a bounded in-memory buffer rather than forever. It also limits how much text an exemplar’s attached labels may hold, which affects how many filtered attributes survive. Other backends and vendor platforms have their own rules. Before you rely on exemplars, read the documentation of your metrics store to learn whether it stores them, for how long, and with what limits.

What the dashboard must support

The dashboard tool needs to know how to draw exemplars and which tracing backend to open when you click one. In practice this means configuring a link from the metrics data source to the tracing data source, usually by telling the tool which field holds the trace identifier. Once configured, the experience is smooth. Until configured, the exemplars are stored but invisible.

Why dashboards show them as dots

An exemplar belongs to one moment in time and has its own value, so dashboards draw it as a single marker at that time and height rather than as part of the line. This is helpful, because the marker sits visually on top of the spike you are investigating. A cluster of markers high above the line tells you that several slow requests were captured in that period, and picking the highest one usually leads to the most informative trace.

Remember that the markers are examples, not a count. Ten markers do not mean ten slow requests happened. They mean that ten examples were kept, and the real number of slow requests is shown by the metric itself.

A link is only as good as the trace it points to.

Because an exemplar contains a trace identifier, it can easily point to a trace that no longer exists, or never reached the backend. This is called a dangling link. You click the marker and the tracing tool answers that it cannot find the trace. The causes are almost always about sampling and retention.

The head and tail sampling interaction

With head-based sampling, the decision is made when the trace starts, and the trace-based exemplar filter sees exactly that decision, so the linked trace is very likely to exist. With tail-based sampling, the final decision comes later, so the SDK cannot know whether a trace will survive. Exemplars created for traces that are later dropped become dangling links. Teams that use tail sampling often accept a small share of such links, or arrange for their sampling policies to keep the slow and failing traces, which are the ones most likely to be chosen as exemplars anyway.

A quiet bonus feature that many teams overlook.

When you record a measurement, you can attach attributes to it, such as the route, the customer tier, or the region. Metrics usually keep only some of these attributes, because every extra attribute multiplies the number of series. A configuration object called a view lets operators choose which attributes a metric retains and which it discards. The discarded attributes seem to disappear.

Exemplars rescue some of that lost context. The attributes that were present on the original measurement but removed from the metric’s series are kept on the exemplar as filtered attributes. That means a chart that shows latency grouped only by route can still hand you, through an exemplar, the customer tier or the region of one concrete slow request, without creating a series for every tier and region.

Be careful what you leave in

Because filtered attributes travel to your metrics backend, they inherit the privacy and size concerns of any exported data. Backends may limit how much attached text an exemplar can hold, and sensitive values should never be placed in attributes that might be copied onto exemplars. We return to this in the security chapter.

An honest balance sheet before you rely on them.

When they are worth it

Exemplars repay their small effort when your teams already use both metrics and traces, when incidents often start from a latency or error chart, and when engineers lose time hunting for representative traces. They add little if you do not trace at all, or if your traffic is so small that you can simply search all the traces.

The cost in numbers

An exemplar is small. Its fixed parts are a value of about eight bytes, a timestamp of about eight bytes, a trace identifier of sixteen bytes, and a span identifier of eight bytes, which together come to roughly forty bytes before any filtered attributes. If each data point carries five exemplars, that is about two hundred bytes of extra payload per data point. A service reporting a thousand series every export interval therefore sends around two hundred kilobytes of additional data in that interval, which is modest compared with the span data it already produces.

The real cost is rarely the bytes. It is the effort of enabling support in every layer, and the attention needed to keep traces around long enough for the links to work. Teams that budget for that effort, rather than for bandwidth, are the ones that get lasting value from the feature.

A small feature that crosses two data stores deserves a short safety review.

Trace identifiers themselves are random numbers and not sensitive, but the places they lead to can be. A trace may contain query text, customer identifiers, internal hostnames, or error messages with private details. An exemplar makes those traces one click away from anyone who can view the metrics dashboard. If access to metrics is broader than access to traces, exemplars can quietly widen who can reach sensitive data.

Where teams find the biggest payoff.

The following examples describe patterns widely used across the industry. In every case the story begins with a chart and ends with a concrete trace.

Habits that keep exemplars useful, and traps to avoid.

flowchart TD
  A["Do you collect both metrics and traces?"] -->|No| B["Set up the missing signal first"]
  A -->|Yes| C{"Do incidents start from charts and alerts?"}
  C -->|No| D["Exemplars are optional"]
  C -->|Yes| E{"Do your backends store and link exemplars?"}
  E -->|No| F["Enable support or choose compatible backends"]
  E -->|Yes| G["Use trace-based filter on latency and error metrics"]
        

Patterns that work

Anti-patterns to avoid

A short checklist for the question “why can I not see any exemplars?”

Exemplars fail quietly. The metric still arrives, the chart still draws, and nothing reports an error. The only symptom is the absence of markers. Because the feature spans five layers, the most efficient approach is a fixed checklist that walks from the application to the screen and stops at the first place where exemplars are missing.

Short answers to the questions people ask most often.

An exemplar is a real sample measurement attached to a metric data point, carrying the trace and span identifiers of the request that produced it, so you can jump from a summary number to a concrete trace.

No. Exemplars are extra payload on an existing data point, not part of the series identity, so they do not increase the number of series. That is the main reason to prefer them over trace identifiers as ordinary attributes.

The data model allows them on sums, histograms, exponential histograms, and gauges. In practice they are most valuable on histograms and counters recorded during real requests.

The standard default is the trace-based filter, which only considers measurements recorded while a sampled span is active. This makes it likely that the linked trace exists in your backend.

Only a few, set by the reservoir capacity. Bucket-aligned reservoirs for explicit histograms can hold up to one per bucket, while fixed-size random reservoirs hold a small configured number.

A bucket gets an exemplar only if at least one eligible measurement fell into it during the interval. Rare buckets combined with low sampling often have none, which is expected behaviour.

The trace may have been dropped by a later sampling stage, may not have been exported, or may have expired because trace retention is shorter than metric retention. Aligning retention and favouring slow and failed traces reduces this.

They can, but the SDK cannot know the final decision when it creates the exemplar. Some exemplars will point to traces that tail sampling later drops, so teams usually keep slow and failed traces and accept a small share of dangling links.

They can, through filtered attributes and through the traces they link to. Keep personal data out of measurement attributes and make sure the people who can see exemplars are allowed to open the traces.

No. Exemplars are a filtered, biased handful of examples. Use the metric counts for frequency and the exemplars for looking at what one real case looks like.

Exemplars solve a very practical problem: a metric tells you that something is wrong, but it forgets which requests were responsible. By keeping a handful of real measurements, each stamped with the trace and span identifiers of the request that produced it, OpenTelemetry lets a chart point straight at a trace. The cost stays small because the number of exemplars per data point is capped by a reservoir, and the number of series does not grow at all.

To make the link work, every layer must cooperate. The SDK needs a filter that admits measurements from sampled spans, a reservoir that picks a few of them, and an active span at the moment of recording. The pipeline must carry the exemplars through, the metrics store must keep them, the trace store must still hold the traces, and the dashboard must know how to connect the two. When the pieces line up, an engineer goes from a latency spike to the exact slow request in a single click. When any piece is missing, the failure is silent, which is why a smoke test and a short checklist are worth having. Like the sticker on the temperature chart, an exemplar is small, but it points to the whole story.