What Is an Exemplar in OpenTelemetry?
A metric tells you that something is slow. An exemplar hands you one real request that was slow, with a trace ID attached, so you can jump from a chart straight to the story behind the number.
Imagine you run a large restaurant and the manager hands you a report that says “average waiting time last night was 14 minutes, and 5 percent of tables waited more than 40 minutes.” You now know there is a problem, but you have no idea why those tables waited. What you really want is a sticky note pinned to the report that says “table 17, seated at 8:42 pm, waited 47 minutes, here is the order ticket.” That sticky note is an exemplar. In the world of OpenTelemetry, it is a tiny, carefully chosen sample that connects a summarized number to the exact request that helped create it.
1The Problem Exemplars Solve
Metrics are cheap and fast, but they throw away the details that explain what happened.
Numbers without stories
Modern systems are made of dozens or hundreds of services that call each other. To keep an eye on them, teams collect metrics such as request counts, error counts, and request durations. Metrics are popular because they are small. Instead of storing every single request, a metric system adds them up into totals, averages, or buckets and stores only the summary.
That summarizing step is exactly what makes metrics efficient, and it is also what makes them frustrating. When a graph shows a sudden spike in the 99th percentile latency, the graph cannot tell you which request caused it. The individual requests have already been folded into the summary, and you cannot unfold them again.
Think of a school report card. It says your child scored 72 percent in maths overall. It does not say which chapters were hard. Teachers keep a few marked answer sheets as evidence, and when parents ask “why only 72?”, they pull out one or two sheets and say “look at this question here.” Exemplars are those marked answer sheets for your metrics.
The classic investigation gap
Without exemplars, an engineer who notices a latency spike usually does the following. They look at the chart, note the time window, then open the tracing tool and search for slow traces in that same window. They guess which service, which endpoint, and which customer might be involved. This works, but it is slow, and it depends on the engineer’s luck and experience.
The gap exists because metrics and traces live in different tools and describe the same events from different angles. Metrics say how much and how often. Traces say what happened inside one request. Nothing in a plain metric points to a plain trace. Exemplars build that missing bridge.
An exemplar does not replace metrics or traces. It is a small link that lets you move from the summary world of metrics into the detailed world of traces in one click.
2A Quick Refresher on OpenTelemetry Signals
Before we meet exemplars, let us make sure the surrounding vocabulary is clear.
What OpenTelemetry is
OpenTelemetry, often shortened to OTel, is an open standard and a collection of tools for producing, collecting, and exporting telemetry data from software. Telemetry simply means data that your software sends out about its own behavior. OpenTelemetry is vendor neutral, which means you can produce data once and send it to many different backends such as Prometheus, Grafana Tempo, Jaeger, or commercial observability products.
The project defines three main kinds of data, called signals. Each signal answers a different type of question, and exemplars sit at the point where two of those signals touch.
Traces
A trace follows a single request across services. It is made of spans, where each span is one step of work with a start time and a duration.
Metrics
Metrics are numeric measurements aggregated over time, such as the number of requests per second or the distribution of response times.
Logs
Logs are timestamped text or structured records describing a specific event, such as an error message or a payment decision.
Context
Context is the invisible baton that carries the current trace ID and span ID through the code so that all signals can refer to the same request.
Measurements and data points
Two more words matter for this topic. A measurement is a single raw number recorded by your code at one moment, for example “this checkout request took 830 milliseconds.” A data point is what the metrics system produces after it combines many measurements, for example “between 10:00 and 10:01, 1,200 requests were recorded, and their durations fell into these buckets.”
Measurements are many and tiny. Data points are few and summarized. An exemplar is a measurement that escaped the summarizing process and was kept alive, with extra details, next to the data point it helped build.
A measurement is a single receipt at a shop. A data point is the daily sales total in the accounting book. An exemplar is one receipt that the accountant clips to the page of the book, so that the total has a real example attached to it.
3So, What Exactly Is an Exemplar?
A plain definition first, then the details that make it useful.
The definition
In OpenTelemetry, an exemplar is a recorded example measurement that was part of a metric, kept in its original, unaggregated form, and stored alongside the metric data point. It carries the actual value that was measured, the time it was measured, and, when the measurement happened inside a traced request, the trace ID and span ID of that request. It may also carry extra attributes that were dropped from the metric during aggregation.
Put even more simply, an exemplar says “here is one real event that contributed to this number, and here is how to find it.”
Where exemplars live
An exemplar is not a new signal. It does not stand alone in its own pipeline. It always belongs to a metric data point. When a histogram data point is exported, the exemplars travel inside it. If the data point is deleted or never exported, the exemplars go with it. This is an important mental model, because it explains why exemplars are small, cheap, and tightly bound to metrics.
Why the trace ID matters most
Every exemplar can carry a value and a timestamp, but the real magic is the trace ID and span ID. These two identifiers are the address of the request in your tracing system. Once a dashboard shows an exemplar, a user can click it and be taken directly to the trace with that ID, where every step of that particular slow request is laid out.
An exemplar only has a trace ID if the measurement was recorded while a span was active. A measurement taken in a background job with no tracing context can still become an exemplar, but it will have no link to a trace.
4The Anatomy of an Exemplar
An exemplar is small, and it has only a handful of fields. Each one has a clear job.
Value
The raw number that was measured, such as 830 milliseconds or a count increment of 3. It can be a whole number or a decimal, matching the type of the metric.
Timestamp
The exact moment the measurement was taken. This lets a dashboard place the exemplar as a dot at the right position on a time chart.
Trace ID
A unique identifier of the whole request journey. Tracing backends use it as the key to find the full trace.
Span ID
A unique identifier of the specific step within the trace that was active when the measurement was recorded. It lets the viewer land on the exact span.
Filtered Attributes
Key-value labels that were present on the original measurement but removed from the aggregated metric, for example a customer ID or a pod name.
Why filtered attributes are a hidden gem
Metrics suffer from a well-known problem called high cardinality. Cardinality means the number of distinct combinations of label values. If you attach a customer ID to a metric and you have two million customers, you suddenly create two million separate time series, and your monitoring system becomes slow and expensive.
To avoid that, teams remove such labels from metrics using a configuration step called a view. The removed labels are gone from the metric, but they are not lost forever if they are copied into an exemplar. The exemplar remembers “this slow request belonged to customer 48213 on pod checkout-7f9,” even though the metric itself knows nothing about customers or pods.
A postal service counts parcels per city to save effort, and does not write each recipient’s name in the monthly report. But one parcel from each city is kept in a glass cabinet with its full label still attached. If someone asks “what does a typical delayed parcel look like?”, the cabinet answers the question without bloating the report.
Exemplars do not contain the whole trace. They contain only the IDs that point to it. The trace itself is stored in your tracing backend, and if that backend never received the trace, the link will lead nowhere.
5The Life of an Exemplar, Step by Step
From the moment a request arrives until a dot appears on a dashboard, an exemplar goes through a short and orderly journey.
A request starts and a span is created
The instrumented application receives a request. The tracing library creates a span and places its trace ID and span ID into the current context.
Your code records a measurement
While the request is being served, the code records a value into a metric instrument, such as a duration into a histogram, along with the usual attributes.
The exemplar filter decides if this measurement is eligible
The SDK asks a gatekeeper called the exemplar filter whether this measurement should even be considered as an exemplar candidate.
The reservoir decides whether to keep it
If the measurement is eligible, it is offered to a small storage area called the exemplar reservoir. The reservoir keeps or discards it according to its own rules.
The value is aggregated as usual
Whatever happens to the exemplar, the measurement is also added into the normal aggregation, so counts and sums stay correct.
At collection time, exemplars are attached to the data point
When the SDK prepares a batch for export, it reads the exemplars from the reservoir and attaches them to the matching data point, then empties the reservoir for the next period.
Export and display
The data point with its exemplars is sent to a collector or backend. A dashboard tool draws the exemplars as dots, and a click on a dot opens the linked trace.
sequenceDiagram
participant U as User Request
participant A as Application Code
participant F as Exemplar Filter
participant R as Exemplar Reservoir
participant G as Aggregator
participant E as Exporter
U->>A: Request arrives with active span
A->>F: Record duration measurement
F->>R: Eligible, offer as candidate
R->>R: Keep or replace stored sample
A->>G: Add value to histogram buckets
Note over R,G: Collection interval ends
G->>E: Data point plus exemplars
E->>E: Send to collector and backend
Notice that the aggregation in step 5 always happens. Exemplars are a side copy, not a replacement. Turning exemplars off never changes the numbers your dashboards show.
6The Exemplar Filter: The Gatekeeper
Not every measurement deserves to become an exemplar. The filter is the first line of selection.
Why a filter is needed
A busy service can record millions of measurements every minute. Even a cheap check on each one adds up. More importantly, an exemplar without a trace ID is much less valuable, because the user cannot jump anywhere from it. The filter exists to make sure effort goes only into measurements that are likely to be useful.
The three standard filter behaviors
Always Off
No measurement is ever offered to the reservoir. Exemplars are effectively disabled, and there is no extra cost.
Always On
Every measurement is eligible, whether or not it is inside a traced request. This gives the widest pool but may produce exemplars with no trace link.
Trace Based
Only measurements taken while a sampled span is active are eligible. This is the usual default, and it guarantees that the exemplar’s trace ID points to a trace that is actually being kept.
Why trace based is the sensible default
Tracing systems rarely keep every trace. They use sampling, which means they store only a fraction of traces to control cost. If an exemplar pointed to a trace that was discarded by the sampler, the user would click the dot and find nothing. The trace based filter prevents this disappointment by checking the sampling decision of the active span before allowing the measurement to become an exemplar.
A museum curator decides which items may enter the display room. “Always off” means the room is closed. “Always on” means anything may enter, even items with no label. “Trace based” means only items that already have a catalogue number in the museum’s archive may enter, so every visitor can look up the full story.
Most SDKs let you choose the filter through a standard environment variable named OTEL_METRICS_EXEMPLAR_FILTER, with the values always_on, always_off, and trace_based. Check your language SDK’s documentation, because exemplar support and maturity differ between languages.
7The Exemplar Reservoir: The Small Storage Box
The reservoir decides which few samples survive out of thousands of eligible candidates.
What a reservoir is
The exemplar reservoir is a tiny buffer attached to each metric data point that is being built. It holds only a handful of exemplars at a time. When a new eligible measurement arrives, the reservoir applies its policy to decide whether to store it, replace an older one, or ignore it.
The reservoir size is deliberately small. This keeps memory use low, keeps exported payloads light, and ensures that an exemplar remains an illustrative example rather than a second copy of all your data.
The two main reservoir designs
Simple Fixed Size Reservoir
Keeps a fixed number of slots and fills them with a random but fair selection of the measurements seen during the period. Every eligible measurement has an equal chance of surviving. The default size is related to the number of processors available to the application.
Aligned Histogram Bucket Reservoir
Keeps at most one exemplar for each histogram bucket. A measurement replaces the stored one in its own bucket, so the final set covers many different value ranges, from fast requests to very slow ones.
Why the bucket aligned design is clever
A histogram divides response times into ranges, for example up to 100 milliseconds, up to 250 milliseconds, up to 500 milliseconds, up to one second, and above. If you keep one exemplar per range, then for every part of the distribution you have a real example. When the tail of the histogram grows, you already have a sample sitting in the slow bucket, ready to be clicked.
With the random fixed size design, there is no such guarantee, and slow outliers might be missed simply because most measurements are fast. That is why explicit bucket histograms generally use the aligned design by default, and other aggregations use the random design.
flowchart TD
M["Incoming measurement"] --> F{"Filter eligible?"}
F -- "No" --> X["Skip exemplar, still aggregate"]
F -- "Yes" --> T{"Reservoir type"}
T -- "Fixed size" --> S["Random fair slot selection"]
T -- "Bucket aligned" --> B["Replace sample in matching bucket"]
S --> H["Hold until collection"]
B --> H
H --> D["Attach to data point and export"]
Picture a photographer at a marathon with room for only a few photos. The fixed size reservoir is like picking runners at random across the whole race. The bucket aligned reservoir is like promising to take one photo at each kilometre mark, so you always have a picture of the front, the middle, and the very last runner.
Because the reservoir is small and sampling is partly random, an exemplar is an example, not a complete list. You cannot count exemplars to count requests, and you cannot assume the slowest request of the period is among them.
8Views, Cardinality, and the Role of Filtered Attributes
Exemplars and views work as a team: views shrink the metric, exemplars remember what was removed.
What a view does
A view is a configuration rule in the metrics SDK that reshapes how a metric is aggregated before export. A view can rename a metric, change its aggregation type, choose different histogram buckets, and, most importantly for us, decide which attributes to keep. Attributes that a view does not keep are dropped from the exported time series.
The cardinality budget
Every distinct combination of attribute values becomes its own time series, and every time series costs storage, memory, and query time. Teams therefore keep a cardinality budget, a limit on how many series a metric is allowed to create. Good attributes for metrics have few distinct values, such as HTTP method, status class, or region. Bad attributes have unbounded values, such as user ID, order ID, or full URL path with identifiers.
| Attribute | Distinct values | Keep on metric? | Put in exemplar? |
|---|---|---|---|
| HTTP method | Under 10 | Yes | Not needed |
| Status class (2xx, 4xx, 5xx) | 5 | Yes | Not needed |
| Region | Tens | Usually yes | Optional |
| Pod or instance name | Hundreds, changing | Often no | Yes, very helpful |
| Customer or user ID | Millions | No | Yes, helpful |
| Order or request ID | Unbounded | Never | Yes, helpful |
How filtered attributes flow
When a view drops an attribute, the SDK can place that dropped attribute on the exemplar as a filtered attribute. The final exported record therefore contains a clean low-cardinality metric plus a few exemplars that still remember their rich details. You get the best of both worlds: a cheap time series and a detailed example.
Example: Tracking down a noisy customer
A payments team keeps a request duration histogram labeled only by payment method and status. They notice a rising slow tail. The exemplars on the slowest bucket all carry the same filtered customer ID, revealing that a single large merchant is sending oversized batch requests. The metric never needed a customer label to expose this.
Treat exemplar attributes as “clues for a human investigator,” not as data for dashboards to group by. You cannot sum, average, or group by an exemplar attribute across time. You can only read it on the individual sample.
9Exemplars Across Different Metric Types
The idea is the same everywhere, but what an exemplar means depends on what the metric measures.
OpenTelemetry offers several kinds of metric instruments and aggregations. Exemplars can accompany the data points of several of them, and the table below summarizes how to think about each one. The exact support varies by language SDK and backend, so always confirm in the tools you use.
| Metric type | What it measures | What an exemplar shows | Typical value |
|---|---|---|---|
| Explicit bucket histogram | Distribution of values in fixed ranges | One real request per range, such as a slow outlier | Very high |
| Exponential histogram | Distribution using automatically scaled buckets | Sample requests across a wide dynamic range | Very high |
| Counter or sum | A running total, such as requests or errors | One increment event, such as one failed request | High for error counters |
| Gauge | A value at a point in time, such as queue depth | A sampled reading with its surrounding trace | Moderate |
Histograms: the natural home
Histograms are where exemplars shine, because a histogram describes a distribution, and the interesting stories live in the extremes of a distribution. A dot sitting in the slowest bucket is a direct invitation to ask “what happened in this request?” Latency histograms with exemplars are the most common and most valuable combination in practice.
Counters: examples of events
For counters, an exemplar represents one increment. If you have an error counter, an exemplar attached to it is a real failing request. When the error rate chart jumps, the dots on it lead you to the failing traces without any guessing. This is especially useful in incident response, when time is short.
Gauges: use with care
A gauge is a snapshot of a current value, for example the number of items waiting in a queue. Exemplars on gauges are possible but less commonly needed, since a gauge reading is not usually tied to a single request. Teams add them where a reading is recorded inside a traced operation and the context is meaningful.
10From the SDK to the Dashboard: The Export Path
An exemplar is only useful if it survives the trip from your application to a screen.
OTLP carries exemplars natively
OTLP, the OpenTelemetry Protocol, is the standard wire format used to send telemetry. Its metric data points have a built-in place for exemplars. Each exemplar in the message carries its value, its timestamp, its trace ID, its span ID, and its list of filtered attributes. This means that if every hop in your pipeline speaks OTLP and understands exemplars, nothing is lost along the way.
The role of the OpenTelemetry Collector
Many organizations place an OpenTelemetry Collector between applications and backends. The collector receives telemetry, optionally processes it, and exports it onward. It can pass exemplars straight through, and it can also convert OTLP metrics into the formats other systems expect. Because processors in the collector can modify or drop data, it is worth checking that none of your processing steps accidentally strips exemplars.
Prometheus and the OpenMetrics world
Prometheus is one of the most widely used metrics backends. It understands exemplars when metrics arrive in the OpenMetrics text format or through its supported ingestion paths, and it must be configured to store them, since exemplar storage is an optional feature with a bounded in-memory buffer. In that format an exemplar appears as a small annotation on a histogram bucket or counter line, carrying the trace ID and span ID as labels along with the value and timestamp.
One practical limit to remember is that OpenMetrics restricts the total size of an exemplar’s labels to a small number of characters. Trace IDs and span IDs fit comfortably, but a long list of filtered attributes may not, so conversion steps may drop some of them.
Dashboards and trace backends
Dashboard tools such as Grafana can draw exemplars as dots over a time series chart. Each dot is clickable, and the tool is configured with a link template that turns the trace ID into a URL for your tracing backend, such as Grafana Tempo or Jaeger. The result is a smooth jump from a latency graph to the exact trace.
graph LR
APP["Application with SDK"] -->|"OTLP metrics with exemplars"| COL["OpenTelemetry Collector"]
APP -->|"OTLP traces"| COL
COL -->|"Metrics, exemplars kept"| MET["Metrics backend"]
COL -->|"Sampled traces"| TRC["Tracing backend"]
MET --> DASH["Dashboard shows dots on chart"]
DASH -->|"Click uses trace ID"| TRC
Teams often enable exemplars in the application and then find no dots on their dashboard. The usual causes are a backend that is not set to store exemplars, a data source in the dashboard tool without the trace link configured, or a conversion step that removed exemplars. Always verify the whole path, not just the first step.
11Using Exemplars in a Real Investigation
Let us walk through a believable incident to see how exemplars change the way engineers work.
The scenario
An online store has a checkout service. A support message arrives saying “checkout feels slow for some people.” The on-call engineer opens the latency dashboard for the checkout service, which shows a histogram-based percentile chart with exemplar dots enabled.
Spot the symptom on the chart
The 99th percentile line rose from under one second to over four seconds starting at 14:10. The median barely moved, so the problem affects only a small group of requests.
Pick an exemplar from the slow region
Several dots sit high on the chart at around four to five seconds. The engineer hovers over one and sees its value, its time, and a filtered attribute showing a particular database replica name.
Open the linked trace
One click opens the full trace. It shows that the request spent almost all its time waiting on a single call to the inventory service, which in turn waited on a database query.
Confirm with a second and third exemplar
The engineer opens two more exemplars and sees the same slow database call on the same replica. One example might be a fluke, but three examples form a pattern.
Act
The team removes the unhealthy replica from rotation, the percentile line returns to normal, and the incident is closed in minutes rather than hours.
What changed compared to life without exemplars
Without exemplars, the engineer would need to switch to the tracing tool, search for slow traces in the right time window, and filter by service name. They would have to guess how to filter by replica, a detail the metric never carried. The investigation might still succeed, but it would take longer, and the chance of being misled by an unrepresentative trace would be higher.
More everyday use cases
Finding the cause of error spikes
An error counter jumps after a deployment. Exemplars on the counter lead to failing traces, whose error spans point to a misconfigured downstream endpoint introduced in the new release.
Validating a performance fix
After optimizing a query, a team checks whether the slowest bucket of the histogram still holds exemplars. If the dots disappear from the slow region, that is visual proof that the long tail has been tamed.
Understanding strange outliers
A histogram shows a small cluster of requests taking exactly thirty seconds. An exemplar from that cluster reveals a retry loop hitting a timeout, a pattern the average would have hidden completely.
Service level objective reviews
When a latency objective is breached, exemplars give the review team real requests to study, making post-incident discussions concrete rather than theoretical.
12Exemplars Compared with Other Ways of Linking Signals
Exemplars are one of several techniques for connecting metrics, traces, and logs. Knowing the others helps you choose wisely.
| Technique | How it links signals | Strength | Weakness |
|---|---|---|---|
| Exemplars | Attach a real sample with trace ID to a metric data point | Direct jump from chart to trace, cheap in storage | Only a few samples, not guaranteed to include the worst case |
| High-cardinality metric labels | Add IDs as labels on the metric | Queryable and groupable | Explodes storage and cost |
| Metrics derived from spans | Compute counts and durations from trace data itself | Complete coverage of sampled traces | Depends on sampling, adds processing cost |
| Log correlation | Put trace ID inside log lines | Rich event detail | Search-based, not a one-click jump from a metric chart |
| Manual time window search | Engineer searches traces by time and service | Needs no setup | Slow and dependent on guesswork |
They work best together
These techniques are not rivals. A mature observability setup typically uses low-cardinality metrics for alerting and trends, exemplars for fast drill-down from metrics to traces, traces for detailed request analysis, and logs enriched with trace IDs for the finest event detail. Exemplars are the shortest bridge between the first and the third of those.
Think of a library. Metrics are the catalogue summary of how many books are on each shelf. Exemplars are the bookmarks sticking out of a few books on each shelf, telling you where to start reading. Traces are the books themselves. Logs are the handwritten notes in the margins.
If you find yourself wanting to add an ID label to a metric just so you can find one request later, stop and use an exemplar instead. That is exactly the job exemplars were designed for.
13Exemplars and Trace Sampling: A Delicate Partnership
The value of an exemplar depends entirely on whether its trace still exists when someone clicks.
Two sampling decisions that must agree
There are really two selection steps in play. The first is trace sampling, which decides whether a trace is kept at all. The second is exemplar selection, which decides whether a measurement is saved as an exemplar. If these two steps do not agree, you end up with exemplars that point to missing traces.
Head sampling versus tail sampling
With head sampling, the decision to keep a trace is made at the very start of the request, before anything is known about how it will turn out. In this case the sampling decision is already attached to the span, and the trace based exemplar filter can read it immediately. Exemplars and kept traces line up neatly.
With tail sampling, the decision is made after the trace is complete, often inside a collector, so that interesting traces such as slow or failed ones can be preferred. Here the application cannot know at recording time whether the trace will be kept. Exemplars may therefore point to traces that the tail sampler later drops. Teams that rely heavily on tail sampling should test how many exemplar links resolve successfully and adjust their sampling rules accordingly.
flowchart LR
R["Request with span"] --> D{"Sampled by head sampler?"}
D -- "Yes" --> K["Trace kept, exemplar eligible"]
D -- "No" --> N["Trace dropped, no exemplar with trace based filter"]
K --> L["Exemplar link resolves in trace backend"]
N --> Q["No dangling link created"]
If you turn trace sampling down to a very low percentage, you will also see fewer exemplars, because the trace based filter only considers sampled requests. Very rare slow requests may then have no exemplar at all. Balance sampling rates with how much drill-down ability you need.
Exemplars are like footnotes that cite a page number in a book. If the library throws away certain pages, those footnotes now lead nowhere. Keeping the footnotes consistent with the pages that actually exist is the job of coordinated sampling.
14Cost, Performance, and Security Considerations
Exemplars are light by design, but light is not the same as free.
Performance overhead
Recording an exemplar involves a quick filter check and, if eligible, a small copy of a few values into the reservoir. For most applications this cost is tiny compared with the work of serving a request. Still, in extremely hot code paths, such as a loop recording millions of measurements per second, it is wise to measure the overhead in your own environment instead of assuming.
Memory and payload size
Because each reservoir holds only a few samples, memory use per time series is small. However, if you have a very large number of time series, then the total number of exemplars multiplies accordingly. Each exemplar also adds bytes to every export. In large fleets, this can show up in network and storage bills, so it pays to keep metric cardinality under control even with exemplars in place.
Security and privacy
Filtered attributes can hold sensitive details such as customer identifiers, email addresses, or internal hostnames. Because exemplars are exported and stored in metrics backends, those details travel and rest in places that might have different access rules than your tracing system. Treat exemplar attributes with the same care as log content.
Advantages
- One-click path from a metric chart to a real trace
- Keeps metric cardinality low while preserving rich detail
- Very small storage and memory footprint per series
- Shortens incident investigation time significantly
- Works through the standard OTLP format across tools
- Gives concrete evidence for performance discussions
Disadvantages / Trade-offs
- Only a small, partly random sample, never a full list
- Depends on trace sampling and a working trace backend
- Support and maturity vary across language SDKs and backends
- Can carry sensitive attributes into the metrics system
- Needs configuration at several points in the pipeline
Monitor your monitoring. Track whether exemplars are arriving at your backend, how many resolve to real traces, and whether payload sizes grow unexpectedly after you enable them.
15Best Practices and Common Anti-patterns
A short list of habits that make exemplars shine, and a few traps that waste their potential.
Habits that work
Instrument traces first
Exemplars are only as good as the traces they point to. Make sure tracing is healthy and context propagates across service boundaries before expecting good exemplars.
Use histograms for latency
Record durations in histograms rather than averages or gauges. This gives exemplars a natural place to attach and keeps tail behavior visible.
Keep the trace based filter
Stay with the sampled-trace default unless you have a clear reason to change it, so links always lead to stored traces.
Drop risky labels with views
Remove unbounded attributes from metrics through views, and let exemplars carry them as clues for investigators.
Configure the whole path
Verify the SDK, collector, metrics backend, dashboard data source, and trace link settings together as one chain.
Teach the team
Show engineers how to read exemplar dots on dashboards during calm periods, so they use them confidently during incidents.
Traps to avoid
Problem
The team adds customer ID or request ID as a regular metric label so that they can find specific requests later.
Why It’s Harmful
Every unique ID creates a new time series. Storage and query costs rise sharply, dashboards become slow, and the metrics backend may start rejecting data.
Correct Approach
Keep the metric labels low in cardinality and allow exemplars to carry the identifiers as filtered attributes alongside a trace link.
Problem
The team treats exemplars as a complete record and draws conclusions such as “there were only three slow requests because we see three dots.”
Why It’s Harmful
Exemplars are a small sample chosen by a reservoir. The number of dots has no relationship to the number of requests, so conclusions about volume will be wrong.
Correct Approach
Use the histogram counts and percentiles for quantities. Use exemplars only to explore why the numbers look the way they do.
Problem
Exemplars are enabled while trace sampling is extremely low or the filter is set to always on without a matching trace strategy.
Why It’s Harmful
Users click on dots and land on “trace not found” pages. Trust in the feature falls, and engineers stop using it.
Correct Approach
Align trace sampling and exemplar filtering so that exemplar links resolve reliably, and test sample links regularly.
Problem
Sensitive personal data such as emails or full names is placed into attributes that end up on exemplars.
Why It’s Harmful
The data is exported and stored in the metrics backend, often with wider access and longer retention than expected, creating privacy and compliance risk.
Correct Approach
Use opaque identifiers, review which attributes are allowed to flow into exemplars, and apply the same data rules you use for logs.
16Frequently Asked Questions
Quick answers to the questions that beginners and intermediate engineers ask most often.
No. OpenTelemetry has traces, metrics, and logs as its main signals. An exemplar is a feature of the metrics signal that carries references to the tracing signal. It always travels inside a metric data point.
No. The measurement is aggregated normally whether or not it also becomes an exemplar. Turning exemplars on or off never alters counts, sums, or percentiles.
No. A data point has exemplars only if eligible measurements were recorded during the period and the reservoir kept some. Quiet periods, disabled filters, and measurements outside traced requests can all result in a data point with none.
Only a few. The exact number depends on the reservoir design and its configured size. Bucket aligned reservoirs keep at most one per histogram bucket, and fixed size reservoirs keep a small fixed number.
You can technically record exemplars with the always on filter, and they will carry values, timestamps, and filtered attributes. However, without trace IDs the main benefit, jumping to a trace, is lost. Most teams enable tracing alongside exemplars.
Check each link of the chain. Confirm that the SDK has exemplars enabled and that requests are inside sampled spans. Confirm that the collector passes exemplars through. Confirm that the metrics backend stores them. Finally, confirm that the dashboard data source is set up to show exemplars and to build trace links.
Span events and attributes live inside a trace and describe things that happened within a request. An exemplar lives inside a metric and points from the metric to a request. They travel in opposite directions: one explains a trace, the other helps you find one.
Usually not, because exemplars are few and small. But they add bytes to every export, so very large fleets with many series should measure the effect. Controlling metric cardinality remains far more important for cost than the exemplar feature itself.
No. Maturity differs across language SDKs, collectors, and backends, and the specification and implementations continue to evolve. Always check the current documentation for your specific tools before planning a rollout.
Alerts are triggered by metric rules, not by exemplars. But some alerting setups include a link to an exemplar or a dashboard panel with exemplar dots, so the person who receives the alert can start investigating immediately.
17Summary and Key Takeaways
Exemplars solve a very human problem: numbers tell you that something is wrong, but people need examples to understand why. By keeping a few real measurements, with their trace IDs and some of the details that aggregation removed, OpenTelemetry lets you hop from a latency chart or an error graph straight into the trace of a real request. The mechanism is small and tidy. A filter chooses which measurements are eligible, a reservoir keeps a handful of them, and the data point carries them to the backend, where a dashboard turns them into clickable dots. Used together with low-cardinality metrics, well-sampled traces, and careful handling of attributes, exemplars turn incident investigation from a search party into a guided tour.
Key Takeaways
- An exemplar is a sample, not a signal — it is a real measurement kept alongside a metric data point, with value, timestamp, and optional trace and span IDs.
- It bridges metrics and traces — the trace ID turns a chart dot into a direct link to the full request journey.
- Filter and reservoir control selection — the filter decides eligibility, usually based on sampled traces, and the reservoir keeps only a few examples per data point.
- Filtered attributes protect cardinality — labels removed from the metric by views can still ride along on the exemplar as investigation clues.
- Histograms and error counters benefit most — they hold the interesting extremes and failures that engineers want to explore.
- Sampling and pipeline alignment matter — exemplar links only help when the trace is stored and every hop from SDK to dashboard preserves the exemplar.
- Treat exemplars as illustrations, not statistics — never count them, never group by their attributes, and handle sensitive details with the same care as logs.