Why anyone needs to throw tracing data away in the first place.

Distributed tracing records the journey of each request through your system as a trace, made of many small timed steps called spans. A single checkout click in a busy online shop might create twenty or thirty spans as it passes through the gateway, the cart, the payment service, the inventory service, and the database. That is useful detail when something goes wrong, because you can see exactly where the time was spent or where the error began.

The trouble is volume. A system that handles five thousand requests per second and produces twenty spans per request creates a hundred thousand spans every second. Each span has a name, timestamps, and a handful of attributes, so storing and indexing them costs real money. Networks must carry them, backends must ingest them, and engineers must search through them. At some scale, keeping every trace becomes either unaffordable or unusable, and sometimes both.

The sensible answer is to keep only a portion of the traces. This is called sampling. The hard part is choosing which portion, because not all traces are equally valuable. Ninety-nine percent of requests are healthy and nearly identical to each other. The one percent that failed or crawled are the ones engineers actually need. Any sampling method that treats them all the same wastes budget on the boring majority and risks losing the precious few.

A shared vocabulary for the rest of the guide.

The goal of sampling

Sampling is not about collecting less information for its own sake. It is about spending a limited budget of storage, network, and attention on the information most likely to be useful. A good sampling strategy keeps the interesting traces, keeps enough ordinary traces to understand normal behaviour, and keeps the total within budget.

Why whole traces matter

A trace is only helpful when it is complete. If a sampling method keeps the span from the gateway but drops the span from the database, the story has a hole in it. Both head-based and tail-based sampling try to keep or drop whole traces together rather than individual spans, which is why the decision is tied to the trace identifier and not to single spans.

The simple approach that tail-based sampling improves upon.

In head-based sampling, the decision is made at the very beginning of the trace, when the root span is created. The sampler inside the SDK looks at a few facts available at that moment, such as the trace identifier or the name of the operation, and answers yes or no. If the answer is no, the SDK can avoid recording the spans at all, saving the work of building and exporting them.

OpenTelemetry SDKs ship with several built-in samplers. One keeps everything, one drops everything, and one keeps a fixed percentage of traces based on the trace identifier. A fourth kind follows the parent: if the incoming request already carries a decision from an upstream service, the sampler simply obeys it. This is how the decision, made once at the entry point, spreads to every other service. It travels inside the trace context header as a flag that says whether the trace is sampled.

Deciding at the end of the story instead of the beginning.

Tail-based sampling means the decision to keep or drop a trace is made after the trace, or most of it, has been observed. The system collects the spans, holds them for a short time, looks at the whole picture, and only then applies rules. Did any span report an error? Was the total duration unusually long? Did the request involve a particular customer or feature? If the answer to an interesting question is yes, the trace is kept. If nothing stands out, it may be dropped or sampled at a low rate.

The word tail refers to the end of the trace, in contrast to the head at its beginning. Because the decision happens at the end, it can use information that did not exist at the start: the final status, the total latency, the number of spans, and the attributes that downstream services added along the way.

The price of seeing the ending

Waiting is not free. To decide at the end, something has to hold on to all the spans of every trace until the end arrives. That something needs memory, it needs a rule for how long to wait, and it needs to receive all the spans of a single trace in one place. Each of these needs shapes the architecture that the rest of this guide describes.

From the first span arriving to the final keep or drop.

sequenceDiagram
  participant A as Applications
  participant C as Collector sampling stage
  participant B as Backend
  A->>C: Spans of trace 42 arrive from several services
  Note over C: Group spans by trace identifier and start a timer
  A->>C: More spans of trace 42 arrive
  Note over C: Timer ends, apply policies to the whole trace
  alt A policy says keep
    C->>B: Export every span of trace 42
  else No policy says keep
    Note over C: Drop trace 42 and remember the decision
  end
  A->>C: A late span of trace 42 arrives
  Note over C: Look up the remembered decision and act on it
        

The component that does the waiting, grouping, and deciding.

Tail-based sampling in OpenTelemetry is implemented by a component of the Collector known as the tail sampling processor. Processors are building blocks that sit in the middle of a Collector pipeline: data comes in from receivers, passes through processors that can modify, filter, or group it, and then leaves through exporters. The tail sampling processor is distributed in the contrib collection of Collector components, so you need a Collector build that includes it.

The processor is configured with a handful of settings that control how long it waits, how much it remembers, and which rules it applies. Understanding these settings in plain language is more important than memorising their exact names, because names and defaults can differ between versions.

How multiple policies combine

Most setups define several policies, such as one for errors, one for latency, and one that keeps a small random share of everything. In the default behaviour, a trace is kept if any one policy votes to keep it. That means each policy adds to the set of traces you keep, which makes it easy to say “keep all errors, and also all slow traces, and also five percent of the rest”. Newer versions add ways to combine rules more precisely, for example by requiring several conditions together or by explicitly dropping certain traces, so check the documentation for your version.

The vocabulary of rules you can apply to a finished trace.

A policy is a named rule that looks at a complete trace and answers a single question: should this one be kept? The processor supports a family of policy types, each aimed at a common reason for caring about a trace. Choosing the right mixture is the creative part of tail-based sampling.

The same ten thousand traces, sampled both ways.

Suppose a shop produces ten thousand traces in a short period. Two percent of them contain an error, one percent are very slow but did not fail, and the other ninety-seven percent are ordinary. The team can afford to keep roughly five hundred traces. Let us see what each approach keeps.

What the numbers tell us

Head sampling at five percent keeps about five hundred traces, but only around ten of them are errors, and only about five are slow. An engineer investigating an outage sees a tiny, unrepresentative slice of the problem. Tail sampling keeps all two hundred errors and all one hundred slow traces, plus a five percent baseline of the rest, for a total of about 785. The storage bill is only about fifty-seven percent higher than head sampling, but the number of useful traces is more than twenty times greater.

If the team wants to stay at exactly five hundred traces, they can lower the baseline for ordinary traces to about 2.1 percent, which keeps around 200 of the ordinary ones. The total becomes roughly five hundred, yet every error and every slow trace is still present. That is the real prize of tail sampling: you move the budget from the boring majority to the valuable minority.

The architectural catch that surprises most newcomers.

To judge a whole trace, a single sampling stage must see all of its spans. This sounds obvious, but in a real system the spans of one trace are produced by many different services, running on many different machines, each sending data to whichever Collector instance happens to be convenient. If you run several Collector instances behind an ordinary load balancer, the spans of one trace get scattered across them. Each instance sees only a fragment, and the decisions it makes are based on incomplete information.

A fragment can mislead badly. One instance may hold the slow database span but not the error span, so it sees no reason to keep the trace. Another instance may hold the error span but not the long duration. Different instances may even make different decisions about the same trace, producing a backend that holds half a trace. The fix is to make sure all spans with the same trace identifier are routed to the same sampling instance.

flowchart LR
  A["Services and SDKs"] --> B["Tier one Collectors"]
  B -->|"Route by trace identifier"| C["Tier two Collector 1 with tail sampling"]
  B -->|"Route by trace identifier"| D["Tier two Collector 2 with tail sampling"]
  B -->|"Route by trace identifier"| E["Tier two Collector 3 with tail sampling"]
  C --> F["Backend"]
  D --> F
  E --> F
        

The two-tier pattern

The standard solution is two layers of Collectors. The first layer receives spans from applications and forwards them using a load balancing exporter that picks the destination by trace identifier. Because the same identifier always maps to the same destination, all spans of one trace end up on one second-layer instance. The second layer runs the tail sampling processor and has a complete view of each trace.

Finding the second tier

The load balancing exporter needs to know which second-tier instances exist. It supports several ways of finding them, such as a fixed list, a lookup through DNS, or discovery through an orchestration platform. When instances are added or removed, the mapping from trace identifiers to instances changes, and a few traces in flight at that moment may be split. This is a normal cost of scaling and a reason to avoid frequent resizing of the sampling tier.

Waiting for the ending costs memory, and the clock never stops.

Holding traces while they finish is the central resource cost of tail sampling. Every span that arrives must stay in memory until its trace is decided. The amount of memory therefore depends on how many traces arrive per second, how large each one is, and how long the processor waits before deciding.

Take a moderate example. If one trace of twenty spans takes about ten kilobytes, and five thousand new traces arrive every second, then a thirty second decision wait means about a hundred and fifty thousand traces in memory at once, which comes to around one and a half gigabytes. Real memory use is usually higher than the raw size of the data, because in-memory structures carry extra overhead, so plan with a generous margin. All of this must be spread across the tier-two instances, which is why the second tier is sized for memory rather than for processing speed.

Choosing the decision wait

The wait is a trade-off between completeness and cost. If it is too short, slow services may not have delivered their spans yet, and the processor decides on an incomplete trace. A trace whose error span arrives after the verdict may be dropped because the policy never saw the error. If the wait is too long, memory use grows and the backend sees data later. A good starting point is a little longer than the slowest typical request in your system, and teams then adjust using what they observe.

Long traces and late spans

Some traces last far longer than any reasonable wait, for example a background job that continues for minutes after the request that started it. A decision made after thirty seconds will cut such a trace short. The decision caches help here. When a span arrives after its trace has already been decided, the processor looks up the remembered outcome and either forwards or discards the span accordingly. This keeps the tail of a kept trace from being lost, though it cannot rescue a trace that was wrongly dropped before the important span arrived.

They are not rivals. Used together, each covers the other’s weakness.

Because tail sampling needs the spans to reach a Collector, it works only on the spans that applications actually send. If an SDK uses head-based sampling to drop ninety percent of traces before export, the Collector never sees those traces, and no tail rule can rescue the errors among them. This is the most important rule for combining the two: whatever the head sampler drops is gone for good.

So there are two common patterns. In the first, applications export everything, and the tail stage does all the sampling. This gives the best results but the highest cost in network traffic and Collector resources. In the second, a gentle head-based sample is applied first, for example to keep half of all traces, and tail sampling refines the half that remains. This reduces load at the cost of losing some interesting traces before the tail stage can see them.

A subtle trap with parent-based samplers

Many SDKs default to a sampler that obeys the decision made by the parent span. If the root service has a sampler that keeps only a small share, the sampled flag in the trace context tells every other service to drop the rest. This spreads the head decision across the whole system, which is exactly what you do not want when you plan to use tail sampling. For tail sampling to see everything, the entry-point sampler must keep all traces, and the downstream services must follow that decision.

Metrics must be computed before sampling

Request counts, error rates, and latency distributions are among the most valuable things you can derive from spans. If you compute them after sampling, they will be wrong, because you have thrown away most of the normal traffic. The remedy is to compute metrics from the full stream of spans before the sampling stage, using a Collector component that turns spans into metrics, and then sample the traces for storage afterwards.

flowchart LR
  A["Receive spans"] --> B["Memory limiter"]
  B --> C["Span to metrics step on all spans"]
  C --> D["Tail sampling"]
  D --> E["Export kept traces"]
  C --> F["Export metrics"]
        

Sampling changes the meaning of the numbers you see.

A sampled set of traces is not a faithful miniature of reality. Tail sampling in particular deliberately over-represents errors and slow requests. If you open your tracing tool and see that half of the traces have errors, that does not mean half of your requests fail. It means your sampling policy kept all the failures and only a trickle of the successes. Reading the data as though it were a random sample leads to badly wrong conclusions.

Re-weighting with recorded probability

To estimate real totals from a sample, you need to know how likely each trace was to be kept. A trace kept with a probability of one in twenty stands for about twenty traces. OpenTelemetry has been developing conventions for recording the sampling probability alongside the data, so that backends can scale counts back up correctly. With a mix of policies, some traces are kept with certainty and others with a small probability, so recording that information per trace becomes essential for any accurate arithmetic.

An honest balance sheet before you commit.

When it is worth it

Tail sampling pays off when your volume is high enough that you cannot keep everything, when failures are rare but costly, and when you have the engineering capacity to run a stateful Collector tier. It pays off less when traffic is small enough to store in full, or when your team has no appetite for operating extra infrastructure.

The sampling tier sees everything, so it deserves protection.

Before sampling, the Collector tier receives the full, unfiltered stream of spans, including attributes that may contain sensitive information such as user identifiers, internal addresses, or fragments of queries. This is a bigger concentration of data than a backend that only receives the sampled subset, so the sampling tier becomes an attractive target and should be treated as a sensitive system.

Sampling rules can be manipulated

Policies that depend on span attributes trust whatever the applications report. If a public-facing client can influence an attribute, such as a custom header that is copied onto a span, an attacker could make their traffic look interesting enough to be kept, or make real problems look boring. Sampling rules should rely on facts that your own services set, not on values controlled by outside callers.

Where organisations commonly apply it.

The following examples describe patterns that are widely used across the industry. They show how the same machinery of waiting, grouping, and judging can serve very different goals.

What works in practice, and the traps that catch experienced teams.

flowchart TD
  A["Can you afford to store every trace?"] -->|Yes| B["Keep everything and skip sampling"]
  A -->|No| C{"Do rare failures matter more than averages?"}
  C -->|No| D["Head-based sampling is probably enough"]
  C -->|Yes| E{"Can you run a stateful Collector tier?"}
  E -->|No| F["Use a managed tail sampling service or gentle head sampling"]
  E -->|Yes| G["Use two-tier tail sampling with pre-sampling metrics"]
        

Patterns that work

Anti-patterns to avoid

A system that decides what you can see must itself be watched closely.

The sampling tier fails in quiet ways. A misrouted trace, a buffer that is too small, or a wait that is too short will not crash anything. They simply cause the wrong traces to be kept, and nobody notices until an important investigation finds an empty search result. The Collector exposes metrics about its own behaviour, and the tail sampling processor reports on things like how many traces were kept and dropped, how many are sitting in memory, how long decisions take, and how many traces were evicted too early or arrived late. Make these part of your standard dashboards.

Short answers to the questions people ask most often.

It normally happens in an OpenTelemetry Collector, using the tail sampling processor from the contrib components. Applications export their spans, and the Collector decides which complete traces to keep.

Not in the usual sense. An SDK only sees the spans of its own service and makes decisions as spans start, so it cannot judge a whole multi-service trace. Whole-trace decisions need a place where all spans meet, which is why Collectors are used.

No. Applications still create and export every span so that the Collector can judge complete traces. Tail sampling reduces what reaches the backend, not what the applications produce.

Because the sampler must see all spans of a trace. The first layer routes spans by trace identifier, and the second layer holds and judges them. Without the routing layer, spans of one trace would scatter across instances.

The processor can remember recent decisions in caches. A late span is checked against the remembered outcome and is kept or dropped to match its trace. A trace dropped before an important span arrived cannot be recovered.

Slightly longer than the slowest normal request in your system, based on measurement rather than guesswork. A longer wait improves completeness but increases memory use and delays traces from reaching the backend.

Yes, but anything the head sampler drops never reaches the tail stage. Use a gentle head sample or none at all if catching rare failures matters, and make sure parent-based samplers do not spread a low keep rate to every service.

They will be wrong if you compute them from sampled traces. Compute metrics from all spans before the sampling stage, and use sampled traces only for investigation.

In the default behaviour, a trace is kept when any policy votes to keep it. Newer versions add ways to require several conditions together or to drop explicitly, so confirm the behaviour in the documentation for your version.

No. It keeps far better traces, but it costs more memory, more network traffic, and more operational effort. For small systems, or where simplicity matters most, head-based sampling or keeping everything can be the wiser choice.

Tail-based sampling answers a simple but powerful question: why decide what to keep before you know what happened? By collecting the spans of a trace, waiting for the story to finish, and then applying rules to the complete picture, it lets you keep every failure, every slow request, and every trace that matters to your business, while discarding most of the uneventful majority. In the worked example, the same budget that head sampling spent mostly on healthy traffic kept every error instead.

That power has a price. The sampling tier must hold traces in memory, receive every span of a trace in one place, and be watched carefully, because its failures are silent. Applications still create and send every span, and metrics must be computed before sampling or they will lie. Used with a two-tier layout, sensible waits, guard rails, and good monitoring, tail sampling turns an overwhelming flood of telemetry into a focused collection of the traces you actually need. Like the editor choosing photographs after the match, it works because it waits for the ending.