Before learning the tool, let us feel the problem it solves.

A quick reminder about traces

A trace is the complete story of one request as it travels through your system. It is made of many small timed steps called spans. For example, when a customer places an order, the trace may include spans for the web gateway, the order service, the payment service, and several database calls.

The numbers get big very quickly

Suppose your online store handles one thousand orders every second. Each order creates a trace with about thirty spans. That is thirty thousand spans per second, which is more than two and a half billion spans every day. Storing, sending and searching all of them costs a lot of money, uses a lot of network and memory, and makes your tools slower.

And here is the surprising part: most of those traces are boring. They are fast, successful requests that look almost the same as each other. You do not need a billion copies of “everything was fine” to understand how your system works.

The short answer first, and then the pieces.

Sampling is the process of deciding which traces to keep and which to throw away. The goal is to reduce the amount of data while keeping enough of it to understand your system, find problems and make good decisions.

The decision is made by a piece of logic. In the SDK inside your application, this logic is called a sampler. In the Collector, it is done by processors designed for sampling. Either way, the key idea is the same: for each trace, ask “should we keep this?”

Why not just keep everything?

Why not keep nothing?

Because you would be blind. The art of sampling is finding a balance: enough data to see patterns and catch problems, but not so much that you drown in cost and noise.

Sampling has a small vocabulary. Learn these words and the rest is easy.

The three possible decisions

When a sampler looks at a new span, it can give one of three answers.

The simplest and cheapest kind of sampling, done inside your application.

Head sampling means the decision is made at the very beginning of the trace, when the first span is created. The sampler looks at a few facts available at that moment, such as the trace ID, the name of the span and its starting attributes, and then says keep or drop. The decision is then carried along with the trace to every other service.

flowchart LR
    A["Request arrives"] --> B["Root span is created"]
    B --> C{"Sampler decides"}
    C -->|Keep| D["Trace is recorded and exported"]
    C -->|Drop| E["Trace is not recorded"]
    D --> F["Decision travels to every service"]
    E --> F
    

The built-in samplers

Every OpenTelemetry SDK comes with a small set of ready-made samplers. Learning these four covers most real situations.

How the ratio sampler picks traces

The ratio sampler uses the trace ID as a kind of random number. The trace ID is a very large, random value. The sampler checks whether this number falls inside the chosen share. For example, if you want to keep ten percent, it keeps traces whose ID value falls in the lowest ten percent of all possible values. Because the decision depends only on the trace ID, any service using the same ratio will reach the same answer for the same trace, even without talking to each other.

Setting a sampler with simple settings

In most SDKs, you do not need to write code to choose a sampler. You use standard environment variables. One setting picks the sampler by name, such as always on, always off, trace ID ratio, or the parent-based versions of these. A second setting gives the ratio, like 0.1 for ten percent. The names are the same in nearly all languages, which makes it easy to manage them from a deployment template.

The most important sampler for anyone running more than one service.

Imagine three services in a row: a gateway, an order service and a payment service. If each one made its own random decision, the gateway might keep the trace, the order service might drop it and the payment service might keep it. The result would be a broken trace with missing pieces. To avoid this, we need one decision at the start, and everyone else must follow it.

The parent-based sampler does exactly this. It checks, “Does this span have a parent?” If it does, it copies the parent’s decision. If it does not, which means it is a new trace, it asks another sampler, usually the ratio sampler, to decide.

flowchart TD
    A["New span is created"] --> B{"Does it have a parent?"}
    B -->|No| C["Ask the root sampler such as the ratio sampler"]
    B -->|Yes| D{"Was the parent sampled?"}
    D -->|Yes| E["Keep this span too"]
    D -->|No| F["Drop this span too"]
    C --> G["Decision is stored in the sampled flag"]
    E --> G
    F --> G
    

sequenceDiagram
    participant G as API Gateway
    participant O as Order Service
    participant P as Payment Service
    G->>G: New trace. Ratio sampler says keep
    G->>O: Call with trace header and sampled flag on
    O->>O: Parent was sampled so keep
    O->>P: Call with trace header and sampled flag on
    P->>P: Parent was sampled so keep
    P-->>O: Response
    O-->>G: Response
    

Why this matters so much

Without parent-based sampling, traces become full of holes, which makes them very hard to trust. With it, a trace is either complete, from the first span to the last, or it is not there at all. This is why many guides say: in a system with more than one service, wrap your chosen sampler in a parent-based one.

Head sampling is simple and cheap, but it has one big blind spot.

When the head sampler decides, the request has only just started. It does not yet know if the request will fail, run slowly or do something unusual. So the choice is made blindly. If you keep ten percent at random, you also throw away ninety percent of your errors and ninety percent of your slow requests, and those are exactly the traces you most want to see.

The smarter, but more demanding, way to sample.

Tail sampling means waiting until a trace is finished, or almost finished, and only then deciding whether to keep it. Because the whole story is known, the rules can be much smarter. For example: “Keep every trace that has an error. Keep every trace slower than two seconds. Keep five percent of the rest.”

In OpenTelemetry, tail sampling is usually done in the Collector, using a special processor. The SDKs must send all their spans to the Collector, which holds the spans of each trace in memory for a short waiting time, and then applies the rules.

sequenceDiagram
    participant A as Applications
    participant C as Collector with Tail Sampling
    participant B as Backend
    A->>C: Send all spans of all traces
    C->>C: Group spans by trace ID and hold in memory
    C->>C: Wait for the decision time
    C->>C: Apply rules such as errors, slowness and a small random share
    C->>B: Send only the traces that passed
    

The kinds of rules you can write

The tail sampling processor works with a list of policies. A trace is kept if any policy says keep. The common kinds are:

The costs of tail sampling

• Memory: the Collector must hold every span of every in-progress trace until it decides. Busy systems need a lot of memory.
• Delay: data reaches the backend only after the waiting time, which may be many seconds.
• The waiting time dilemma: if you wait too short a time, late spans arrive after the decision and are lost or split off. If you wait too long, memory use and delays grow.
• Extra infrastructure: you must run, monitor and scale Collectors for this job.
• Still need to send everything: since the application cannot drop anything early, all spans must travel to the Collector first, so network and application overhead remain.

The number one reason tail sampling setups break.

For tail sampling to make a correct decision, all the spans of one trace must arrive at the same Collector instance. If a single Collector sees only half the spans of a trace, it may think the trace is fine, when the other half contains the error. Its decision will be wrong.

The solution, when you run more than one Collector, is a two-layer design. The first layer is a set of Collectors that act as a router. They use a special load-balancing exporter that looks at the trace ID of each span and sends all spans with the same trace ID to the same second-layer Collector. The second layer does the tail sampling.

flowchart LR
    A1["Service A"] --> R1["Routing Collector 1"]
    A2["Service B"] --> R2["Routing Collector 2"]
    A3["Service C"] --> R1
    R1 -->|"Route by trace ID"| T1["Sampling Collector 1"]
    R1 -->|"Route by trace ID"| T2["Sampling Collector 2"]
    R2 -->|"Route by trace ID"| T1
    R2 -->|"Route by trace ID"| T2
    T1 --> B["Backend"]
    T2 --> B
    

Most real systems use a mix, and the choice depends on volume, budget and what you need to find.

flowchart TD
    Q["How much trace data do you have?"] --> A{"Is cost and volume a real concern?"}
    A -->|No| B["Keep everything with the always on sampler"]
    A -->|Yes| C{"Must you keep all errors and slow traces?"}
    C -->|No| D["Use parent-based sampling with a ratio sampler"]
    C -->|Yes| E{"Can you run Collectors with enough memory?"}
    E -->|Yes| F["Use tail sampling in the Collector, often with light head sampling first"]
    E -->|No| G["Use head sampling and capture errors through metrics and logs"]
    

Combining both

A very common design uses both. Head sampling in the SDK removes a large share of obviously useless traffic early, such as health checks, saving cost in the applications. Then tail sampling in the Collector looks at what remains and keeps the errors and slow traces. This way, you get savings at the start and smart choices at the end.

Sampling traces can quietly affect metrics and logs. Here is how.

Metrics: do not count from sampled traces

Suppose you want a count of requests per minute. If you compute it by counting the traces you stored, and you only kept ten percent, your number is ten times too small. There are two safe ways to avoid this. One is to record metrics directly in the SDK, which is not affected by trace sampling. The other is to use a Collector feature that turns spans into metrics, and to place it before the sampling step, so it sees every span.

flowchart LR
    S["All spans arrive at the Collector"] --> M["Span to metrics step sees every span"]
    S --> T["Tail sampling decides which traces to keep"]
    M --> MB["Metrics backend with accurate counts"]
    T --> TB["Tracing backend with kept traces"]
    

Logs: the case of the missing trace

Log lines often carry a trace ID so you can jump from a log to its trace. But if that trace was sampled out, the link leads nowhere, because the trace was never stored. This is normal and expected. It is one more reason to keep errors, since an error log without its trace is frustrating. Some teams also sample logs in step with traces so that kept traces have their logs, and dropped traces do not leave a trail of orphan logs.

Counting correctly after sampling

If you keep one in ten traces, then each kept trace “stands for” ten. Some systems record the sampling probability along with the trace, so that tools can scale counts back up for estimates. If your tools do not do this for you, remember that numbers computed from sampled traces are only estimates, and write them down that way.

There is no single right number. Here is how to think about it.

Different rates for different things

One rate for everything is simple, but often wasteful. A checkout request is more valuable than a request for a static image. A service that gets ten requests an hour should be sampled at one hundred percent, since there is little data to start with. A service that gets ten thousand requests a second may need a very low rate. Tail sampling policies and attribute-based head sampling let you set different rates for different cases.

An honest look at what sampling gives and what it costs.

Key trade-offs

Sampling reduces data, but it does not make data safe by itself.

You must be able to see what your sampling is doing, or you will not trust your traces.

• Kept versus total: track how many traces or spans are created, and how many are kept. The ratio should match your plan.
• Share of complete traces: if many traces have only one or two spans, parent-based sampling may be missing in some service.
• Tail sampling health: the Collector reports metrics about traces held in memory, traces sampled, and traces dropped too early. These names start with the Collector’s name, then processor and tail sampling.
• Dropped too early: a rising count means traces are being pushed out before a decision, a sign that memory limits or waiting times are too small.
• Memory and CPU: watch the sampling Collectors closely, especially at traffic peaks.
• Routing balance: check that traces are spread fairly across the sampling Collectors.
• Error coverage: compare the error counts from metrics with the number of error traces you keep. They should be close if your policy is to keep all errors.
• Data delay: measure how long after a request the trace appears in the backend. Tail sampling adds the waiting time.

Common symptoms and likely causes

Habits worth copying and traps worth avoiding.

Good design patterns

Anti-patterns

A practical checklist for sampling you can trust.

Best practices

• Choose on purpose. Do not leave the default sampler on production without deciding.
• Use parent-based sampling. Make sure all services follow one decision per trace.
• Always keep errors and slow requests. If you can run tail sampling, make this your first policy.
• Keep a baseline. Retain a small random share of normal traces to understand healthy behaviour.
• Sample by value. Give important routes and customers higher rates than unimportant ones.
• Protect metric accuracy. Compute counts and rates before sampling or in the SDK.
• Put sampling after filtering and before batching in the Collector. Remove noise first, choose what to keep, then pack the data.
• Size tail sampling carefully. Set the waiting time and trace limits using real traffic, and add a memory limiter.
• Measure kept versus total. Check regularly that the real keep rate matches your plan.
• Write down your policy. Record the rates, rules and reasons, so teammates know why a trace may be missing.

Common mistakes beginners make

• Believing that a ten percent rate means “we see ten percent of the problems.” Random sampling may miss rare errors entirely.
• Forgetting that sampled-out traces are gone, and then being unable to debug a specific complaint.
• Using different sampling ratios in different services without parent-based sampling.
• Confusing “recording” with “sampled.”
• Setting the tail sampling waiting time too short, so late spans are lost.
• Forgetting that tail sampling needs all spans of a trace in one place.
• Assuming sampling removes privacy risk.

How sampling shows up in tools and platforms you may already know.

A typical journey for a growing company

Quick answers to questions beginners ask most.

If you remember only a few things, remember these.