Before learning the tool, let us feel the problem it solves.

A quick reminder about traces

A trace is the complete story of one request as it travels through your system. A span is one timed step inside that story, such as “check stock” or “charge the card”. Spans are joined together by a shared trace ID, and together they make a tree that shows where the time went.

Why not record every span?

A busy system can create billions of spans every day. Each one must be created, filled with details, stored in memory, packed into a batch, sent over a network, stored on disk and later searched. Every step costs CPU, memory, network and money. Most spans come from boring, healthy, nearly identical requests. Keeping all of them is like filming every second of a quiet street for a year, just to catch the one moment when something happens.

So we need a way to keep only a share of traces. This is called sampling. The big question is: who decides, and when?

The earliest possible moment

The cheapest time to decide is at the very start. If we decide at the start that a trace will not be kept, we can skip almost all the work for it: no details to collect, no memory to hold, no bytes to send. The component that makes this early decision is the sampler. It is cheap and fast, but it has one big limit: at the start, it does not yet know how the request will end. It cannot know if the request will fail, be slow or be special. It must decide using only what is known at that moment.

The short answer first, and then the pieces.

A sampler is a small component inside the OpenTelemetry SDK, the engine that sits behind the API in your application. Its single job is to answer one question each time a new span is about to be created: “Should this span be recorded and kept?” It is attached to the tracer provider, which is the object that hands out tracers, and every tracer under that provider uses it.

What a sampler is not

• It is not in the API. The API only describes how code asks for spans. The sampler is part of the SDK.
• It is not a processor. A span processor works on spans after they exist and end. A sampler works before the span is even fully created. This is one of the most common confusions, and we will return to it.
• It is not tail sampling. Tail sampling happens later, usually in the Collector, after the whole trace is known. The sampler is the early, head-style decision inside the application.
• It does not affect metrics. Metrics recorded through the metrics API do not pass through the trace sampler.
• It is not a security filter. It decides which traces to keep, not what sensitive data they contain.

Let us follow what happens when your code starts a span.

sequenceDiagram
    participant Code as Your Code
    participant API as OpenTelemetry API
    participant Tr as Tracer in SDK
    participant Sm as Sampler
    participant Sp as Span
    participant Pr as Span Processor
    Code->>API: Start a span called charge card
    API->>Tr: Create span
    Tr->>Sm: Should this span be sampled? Here are the facts
    Sm-->>Tr: Decision plus optional extra attributes
    alt Decision is drop
        Tr-->>Code: Return a non-recording span with the same identity
    else Decision is record and sample
        Tr->>Sp: Create real recording span
        Tr-->>Code: Return recording span
        Code->>Sp: Add attributes and end the span
        Sp->>Pr: Hand over finished span for export
    end
    

What happens to a dropped span

When the answer is “drop,” the SDK does not throw an error and does not return nothing. It returns a lightweight non-recording span. Your code can still call it, add attributes and end it, and the calls simply do nothing. This is crucial for two reasons. First, your code never needs to check whether sampling is on. Second, the span still carries the trace identity, so the “not sampled” decision can be passed on to the next service.

The sampler runs before the span processor

A span processor, such as the batch processor, only ever sees spans that were created as recording spans and then ended. A span that the sampler dropped never reaches it. This is why a sampler can save so much more work than any later step: the dropped span never gets details, never joins a queue, and never reaches an exporter.

A sampler is like a function: facts go in, a decision comes out.

The inputs: the facts a sampler can use

When the SDK asks the sampler for a decision, it hands over a small bundle of facts about the span that is about to start.

The outputs: what a sampler returns

The sampler returns a small answer with up to three parts.

Samplers have a name

Every sampler also provides a short description string, such as “always on” or “ratio based with 0.1”. This is not used in decisions. It helps people see in logs and debug output which sampler is active, which is very handy when you are puzzled about why traces are missing.

Three answers, and a small flag that carries the answer across the network.

The sampled flag

The sampled flag is a single on or off switch inside the trace flags, which are part of the span’s identity, called the span context. It travels from service to service in the standard trace header. When it is on, downstream services learn “this trace is being kept.” When it is off, they learn “this trace is not being kept.”

Recording versus sampled: two ideas, not one

These two words sound alike but mean different things.

• Recording is a local fact. It means this span, inside this program, is collecting details such as attributes and events.
• Sampled is a shared fact. It is a message to the whole trace that says the trace will be kept and exported.

Most of the time the two go together. The special case “record only” separates them. It lets a program collect details for its own use, such as turning spans into metrics inside a span processor, without declaring that the trace should be exported or promising anything to other services.

Two samplers that never think. They always give the same answer.

When “always on” is the right answer

• Development and testing: you want to see everything while you learn.
• Low-traffic services: if a service handles a few requests a minute, keeping all traces costs almost nothing and gives complete visibility.
• Short-lived experiments: for a short debugging period, you may want full detail.

When “always off” is the right answer

• Emergency brake: if tracing is causing trouble, switching to always off stops it at once.
• Excluded areas: a custom setup may use “always off” for certain operations, such as health checks.
• Cost freeze: temporarily pausing trace collection while you fix a budget problem.

The workhorse for cutting volume in a simple and fair way.

The trace ID ratio sampler keeps a fixed share of traces, such as one in ten. It does so without needing a counter or any memory of earlier decisions. It uses the trace ID itself as a lottery number.

How the lottery works

A trace ID is a very large random number. Think of it as a ticket number in a lottery with an enormous range of possible tickets. The sampler picks a cut-off point. If you want to keep ten percent, the cut-off is set so that ten percent of all possible ticket numbers fall below it. For each new trace, the sampler checks whether the trace ID falls below the cut-off. If yes, keep. If no, drop.

flowchart LR
    A["New trace with random trace ID"] --> B["Turn part of the ID into a number"]
    B --> C{"Is the number below the cut-off for the chosen ratio?"}
    C -->|Yes| D["Record and sample"]
    C -->|No| E["Drop"]
    

Why this design is clever

• No shared memory needed. Every service can compute the same answer from the same trace ID, without talking to any other service.
• Even and unbiased. Because trace IDs are random, the kept traces are a fair mix, not favouring any route or customer.
• Very fast. A simple comparison of numbers, with no waiting and no network.
• Predictable. Over a large number of traces, the share kept is very close to the ratio you chose.

Important cautions about the ratio sampler

The most important sampler for any system with more than one service.

Imagine three services in a chain: a gateway, an order service and a payment service. If each service made its own independent decision, you would get traces with holes in them. The parent-based sampler solves this with a simple rule: if the span has a parent, follow the parent’s decision. If it does not have a parent, ask another sampler.

The five slots

The parent-based sampler is really a small switchboard with five slots. Each slot holds another sampler that is used in a specific situation. This is where its power, and sometimes its confusion, comes from.

flowchart TD
    A["New span is created"] --> B{"Is there a parent span context?"}
    B -->|No| R["Use the root sampler such as the ratio sampler"]
    B -->|Yes| C{"Is the parent remote?"}
    C -->|Yes| D{"Is the parent sampled?"}
    C -->|No| E{"Is the parent sampled?"}
    D -->|Yes| RS["Use the remote parent sampled slot"]
    D -->|No| RN["Use the remote parent not sampled slot"]
    E -->|Yes| LS["Use the local parent sampled slot"]
    E -->|No| LN["Use the local parent not sampled slot"]
    

Why “remote” and “local” are separated

A remote parent is a span that lives in another program and reached us through a request header. A local parent is a span inside the same program. The difference matters for trust. A local parent was created by your own code, so you trust its decision. A remote parent was created by someone else, possibly outside your company, so you may want to treat its “keep” with caution. This leads to a very useful trick, which we describe in the chapter about public edges.

sequenceDiagram
    participant G as API Gateway with ratio 10 percent
    participant O as Order Service with parent based
    participant P as Payment Service with parent based
    G->>G: No parent so root sampler rolls the lottery
    G->>O: Request with trace header and sampled flag on
    O->>O: Remote parent sampled slot says keep
    O->>P: Request with trace header and sampled flag on
    P->>P: Remote parent sampled slot says keep
    P-->>O: Response
    O-->>G: Response
    

Real systems use a few well-known combinations.

flowchart TD
    Q["Choose your sampler"] --> A{"Is traffic small enough to keep everything?"}
    A -->|Yes| B["Parent based with always on"]
    A -->|No| C{"Is this a front door service that starts traces?"}
    C -->|Yes| D{"Do all routes deserve the same rate?"}
    D -->|Yes| E["Parent based with a ratio root"]
    D -->|No| F["Parent based with a custom rule-based root"]
    C -->|No| G["Parent based, following the decision of callers"]
    

The “who owns the decision” idea

A clean design gives one place the job of starting traces and making the fresh decision, usually the first service that receives a user request. Every other service uses parent-based sampling and simply follows. This keeps traces whole, and it makes the system easy to reason about: to change the keep rate for the whole company, you change one setting at the front door.

Most of the time, you choose a sampler with standard settings.

OpenTelemetry defines standard environment variables, which are named settings handed to a program when it starts. Two of them control the sampler, and they work in nearly all languages and in auto-instrumentation agents.

How to read these names

• always_on and always_off are the plain samplers, with no parent logic.
• traceidratio is the ratio sampler on its own, with no parent logic.
• parentbased_ followed by a name means: use a parent-based sampler, with that named sampler as the root. So parentbased_traceidratio means “follow the parent, and use a ratio for new traces.”

Code-based and agent-based setups

• Manual SDK setup: when you build the SDK in code, you pass a sampler object to the tracer provider. This is also how you plug in a custom sampler or fine-tune the five parent-based slots.
• Auto-instrumentation agents: agents read the same environment variables and usually offer extra, language-specific settings for more advanced samplers.
• Operator-based setups: in Kubernetes, the OpenTelemetry Operator lets you set a sampler as part of the instrumentation definition, so that a team can control sampling for many services in one place.

When the built-in samplers are not enough, you can add your own rules.

The sampler is a small, well-defined piece, with a clear contract: facts in, decision out. This makes it easy to write your own, or to use ready-made ones from the OpenTelemetry contributions. A rule-based sampler looks at the facts and applies a list of rules, such as: “if the span is a request to the health-check address, drop it. If it is a checkout request, keep all of them. For everything else, keep ten percent.”

What rules usually look at

The golden rule: only start-time facts

A rule can only use facts that are known when the span starts. This leads to a common surprise. Web instrumentation libraries usually give the sampler the HTTP method and the request path when a server span starts. But some details, such as the final route template, the response status code and any error, are only known later, after the request has been handled. A rule that depends on them cannot work in a sampler.

Writing your own sampler: the rules of the road

• Be fast. The sampler runs on the path of every span, thousands of times a second. It must do only quick work, with no network calls, no file reads and no waiting.
• Be thread-safe. Many threads may ask for decisions at once. The sampler must work correctly without them stepping on each other.
• Be deterministic where possible. The same trace ID and facts should give the same answer, so that services can agree.
• Delegate to existing samplers. Build rules on top of the parent-based and ratio samplers rather than rewriting their logic.
• Be careful with state. A sampler that counts or limits rates needs shared state, which makes decisions less consistent across services and copies.
• Describe yourself clearly. Provide a helpful description string, so that debug output tells you which rule set is active.

What if the sampling rate could be controlled from a central place?

Restarting hundreds of services just to change a sampling rate is slow and risky. A remote sampler solves this by asking a central server, at regular intervals, “what rules should I use now?” The server answers with the current policy, and the sampler updates itself while the application keeps running.

sequenceDiagram
    participant App as Application with remote sampler
    participant Srv as Central sampling server
    participant Op as Operator or engineer
    Op->>Srv: Update the sampling policy for checkout
    App->>Srv: Poll for current policy every interval
    Srv-->>App: Return updated rules and rates
    App->>App: Apply new rules to future spans
    

Well-known remote sampling systems

Benefits and costs

A newer idea that makes sampling easier to combine and easier to count.

The plain ratio sampler has two small weaknesses. First, if two services use different ratios, their decisions may not nest neatly. Second, once a trace is kept, it is hard for later tools to know how likely each trace was to be kept, which makes it difficult to scale counts back up and estimate true totals.

Consistent probability sampling addresses both. The idea is that the sampler records, inside the trace state, a small piece of information that describes the sampling probability that was used, often as a threshold value, and sometimes a randomness value. Every later service and tool can read that note.

The sampled flag arrives in a request header. Anyone can write a header.

Inside your own network, trusting the incoming sampled flag is usually fine. But at the public edge, where requests arrive from the open internet, the trace header may have been written by anyone. A mischievous or careless caller could set the sampled flag to “on” on every request. If your front door simply obeys, you will record and store every one of those traces, and you will pay for them.

Using the five slots to protect the edge

This is exactly why the parent-based sampler has separate slots for remote parents. At your public edge, you can change what happens when the incoming parent is remote.

flowchart TD
    A["Public request arrives with trace header"] --> B{"Is the parent remote?"}
    B -->|Yes| C{"Is the incoming sampled flag on?"}
    C -->|Yes| D["Remote parent sampled slot uses a ratio or rate limit, not always on"]
    C -->|No| E["Remote parent not sampled slot stays off"]
    B -->|No| F["Root sampler decides as usual"]
    D --> G["Your own decision is stored in the new flag"]
    E --> G
    F --> G
    

Another option: start a fresh trace and link back

Some teams go further at the public edge. They ignore the incoming trace identity for sampling and parenting, start a brand new trace of their own, and record the caller’s trace identity as a link. This gives the caller a way to find the related trace, without letting them control your costs or trace structure.

The trace sampler touches only traces, but its effects spread further than you might expect.

Metrics

Metrics recorded through the metrics API are not affected by the trace sampler. A counter of orders still counts every order. But metrics derived from stored traces, such as request counts computed from spans, will be too low in proportion to how many traces you dropped. To avoid this, record real metrics in the SDK, or compute span-based metrics in a step that sees every span, before any tail sampling in the Collector.

Exemplars

Exemplars are example trace IDs attached to metric points, so that you can jump from a graph spike to a real trace. By default, SDKs usually attach exemplars only from spans that were sampled, because linking to a trace that was never stored would lead nowhere. So the sampler also decides which metric points can carry an exemplar.

Logs

Log records written while a span is active often carry its trace ID and span ID. If the sampler dropped that span, the log still carries the IDs, but there is no stored trace to open. This is normal. It is one more reason to keep more traces for important routes, since a log line with a link to nowhere is frustrating when you need it most.

flowchart LR
    S["Sampler decision on each span"] --> T["Traces: only sampled spans are exported"]
    S --> E["Exemplars: usually only from sampled spans"]
    S --> L["Logs: still carry IDs even when the trace is dropped"]
    M["Metrics API"] --> MB["Metrics: not affected by trace sampling"]
    

Both reduce data. They work at different times, with different strengths.

flowchart LR
    A["Application with parent-based sampler"] -->|"Only sampled spans are sent"| C["Collector"]
    C --> T["Tail sampling keeps errors and slow traces"]
    T --> B["Backend"]
    

An important warning about combining them

If the SDK sampler drops a trace, the Collector never sees it, so tail sampling cannot rescue it. If you want the Collector to keep every error, the SDK must not drop that trace at the start. Teams that use tail sampling for errors often set the SDK to keep everything, or to drop only obvious noise such as health checks, and then let the Collector do the real reduction. The price is that applications must record and send everything that is not dropped.

The sampler is on the path of every span, so it must be light.

A good sampler costs a tiny amount of time per span, usually a few simple comparisons. Even so, at very high volumes, small costs add up, so it is worth understanding where the savings and costs lie.

Practical tips

• Keep attributes passed at span start small and cheap. Add heavy details after creation, only if the span is recording.
• Many APIs let code check whether a span is recording before preparing costly details. Use this.
• Test a custom sampler under load, with and without it, before rolling it out.
• Prefer simple, stateless rules over complicated shared counters.

An honest look at what an SDK sampler gives and what it costs.

Key trade-offs

Sampling saves money. It does not make telemetry safe by itself.

If you cannot see what the sampler is doing, you cannot trust your traces.

• Kept versus total: compare the number of spans or traces created with the number kept. The real share should match your plan.
• Share of complete traces: if many traces have only one or two spans, a service may not be using parent-based sampling.
• Per-service keep rates: check each service on its own. One service with a very different rate is a clue.
• Sampler description in debug output: turn on debug logging briefly and read the active sampler’s description at startup.
• Edge behaviour: watch the share of requests with the incoming sampled flag on, and compare with the share you actually keep.
• Cost trend: a sudden jump in trace volume after a release often points to a sampler setting that was lost or changed.
• Error coverage: compare error counts from metrics with error traces you hold. If the gap is large, head sampling is dropping most errors.
• Remote sampler health: if used, track fetch failures and the age of the policy each instance is using.

Common symptoms and likely causes

Habits worth copying and traps worth avoiding.

Good design patterns

Anti-patterns

A practical checklist for a sampler you can trust.

Best practices

• Choose on purpose. Do not leave the default on a busy production system without deciding.
• Always use a parent-based sampler. It is the simplest way to keep traces whole.
• Pick a single decision owner. One front door sets the rate. Everyone else follows.
• Drop noise early. Use rules for health checks and other low-value traffic.
• Raise rates for what matters. Checkout, payments and sign-in are worth more than static pages.
• Protect the edge. Set remote-parent slots so that outsiders cannot force recording.
• Keep it fast and simple. Custom samplers should be quick, stateless and well tested.
• Check the arithmetic across layers. If you also use tail sampling, work out what survives the SDK first.
• Measure kept versus total. Confirm the real keep rate matches your plan.
• Document the policy. Write down the rates, the rules and the reasons, so people know why a trace may be missing.
• Review regularly. Traffic changes, and so should your rates.

Common mistakes beginners make

• Believing a ten percent rate means ten percent of problems are visible. Random sampling may miss rare errors completely.
• Using the ratio sampler alone and expecting it to follow the parent.
• Confusing a sampler with a span processor, or with Collector sampling.
• Writing rules that depend on facts not known at the start.
• Mixing up “recording” and “sampled.”
• Forgetting that changing environment variables requires a restart.
• Not realising that a dropped trace is gone forever.
• Assuming that sampling removes privacy risk.

How samplers show up in tools and platforms you may already know.

A typical journey for a growing company

Quick answers to questions beginners ask most.

If you remember only a few things, remember these.