Consistent Probability Sampling in OpenTelemetry

Consistent Probability Sampling in OpenTelemetry

A complete, plain-English guide to how every service in a system can agree on which traces to keep, using one shared random number and one simple threshold, so sampled traces stay whole and the numbers you compute from them stay honest.

Picture a courier company that moves parcels through five warehouses. Management wants a photo record of the journey of some parcels, but photographing everything would fill every hard disk in the company. So each warehouse manager decides on their own, by flipping a coin, whether to photograph the parcel in front of them. At the end of the month, management opens the album hoping to follow a parcel from the first warehouse to the last, and discovers that almost no parcel has photos from all five stops. Most have one photo, a few have two, and the story of any single journey is full of holes. The photos are not useless, but they cannot answer the question that matters: what happened to this parcel from start to finish?

This is exactly the situation that distributed tracing faces when many services each decide independently what to keep. Consistent Probability Sampling is the OpenTelemetry approach that fixes it. Instead of every warehouse flipping its own coin, every warehouse looks at the same lucky number written on the parcel label and compares it with its own personal cut-off line. Because everyone reads the same number, the decisions line up. In this tutorial we will build the idea from the ground up, one small step at a time, so that you can explain it to a teammate, reason about it in a design review, and recognise it when you see it in a trace header.

1Why Sampling Exists at All

Before we talk about doing sampling consistently, we need to be clear about why we sample in the first place.

Modern software is rarely one program. A single click on a shopping website may touch a web gateway, a login service, a catalogue service, a pricing service, a payment service, and several databases. To understand what happened during that click, engineers record a trace, which is the full story of the request, made of many small pieces called spans. Each span describes one step, such as one function call, one database query, or one network request, and records when it started, how long it took, and whether it succeeded.

Recording every span for every request is the dream of every engineer and the nightmare of every finance team. A busy system can create millions of spans every minute. Each span has to be built in memory, sent over the network, received by a collector, processed, stored, indexed, and later searched. Each of those steps costs processor time, network bandwidth, disk space, and money. At large scale, the bill for keeping everything can rival the bill for running the actual product.

Millions
of spans per minute in a busy system
5 costs
CPU, network, storage, indexing, query
1 lever
keep a representative subset
Simple Analogy

Think of a national opinion poll. Nobody asks every citizen what they think about a new law, because that would take years and cost a fortune. Pollsters ask a carefully chosen small group and use the answers to describe everyone. Sampling in observability works the same way: you keep a small, representative part of your traces and use it to understand the whole.

The key promise of sampling is representativeness. If you keep one trace out of every hundred, and you keep them fairly, then what you see in the kept traces should resemble what happens in all of them. You can still find slow requests, still see typical error patterns, and still estimate how many requests of each kind happened. The word fairly is doing a lot of work in that sentence, and the rest of this tutorial is about what fairness means when many independent programs have to cooperate.

i
Good to know

Sampling is a cost and quality lever, not a bug fix. Done well, you keep nearly all the insight for a small fraction of the cost. Done badly, you keep the cost of tracing and lose the ability to explain what happened.

2The Vocabulary You Need First

Every new term below comes with what it is, why it exists, where you meet it, and a picture to hold in your head.

Term 1

Trace

The whole journey of one request through a system. It exists so humans can follow cause and effect across many programs. Like a parcel’s complete delivery history.

Term 2

Span

One timed step inside a trace. A trace is a family tree of spans, with a root span at the top and children below. Like one stamp in a passport.

Term 3

Sampling probability

The chance that a span is kept. A probability of 0.25 means roughly one span in four survives. It is usually written as a number between zero and one.

Term 4

Sampler

The small piece of logic that decides keep or drop. It lives inside the tracing library of each program and, later, inside collectors.

Term 5

Head sampling

A decision made when a span begins, before anyone knows how the request will end. Cheap and fast, but it cannot see errors that come later.

Term 6

Tail sampling

A decision made after spans finish, usually in a collector, so the decision can use what actually happened, such as slowness or failure.

Term 7

Adjusted count

How many real spans one kept span stands for. If one in a hundred is kept, each kept span counts as one hundred. It is simply one divided by the probability.

Term 8

Consistent decision

A set of decisions that never contradict each other in a harmful way. The precise meaning is given in Chapter 8.

A word of caution about the word “head”

The word head sampling is used in two slightly different ways in the industry. Sometimes it means the exact moment a tracing library creates a new span and decides whether to record it. Other times it describes a whole style of tracing setup in which every child follows its parent’s decision, so the choice made at the very start of the trace governs everything after it. When someone says they use head sampling, it is worth asking which meaning they have in mind. Tail sampling, similarly, may mean thinning individual spans as they pass through a collector, or it may mean waiting to assemble a whole trace and then deciding about the entire thing at once.

!
Common confusion

Consistent probability sampling is not a rival of head or tail sampling. It is a rule for how probability-based decisions are made and recorded, and it can be used by head samplers and by downstream samplers alike.

3The Problem: Independent Coin Flips Break Traces

To appreciate the cure, let us feel the pain of the disease with real numbers.

Suppose a request travels through five services, and every service keeps a span with a probability of fifty percent, deciding by its own private coin flip. For the trace to be complete, all five coins must come up heads. The chance of that is one half multiplied by itself five times, which is about three percent, or roughly one trace in thirty-two. The other thirty-one traces are partial: they have some spans but not others, which means they are full of missing pieces and broken parent-child links.

It gets dramatically worse as the keep rate falls. Real systems rarely keep half of their traces. If each of the five services keeps ten percent, the chance that all five happen to keep the same request is one in a hundred thousand. You would pay to store spans from almost every request, yet nearly every stored trace would be unusable.

Keep rate per serviceServices in the request pathChance the whole trace survivesWhat you get
50%5About 3% (1 in 32)Mostly fragments
10%5About 0.001% (1 in 100,000)Almost no whole traces
10%10One in ten billionEffectively none
10% with consistent samplingAny number10%One in ten traces is whole
flowchart LR
    A["Gateway: coin flip says KEEP"] --> B["Login service: coin flip says DROP"]
    B --> C["Catalogue service: coin flip says KEEP"]
    C --> D["Payment service: coin flip says DROP"]
    D --> E["Database: coin flip says KEEP"]
        
FIG 1 — Independent decisions leave holes: three spans kept, two missing, and the trace tells a broken story

There is a second, quieter problem. Even when you do not care about complete traces, you often want to count things from sampled data, for example to estimate how many payments were attempted. If different services keep spans at different rates and the data does not record those rates, any total you compute will be wrong in an unknowable way. A trustworthy count needs to know the keep probability that applied to each span.

The two pains in one sentence

Independent sampling produces broken stories (incomplete traces) and untrustworthy arithmetic (unknown probabilities). Consistent probability sampling is designed to cure both with a single mechanism.

4The Core Idea in One Page

The whole mechanism rests on two ingredients and one comparison.

The first ingredient is a shared random number, called the randomness value. It is created once, when the trace begins, and every participant that touches the trace can read it. The second ingredient is a personal cut-off line, called the rejection threshold. Each participant chooses its own cut-off according to how much data it wants to keep. The comparison is a single question: is the shared number at or above my cut-off? If yes, keep the span. If no, drop it.

Simple Analogy

Imagine a school where, on the first day, every student is handed a ticket with a random number from zero to one hundred. For a photo day, the art club says “we photograph students whose ticket is 90 or higher” and the sports club says “we photograph students whose ticket is 50 or higher”. Nobody has to talk to each other. Yet every student the art club photographs is also photographed by the sports club, because 90 or higher always means 50 or higher. The ticket is the shared random number, and the club rules are the thresholds.

1

A trace begins and a random number is fixed

The first service creates the trace and makes sure a 56-bit random value exists for it, either hidden inside the trace identifier or written explicitly into the trace headers.

2

Each sampler turns its keep rate into a threshold

A sampler configured to keep ten percent computes the matching cut-off once and reuses it. Keeping everything means a cut-off of zero.

3

The comparison decides

Random number at or above the threshold means keep. Anything lower means drop. There is no extra coin flip and no extra communication.

4

The threshold is recorded for kept spans

The cut-off that was applied is written into the trace headers, so later stages and the final storage know the exact keep probability for that span.

“Everybody reads the same number, so everybody’s decisions line up.”

Notice how little is required. There is no central coordinator, no shared database, and no network call to ask what others decided. The only shared thing is a number that travels with the request anyway, inside the trace context. This is what makes the idea practical: it adds almost no cost to the request, yet it gives every participant the power to make decisions that fit together.

5The Randomness Value: The Shared Lucky Number

Everything depends on this number being truly random and truly shared, so let us look at where it comes from.

The randomness value, often written as R, is a 56-bit number. A bit is a single zero or one, so 56 bits give about seventy-two quadrillion possible values. That is a very large range, which matters because the range limits how finely you can express a keep rate. With 56 bits, you can sample at probabilities as small as two raised to the power of minus fifty-six, far smaller than anyone realistically needs.

OpenTelemetry supports two places where this number can live.

Source A

The trace identifier itself

Every trace has a 16-byte identifier. The last seven bytes, which are exactly 56 bits, can serve as the randomness value, provided the identifier was generated randomly. This costs nothing extra on the wire.

Source B

An explicit value in the trace state

A random 56-bit number can be written into a sub-key called rv inside the OpenTelemetry part of the trace state header. It is used when the identifier cannot be trusted to be random.

Why the trace identifier can be used

The W3C Trace Context standard, in its second level, defines a flag that a tracing library can set to promise that the trace identifier’s last seven bytes were generated randomly. This is called the trace random flag. When the flag is present, any downstream participant can safely read those seven bytes as the shared lucky number without any additional header. When it is absent, a participant cannot be sure, so it should not assume randomness and may rely on an explicit value instead.

Simple Analogy

A trace identifier is like a parcel tracking number. If the post office promises that the last seven digits are assigned by a fair lottery machine, you can use those digits as your lucky number. If the post office assigns numbers in order, like one, two, three, then those digits are not lottery numbers, and you would rather write a fresh lottery number on the label.

Who is allowed to create or change the number

The rule is strict and simple: only the root sampling decision, the very first one in a trace, may create or change the explicit randomness value. Everyone after that must pass the same value along unchanged. If a middle service replaced the number, then services before and after it would be comparing their thresholds against different numbers, and the neat agreement would be destroyed. A consistent system is only as consistent as its most careless participant.

!
Common mistake

Using identifiers that are not random, such as ones built from timestamps or counters, as a source of randomness. Sampling would then be biased: some kinds of requests would be kept much more often than others, and any statistics computed from the kept data would be skewed.

Sharing luck across several traces

Because the randomness value can be explicit, the same value can be given to several independent traces on purpose. For example, all traces belonging to one user session could receive the same number, so that a session is either mostly kept or mostly dropped as a unit. The root decision is the only place where this is allowed to happen.

6The Rejection Threshold: Your Personal Cut-Off Line

If the randomness value is the lucky number, the threshold is the line you draw on the number ruler to separate keep from drop.

The rejection threshold, often written as T, is also a 56-bit number, and it is derived directly from the keep probability a sampler wants. The name tells you its meaning: it is the count of values, out of all the possible ones, that will be rejected. If you want to keep a quarter of spans, you reject three quarters of the number range, so the cut-off sits three quarters of the way up the ruler. Everything below that line is dropped, and everything at or above the line is kept.

In words, the formula is short: take one minus the keep probability, and multiply it by the size of the whole range, which is two raised to the power fifty-six. The extreme cases are easy to remember. Keeping everything gives a threshold of zero, because nothing is rejected. Keeping a tiny sliver gives a threshold very close to the top of the range.

Simple Analogy

Think of a doorway with a height bar, like the one at a fairground ride. A tall bar lets very few people in, and a bar set at floor level lets everyone in. The threshold is the height of that bar. The randomness value is each visitor’s height. The rule is simple: if you are at least as tall as the bar, you may enter.

Valid range and the “never” trap

Sampling probability in OpenTelemetry is valid from two raised to the power of minus fifty-six up to exactly one. Zero is deliberately not part of the range. The reason is conceptual: a rule that never keeps anything is not a probability sampler at all, because it keeps no data from which anything could be estimated. A sampler that drops everything is a perfectly legitimate configuration choice, but it is not a form of probability sampling and it does not carry a threshold.

How a threshold looks when written down

Thresholds are written as hexadecimal numbers, which is a compact way of writing big numbers using the digits zero to nine and the letters a to f. Trailing zeros are removed to save space, which also means the writer can choose how precise to be. Here is how common keep rates look when written with four digits of precision.

Keep one inKeep probabilityWritten thresholdWhat it tells you
1 (everything)100%0Nothing is rejected
250%8Half the range is rejected
425%cThree quarters rejected
812.5%eSeven eighths rejected
1010%e666Not a power of two, so digits repeat
1001%fd70aCut-off very near the top
10000.1%ffbe77More leading f digits as keep rate shrinks

Two patterns are worth noticing. Powers of two, like one half or one quarter, produce short, clean thresholds. Other rates, like one in ten, never divide evenly into binary, so the digits repeat and the writer has to decide where to stop. And as the keep rate gets smaller, the threshold starts with more and more f characters, which is just the number creeping closer to the top of the range.

i
Precision tip

The specification suggests that tracing libraries use four digits of precision by default. That is plenty for sampling, since nobody needs to distinguish a keep rate of ten percent from ten percent plus a few parts per million.

7The Decision Rule, Worked Through by Hand

The rule is one comparison, and it is worth practising it on a friendly small ruler before trusting it on a giant one.

The entire decision is this: if the randomness value is greater than or equal to the threshold, keep the span; otherwise drop it. To make the arithmetic easy, let us shrink the giant 56-bit range into a simple ruler from zero to one hundred. A keep probability of fifty percent means a threshold of fifty. A keep probability of twenty percent means a threshold of eighty, because eighty percent of the ruler is rejected. A keep probability of ninety percent means a threshold of ten.

Now imagine four requests arriving, each with its own random number: 12, 55, 83 and 97. We have three samplers, with different keep rates. Look at which requests each one keeps.

Request random numberSampler A keeps 90% (threshold 10)Sampler B keeps 50% (threshold 50)Sampler C keeps 20% (threshold 80)
12KeepDropDrop
55KeepKeepDrop
83KeepKeepKeep
97KeepKeepKeep

Read the table column by column and then row by row. Sampler C keeps the fewest requests, and every request it keeps is also kept by B and by A. Sampler B keeps more, and everything it keeps is also kept by A. There is no request that C keeps while B drops. This staircase pattern, in which the more selective sampler always keeps a subset of what the less selective sampler keeps, is the mathematical heart of the whole system.

flowchart TD
    S["Span arrives with shared random number R"] --> Q{"Is R at or above my threshold T?"}
    Q -->|"Yes"| K["Keep the span and record T"]
    Q -->|"No"| D["Drop the span"]
        
FIG 2 — The complete sampling decision: one comparison, no extra randomness
i
Why this is cheap

The decision involves one comparison of two integers. There is no random number generation at decision time, no lookup, and no network call. Even a service handling enormous traffic can afford to do this for every span.

Does the fairness hold?

If the random number is truly uniform, meaning every value is equally likely, then the fraction of requests with a number at or above the threshold equals the keep probability. For the ninety-percent sampler above, ninety out of every hundred requests will have a number of ten or more. The rule therefore delivers exactly the keep rate the operator asked for, on average, while making the decisions line up perfectly across different samplers.

8The Consistency Guarantee, Stated Precisely

Now that we have seen it in action, we can say exactly what consistent means.

The specification defines a consistent sampling decision like this: if a span is kept by a sampler using probability p1, then every span belonging to the same trace is also kept by any sampler using a probability p2 that is greater than or equal to p1. In plain language, the keep decisions are nested, like a set of Russian dolls. The group kept by the strictest sampler fits inside the group kept by the next strictest sampler, which fits inside the group kept by the most generous one.

Simple Analogy

Imagine water rising in a tank with marks on the wall. A mark at ten litres is passed before a mark at fifty litres. Any moment when the water is above the fifty-litre mark that is also above the ten-litre mark is impossible; the reverse is what always holds: whenever the water reaches the higher mark, it has already passed every lower mark. Thresholds behave the same way along the number ruler.

What this buys you in practice

Benefit 1

Whole traces

If every service in a request path uses the same rule, the whole trace is kept or the whole trace is dropped. The cost of a trace that is kept at rate p is that you really do get complete traces at rate p.

Benefit 2

Different rates, still coherent

A busy backend can use a low keep rate and a small gateway can use a high one. The backend’s spans are always a subset of the gateway’s, so what you keep never contradicts itself.

Benefit 3

Safe further thinning

A collector can drop even more data later by raising the threshold, and the result remains consistent with everything that was decided earlier.

Benefit 4

Known probabilities

Because the threshold is recorded, the exact keep probability of each kept span can be reconstructed and used for counting.

graph TD
    A["All spans"] --> B["Kept at 50 percent: threshold 8"]
    B --> C["Kept at 25 percent: threshold c"]
    C --> D["Kept at 10 percent: threshold e666"]
    D --> E["Kept at 1 percent: threshold fd70a"]
        
FIG 3 — Nested sets: every stricter sampler keeps a subset of the more generous one
“Stricter samplers never keep something that a more generous sampler dropped.”

One more point deserves emphasis. Consistency does not require every service to use the same keep rate. It only requires that they all compare against the same random number. That freedom is what lets teams tune cost per service while still ending up with traces that make sense when they are stitched back together.

9How the Information Travels: The Trace State Header

Numbers are only useful if they reach the next participant, so we need a place to carry them.

Distributed tracing already passes a small packet of context from service to service with every request. One part of that packet, defined by the W3C Trace Context standard, is the trace state, a list of vendor-specific entries. Each entry has a short key naming the vendor or project and a value that the owner is free to define. OpenTelemetry reserves the key ot for itself, and inside that value it defines small named sub-keys. Two of them matter for sampling.

Sub-key

th (threshold)

Records the rejection threshold that applied when a span was sampled. It exists so later stages and the final backend know the keep probability. It is only written for spans that were kept.

Sub-key

rv (randomness value)

Records an explicit 56-bit random value, used when the trace identifier cannot be trusted to be random. Once written by the root, every later participant passes it on unchanged.

Three details that confuse beginners

1

Only kept spans carry a threshold

A dropped span is never exported, so there is nobody to read its threshold. The header only needs to describe spans that survive.

2

Trailing zeros are removed

A threshold for a one percent keep rate, written to five digits, appears as fd70a4, not fd70a400000000. Fewer digits means less header weight and a way to express how precise the value is.

3

Missing threshold means unknown probability

If a kept span has no threshold, it means we do not know what probability applied, perhaps because a sampler outside this scheme made the decision. Such spans can be stored but cannot safely be used for counting.

!
Do not confuse the two numbers

A threshold of zero is written as the single digit zero and means one hundred percent, keep everything. It does not mean the span was dropped. Mixing up a low threshold with a low keep rate is one of the most frequent reading mistakes.

Because the information lives in an ordinary header that already travels with every request, no new network path is introduced. Services that do not understand the OpenTelemetry entry simply pass it along, which is the usual behaviour for trace state entries, so mixed systems keep working even when only some participants are upgraded.

10Sampling Stages: Two Moments, Two Dimensions

A span can be judged more than once in its life, and understanding when helps you place each sampler in the picture.

The specification describes two kinds of stages. The first is parent and child sampling, which happens inside the tracing library while a request is still running. These decisions follow the direction of cause and effect: the first span decides, and its children usually follow. The second is downstream sampling, which happens after spans have finished and are travelling through a collection pipeline of agents and gateways. A finished span has therefore been judged exactly once by a parent and child sampler and zero or more times by downstream samplers.

flowchart LR
    subgraph SDK["Inside the application: parent and child stage"]
        R["Root span decides"] --> C1["Child span follows"] --> C2["Grandchild follows"]
    end
    subgraph PIPE["Collection pipeline: downstream stage"]
        AG["Agent collector"] --> GW["Gateway collector"] --> ST["Storage backend"]
    end
    C2 --> AG
        
FIG 4 — Two stages: decisions inside the application, then decisions along the collection path

Why the two stages are treated differently

Inside the application, decisions are made live, while the request is in flight, and they can only use information that exists at that moment. Downstream, spans are historical records. A collector can look at a finished span, notice that it was slow or failed, and make a smarter choice, but it must also respect a rule that does not apply inside the application: it may only raise the threshold, never lower it. We will see why in Chapter 13.

Simple Analogy

Think of a newspaper. Reporters in the field decide in the moment what to write down. Later, editors in the office look at everything that came in and cut it down to fit the page. An editor can remove a story, but cannot add a quote the reporter never wrote. In the same way, a downstream sampler can thin the data further but can never resurrect what an earlier stage threw away.

Combining both stages

A very common design keeps a modest share at the application level, for instance ten percent of traces, and then lets a collector pipeline apply extra thinning or special rules. Because all stages read the same random number and record their thresholds, the final kept data still has a precisely known probability for each span.

11Architecture and Components: Who Does What

Before diving into the finer rules, let us map the cast of characters and the job each one performs.

Consistent probability sampling is not one single program. It is a shared agreement among several components, each doing a small, well-defined job. Seeing all of them on one page makes the later rules much easier to follow.

Component 1

Tracing library (SDK)

Lives inside each application. Creates spans, makes sure randomness exists for the trace, runs the sampler, and writes the threshold into outgoing context.

Component 2

Sampler

The decision maker inside the library. Compares randomness with its threshold and returns keep or drop, along with the threshold to record.

Component 3

Context propagator

Carries the trace identifier, flags and trace state from one service to the next, so the shared number and recorded thresholds travel with the request.

Component 4

Collector agent

A small process near the application that receives spans, can apply downstream sampling, and forwards what remains.

Component 5

Collector gateway

A central collection point that sees data from many sources and often applies stricter or smarter sampling before storage.

Component 6

Storage and analysis backend

Stores kept spans along with their thresholds and uses them to show traces and to estimate counts and rates.

flowchart LR
    subgraph APP["Application process"]
        SP["Span creation"] --> SM["Sampler: compare R with T"]
        SM --> EX["Exporter"]
    end
    SM --> PR["Propagator writes context with th and rv"]
    PR --> NEXT["Next service in the request path"]
    EX --> AGT["Collector agent"]
    AGT --> GWY["Collector gateway"]
    GWY --> BE["Backend: store spans with thresholds"]
        
FIG 5 — Components and the direction of information flow
Simple Analogy

Think of an airport. Check-in counters (the libraries) tag every bag. Security lines (collectors) can remove bags that look unnecessary but cannot change the tag. The baggage hall (the backend) reads the tags to know which flight each bag belongs to and how it was handled. The system works because everyone trusts the same tag format.

12The Base Cases: Root, Parent-Based and Probability Samplers

Real configurations are built out of a few simple building blocks, each behaving differently with respect to the shared number.

Within the parent and child stage, the specification identifies three primary base cases. Learning these three gives you the vocabulary to read almost any sampler configuration.

Root sampling

The very first decision in a trace. There is no earlier threshold to respect, so it always decides on its own. It is also the only place allowed to create or modify the explicit randomness value. If the trace identifier is already trusted as random, nothing extra needs to be written.

Parent-based sampling

A child span copies the decision of its parent. The parent and child end up with matching thresholds, which is how a whole trace inside one application stays together. This is the most common choice for non-root spans.

Consistent probability sampling

A sampler that applies its own fixed probability regardless of what the parent decided. Always-on and always-off behaviours belong to this family, as does the ratio-based sampler that keeps a configured fraction. Such a sampler ignores the parent’s threshold, which can lead to incomplete traces if the rates are mixed carelessly, but when randomness is shared the results still nest consistently.

Choosing among them

SituationTypical choiceWhy
The entry point of a request, such as an API gatewayRoot with a ratioSets the keep rate for the whole trace
Internal services receiving a call from a trusted callerParent-basedFollows the entry point so traces stay complete
A service that must always be traced, such as a payment stepAlways-on probabilityGives complete visibility, accepting the cost
A noisy background job with no value in tracingAlways-offRemoves cost, but it is not probability sampling and carries no threshold
!
Common mistake

Putting a ratio-based sampler on every service as if it were the root. Each service then makes its own independent keep decision. With shared randomness the decisions nest, but different rates still produce traces with missing branches. Reserve independent samplers for places where partial traces are acceptable.

13Downstream Sampling: Raising the Bar After the Fact

Collectors can thin data further, but they must follow one rule above all others: the threshold can only go up.

Why can a downstream sampler raise a threshold but never lower it? Because lowering would be a lie. A span that arrives with a recorded threshold has already survived a stage with a particular keep probability. Pretending that the stage had been more generous than it really was, by lowering the recorded number, would claim that other spans had been kept when they were in fact thrown away earlier. Any count computed from such a claim would be too small. Raising the threshold, on the other hand, honestly records that an extra round of thinning happened.

The specification names two standard styles of downstream probability sampler.

The equalizing sampler

An equalizing sampler wants every span leaving it to carry the same threshold. If an incoming span already has a higher threshold than the sampler’s own, the span cannot be equalized, so it passes through unchanged. If the incoming threshold is lower, the sampler applies its own threshold: spans whose random number is at or above it are kept and marked with the new threshold, and the rest are dropped. The effect is to bring mixed data to a common keep rate, which makes later analysis simpler.

The proportional sampler

A proportional sampler wants to reduce traffic by a fixed factor regardless of what arrives. It multiplies the incoming keep probability by its own configured probability to compute a new, stricter threshold for each incoming span, and then applies the comparison. If the multiplied probability would fall below the smallest representable one, the span is not kept.

Incoming keep rateEqualizing sampler set to 10%Proportional sampler set to 20%
100% (threshold 0)Outgoing rate 10%Outgoing rate 20%
50%Outgoing rate 10%Outgoing rate 10%
10%Outgoing rate 10%, nothing more droppedOutgoing rate 2%
5%Passes through unchanged at 5%Outgoing rate 1%
Simple Analogy

An equalizing sampler is like a teacher who says “every student’s essay will be cut down to one page”, leaving short essays alone and trimming long ones. A proportional sampler is like a teacher who says “every essay will lose half its length”, so a long essay stays longer than a short one but both shrink by the same ratio.

i
When to pick which

Choose equalizing when you want uniform data for analysis. Choose proportional when you need a guaranteed reduction in traffic volume, for example to protect a downstream storage budget.

14The Rules That Keep the Numbers Honest

A handful of rules, if followed by every participant, protect both whole traces and trustworthy statistics.

The specification lists general requirements for every stage that touches the sampling threshold. They look dry at first, but each one blocks a specific way things can go wrong. Here they are in plain language.

1

Whoever changes the threshold must record the change

If a stage raises the effective threshold, it must re-encode the trace state so that the new value is visible. Otherwise the recorded probability would not match reality.

2

Do not let the rule depend on the shared number

A conditional stage, such as one that keeps errors more often, must not base its condition on the randomness value or anything derived from it. It may use independent randomness. Peeking at the shared number would break the even spread that the arithmetic relies on.

3

Erase the threshold when probability is unknown

A stage that produces spans of unknown probability, such as a parent-based sampler that receives context with no threshold, must remove the threshold in its output instead of inventing one.

4

Check for obvious inconsistency

When it is a cheap test, a stage should verify that a sampled span really satisfies the rule that its random number is at or above its recorded threshold, and erase the threshold if it does not. This catches bugs and bad inputs early.

5

Pass the explicit randomness value forward unchanged

If randomness came from an explicit value, the same value must appear in the outgoing trace state. Only the root decision may alter it.

ANTI-PATTERN-01 Avoid
Problem

A rule-based stage wants to keep slow requests more often, so it uses the shared random number itself to decide how much extra to keep.

Why It’s Harmful

The shared number is supposed to be spread evenly in every slice of the data. Once a rule filters data by that number, the surviving spans no longer form an even spread, and the recorded probability no longer matches the real chance of being kept. Counts computed from the data become biased without any visible warning.

Correct Approach

Let the rule choose its own threshold based on the condition, such as slow or failed, and compare that threshold with the shared number. Where extra randomness is needed, draw it from a separate, independent source.

15Adjusted Count: Turning Samples Back into Totals

This is the payoff that makes the recorded threshold so valuable: you can estimate what you did not keep.

The adjusted count of a kept span is one divided by its keep probability. If a span was kept at a probability of one percent, then it stands for roughly one hundred real spans, so its adjusted count is one hundred. If it was kept at fifty percent, its adjusted count is two. To estimate how many requests of some kind happened, you add up the adjusted counts of the kept spans of that kind, rather than counting spans one by one.

1 ÷ p
adjusted count of one kept span
100
what a span kept at 1% represents
2
what a span kept at 50% represents
Simple Analogy

Suppose a school survey asks every tenth student a question and eighteen of the students asked say yes. You would estimate that about one hundred and eighty students in the whole school would say yes, because each answer represents ten students. Adjusted count is that multiplication, done automatically for every span.

Mixed probabilities are fine, as long as they are recorded

A backend might hold spans kept at fifty percent from a quiet service and spans kept at one percent from a very busy one. A naive row count would give the quiet service far too much weight. Summing adjusted counts corrects for this automatically, because each span carries a number that reflects how many others it stands for. That is why the recorded threshold matters even for traces nobody ever opens: it powers the numbers on dashboards.

Turning traces into metrics

Using adjusted counts to derive metrics such as request rates and error rates from sampled spans is sometimes called span-to-metrics. The rules in Chapter 14 exist so that these derived numbers are reliable. If a stage secretly changed its keep rate without updating the threshold, every rate computed afterwards would be quietly wrong.

!
What to avoid

Adjusted count is not defined for kept spans that carry no threshold. Never fill in a guess. Leave such spans out of count calculations, or report them separately as spans of unknown weight.

i
Precision matters a little

Because thresholds are rounded to a limited number of hexadecimal digits, the recorded probability can be slightly different from the target. With four digits the difference is tiny, but it is the recorded value, not the intended one, that should be used to compute the count.

16Composable and Rule-Based Samplers

Real teams do not want a single flat keep rate; they want rules like “keep all errors” and “keep very little of the health check traffic”.

A related OpenTelemetry proposal introduces composite samplers, which combine several smaller sampling rules into one decision. Each rule can be simple, such as “requests to the health endpoint keep at a very low rate” or “requests for checkout keep at a high rate”, and a composition picks the matching rule for each span. The consistent probability building blocks described in this tutorial are the base pieces from which such compositions are assembled.

Rule 1: protect valuable traffic

Give important operations, such as payments or sign-ups, a generous keep rate. Their threshold is low, so a large share of them are kept and the shared number still lines everything up.

Rule 2: starve noisy traffic

Give high-volume, low-value operations such as health checks a very high threshold, so only a trickle is kept. Their recorded thresholds let you still estimate their true volume.

Rule 3: use a sensible default

Everything that matches no special rule falls into a default rate. This is the safety net that ensures no part of the system is left without a decision.

Simple Analogy

Think of a library that keeps every copy of rare manuscripts, a few copies of popular novels, and a single copy of old phone directories. Each shelf has its own policy, yet all policies are applied with the same cataloguing system, so the librarian can always estimate how many books of each type the library really owned.

The key point is the rule from Chapter 14: every rule may choose its own threshold, but none may use the shared random number to decide which rule applies. The category a span belongs to is decided by attributes like the operation name, never by the random number. As long as that holds, rule-based sampling keeps all the benefits of consistency.

17Data Flow and Lifecycle: One Request from Start to Storage

Let us follow a single request through a realistic system, step by step, and watch the random number and the threshold at work.

A customer taps the pay button in a mobile app. The request enters through a gateway, moves to an order service, then to a payment service, and finally the finished spans flow through a collector to storage. Assume the gateway keeps ten percent of traces as the root sampler, the order and payment services use parent-based sampling, and the collector applies a further proportional reduction.

sequenceDiagram
    participant G as Gateway
    participant O as Order service
    participant P as Payment service
    participant C as Collector
    participant S as Storage
    G->>G: Make sure random value R exists
    G->>G: Compare R with threshold e666
    G->>O: Call with context carrying th and R
    O->>O: Parent-based: copy the parent decision
    O->>P: Call with the same context
    P->>P: Parent-based: copy the parent decision
    P-->>C: Export kept spans with th
    O-->>C: Export kept spans with th
    G-->>C: Export kept spans with th
    C->>C: Proportional thinning raises th
    C->>S: Forward survivors with updated th
        
FIG 6 — Lifecycle of one request: decide once, propagate, thin, record
1

Birth of the trace

The gateway creates the trace identifier. If its library promises that the identifier is random, the last seven bytes already serve as the shared number. Otherwise it writes an explicit value.

2

The root decision

The gateway compares the shared number with its ten percent threshold. If the number is at or above it, the span is kept and the threshold is recorded for propagation.

3

Propagation

The context travels with each outgoing call. Downstream services read the trace identifier, the flags and the OpenTelemetry trace state entry.

4

Child decisions

Parent-based samplers copy the parent’s outcome and keep the threshold unchanged. Whole-trace consistency inside the application chain comes for free.

5

Export and collection

Finished, kept spans leave each application and arrive at the collector, each carrying the recorded threshold.

6

Downstream thinning

The collector may apply another round of sampling. Because every span of the same trace shares the same random number, a stricter threshold removes the same trace’s spans together, keeping trace completeness.

7

Storage and analysis

The backend stores spans along with their final thresholds, and computes adjusted counts so that dashboards show estimated true volumes, not sampled volumes.

18Migrating from Older Ratio Samplers

Most systems already run some form of ratio sampling, so the practical question is how to move without breaking traces along the way.

Early OpenTelemetry ratio-based samplers were meant to be consistent, but the exact way they turned a trace identifier into a decision was never fully pinned down when those samplers became stable. In practice, different language libraries, and even different versions of the same library, could compute the decision differently for the same identifier. Because of that, the original guidance said to use such a sampler only at the root of a trace, together with a parent-based sampler for everything else.

The modern specification pins down the behaviour and ties it to the W3C Trace Context randomness guarantee. During the transition, the guidance is to keep following the older advice until every tracing library in the system has been upgraded. You can tell that the upgrade is complete when all spans in the system carry the trace random flag. To help, the ratio-based sampler issues a warning the first time it assumes randomness for a context where the flag is not set.

1

Inventory your libraries

List every service and its tracing library version. Identify which ones can set the random flag and which still cannot.

2

Keep root-only ratio sampling

Until all libraries are upgraded, apply the ratio sampler only at trace roots and rely on parent-based sampling elsewhere.

3

Upgrade and watch the flag

As libraries upgrade, monitor how many spans carry the random flag. A rising share is your progress meter.

4

Turn on consistent behaviour widely

Once all spans carry the flag, you can safely rely on the shared randomness in more places, including downstream samplers.

!
Transition risk

The danger window is the period when some libraries produce decisions the new way and others the old way. In that window, the same request can be kept by one service and dropped by another. Keeping root-only decisions during the window prevents that.

19Security and Trust Considerations

Whenever data travels in a header that outsiders can influence, it is worth asking what a misbehaving sender could do with it.

The trace context travels in headers of ordinary requests. At the edge of your system, where requests come from the public internet or from third parties, those headers are untrusted input. A caller could send a crafted trace state with a chosen random value or a fake threshold. The consequences are mostly about data quality and cost, but they deserve thought.

Risk 1

Forcing everything to be kept

A caller who supplies a very high random value, or a context claiming the span is sampled, could make parent-based samplers keep spans, raising your cost.

Risk 2

Hiding from sampling

A caller who supplies a very low random value could try to keep their traffic out of your records, which matters if tracing supports audits.

Risk 3

Poisoned statistics

False thresholds distort adjusted counts, making dashboards report wrong volumes.

Risk 4

Header bloat

Unbounded or malformed trace state entries can inflate request size or cause parsing problems.

DECISION-01 Recommended
Problem

Edge services accept trace headers from callers they do not control.

Decision

At trust boundaries, treat incoming sampling information as a hint and start a new root decision for requests from untrusted sources, replacing outside trace state with your own. Internally, trust the context that your own services produce.

Consequence

Outsiders cannot dictate your sampling cost or distort your numbers, at the price of losing end-to-end traces that begin outside your organisation, unless you explicitly agree to share them with trusted partners.

Also remember that traces may contain sensitive attributes such as user identifiers or request details. Sampling reduces the amount of such data stored, but it is not a privacy control. Sensitive values still need to be scrubbed or avoided in the spans that survive.

20Monitoring, Logging and Health Metrics

A sampling setup that nobody watches will drift, so build a small dashboard that tells you whether it is behaving.

Because the whole benefit of this scheme is trustworthy numbers, the most important monitoring checks compare what the data says with what you expect. Here are the checks that give the most value for the least effort.

1

Share of spans with a threshold

Track the fraction of stored spans that carry a threshold. A drop usually means some service or stage is erasing it or not writing it.

2

Share of spans with the random flag

Track how many spans carry the trace random flag. Anything below full coverage points to libraries that still need upgrading.

3

Effective keep rate versus configured rate

Compare the observed fraction of kept traces with the configured probability. Large, persistent gaps suggest biased randomness or misconfiguration.

4

Estimated total versus an independent count

Compare the sum of adjusted counts for a key operation with a trustworthy count from another source, such as a load balancer metric. Close agreement is the best proof that your thresholds are honest.

5

Trace completeness

Measure the share of kept traces that have no missing parents. A fall in completeness is the earliest sign that decisions are no longer lining up.

i
Logging tip

When a stage erases or rewrites a threshold because of an inconsistency, log that event at a low volume with a counter. These events are rare when everything works, and a sudden burst is an excellent early warning.

21Design Patterns and Anti-Patterns

Patterns capture what tends to work; anti-patterns capture what tends to hurt, and both are cheaper to learn from a page than from an outage.

Patterns that work

Decide once at the edge

Make the root decision at the entry of the system and let parent-based samplers follow. Whole traces are kept or dropped, and the keep rate is easy to reason about.

Thin gradually along the pipeline

Keep a moderate share in applications, then use collectors to apply stricter reductions as data approaches expensive storage. Each stage records its change.

Rule-based rates by importance

Assign generous rates to valuable operations and tiny rates to noisy ones, relying on recorded thresholds to restore true volumes in analysis.

Anti-patterns to avoid

ANTI-PATTERN-02 Avoid
Problem

Every service applies its own ratio sampler with a private random source, such as a fresh random number at each service.

Why It’s Harmful

This is the original coin flip problem. Traces are fragmented and the chance of a whole trace falls exponentially with the number of services.

Correct Approach

Derive the decision from the shared random number carried with the trace, and let non-root services follow the parent unless there is a deliberate reason not to.

ANTI-PATTERN-03 Avoid
Problem

A downstream stage keeps spans using its own criteria and leaves the recorded threshold untouched.

Why It’s Harmful

The stored threshold now claims a higher keep probability than the one that really applied, so adjusted counts are too low and dashboards under-report.

Correct Approach

Every stage that changes the effective keep probability must update the threshold. If the new probability cannot be expressed, erase the threshold so the span is treated as having unknown weight.

ANTI-PATTERN-04 Avoid
Problem

Treating a non-random identifier as a source of randomness because it is convenient.

Why It’s Harmful

Some requests are systematically kept more than others, which produces biased samples and misleading statistics, and cannot be fixed afterwards.

Correct Approach

Use identifiers only when the library promises random generation, and supply an explicit random value at the root otherwise.

22Best Practices and Common Mistakes

A short checklist is often more useful than a long theory, so here are the habits that separate smooth rollouts from painful ones.

Best practices

  • Make the root decision at one clear place, usually the entry gateway.
  • Use parent-based sampling for internal services unless you have a specific reason not to.
  • Pass trace state through every hop without dropping the OpenTelemetry entry.
  • Use around four digits of precision for thresholds.
  • Keep a dashboard comparing estimated totals with an independent source.
  • Document each stage’s role: root, parent-based, equalizing or proportional.

Common mistakes

  • Letting several services create their own randomness for the same trace.
  • Lowering thresholds downstream, which wrongly implies a higher earlier keep rate.
  • Counting spans directly and ignoring adjusted counts.
  • Confusing a low threshold with a low keep rate.
  • Trusting outside callers’ trace headers without checking.
  • Mixing old and new ratio samplers during a migration without root-only safeguards.
i
A rule of thumb

If you can answer three questions for every trace you store, your setup is healthy: where did the shared random number come from, which stage last changed the threshold, and what keep probability does the stored threshold say.

23Real-World Scenarios

Examples help the ideas settle, so here are four situations that teams commonly meet.

An online shop with a flash sale

Traffic jumps tenfold for an hour. The team lowers the root keep rate at the gateway so cost stays flat. Because thresholds are recorded, the dashboards still show the true surge in orders, and traces that are kept remain complete for investigating slow checkouts.

A bank with strict storage budgets

Applications keep a generous share, and an equalizing collector brings everything down to one common rate before storage. Analysts then work with uniform data, and auditors can see exactly which rate applied.

A streaming service with a noisy health check

Health checks make up most of the traffic. A rule gives them a tiny keep rate, while playback requests keep a large one. Estimated volumes for both are recovered from their thresholds, so capacity planning stays accurate.

A company with several teams and languages

Teams use different libraries and versions. By following the migration steps and watching the random flag coverage, the platform group moves everyone to consistent sampling without breaking traces mid-rollout.

“Sample cheaply at the edge, thin carefully downstream, and always record what you did.”

24Advantages, Disadvantages and Trade-offs

No design is free, so let us weigh what consistent probability sampling gives and what it asks in return.

Advantages

  • Complete traces at the configured keep rate, not just fragments.
  • Different services can use different rates without contradicting each other.
  • Collectors can thin further without losing correctness.
  • Recorded thresholds allow accurate estimates of true volumes.
  • The decision is one cheap comparison with no coordination or extra network calls.
  • Works across languages and vendors when everyone follows the same rules.

Disadvantages and trade-offs

  • Needs genuinely random identifiers or an explicit random value at the root.
  • Requires every participant to propagate and respect the trace state entry.
  • Cannot make content-based choices from the shared number, so keeping errors needs tail techniques.
  • Mixed old and new libraries create a risky transition period.
  • Adds a small amount of header content and some operational learning.
ApproachComplete tracesCan favour errors or slow requestsAccurate volume estimatesExtra infrastructure
Independent random decisionsRarelyNoNo, rates unknownNone
Consistent probability samplingYes, at the configured rateNot by itselfYes, via recorded thresholdsNone beyond trace state
Tail sampling on whole tracesYes, for kept tracesYesOnly if rates are recorded carefullyCollector tier that buffers traces

The comparison shows that these approaches are not enemies. Consistent probability sampling is a solid, cheap baseline. Tail techniques, which can look at the finished trace and favour errors or slowness, can be layered on top, as long as they record the probabilities they apply.

25The Math Under the Hood, Without the Fear

You do not need advanced mathematics to trust this scheme; you only need to see why one comparison gives the right keep rate.

Start with a fact about fair randomness: if a number is chosen uniformly from a range, then every equal-sized slice of the range is equally likely to contain it. Cut the 56-bit range at the threshold. The slice below the line holds a fraction of the range equal to the rejection share, and the slice at or above the line holds the keep share. Since the random value lands in each slice with a probability equal to that slice’s share, the chance of landing in the keep slice is exactly the keep probability. That is the entire proof, and it is why the formula for the threshold uses one minus the keep probability.

Simple Analogy

Imagine throwing a dart at a long strip of paper, blindly, with every spot equally likely to be hit. If you colour the right-hand quarter of the strip red, a quarter of your throws will land in the red zone. The threshold is the place where the colour changes, and the strip is the number range.

Why 56 bits is plenty

Fifty-six bits give roughly seventy-two quadrillion distinct values. Even a keep rate of one in a million leaves tens of billions of values in the keep slice, so the slice is never too narrow to behave like a smooth probability. The specification limits keep probabilities to a range that begins at two raised to the power of minus fifty-six, which is the smallest slice that contains at least a single value.

Rounding and the small gap between target and actual

Because thresholds are written with a limited number of hexadecimal digits, the keep probability that a stored threshold really represents can differ slightly from the one the operator typed. The specification gives helpful examples with four digits of precision. A target of one in ten becomes a stored threshold whose real keep probability is about 0.100006, and its adjusted count is about 9.9994 instead of exactly ten. A target of one in three has a real probability of about 0.333328 and an adjusted count of about 3.00005. These gaps are tiny, but they show why the stored value, not the intended one, should be used when computing counts.

Target keep rateStored threshold (4 digits)Real keep probabilityAdjusted count of one kept span
1 in 28Exactly 0.5Exactly 2
1 in 3aaabAbout 0.333328About 3.00005
1 in 10e666About 0.100006About 9.9994
1 in 100fd70aAbout 0.0100002About 99.998
i
Practical meaning

For everyday use the difference is invisible. It matters only if you compare very precise totals, in which case you should always rely on the recorded threshold.

Why powers of two are special

Rates like one half, one quarter and one eighth fit perfectly into binary, so their thresholds are short and exact. An earlier line of thinking about consistent sampling restricted probabilities to powers of two for exactly this reason: it makes the arithmetic simple. The newer approach keeps the same shared-number idea but allows nearly any probability by using a full-width threshold, which removes the awkward restriction without giving up consistency.

26A Full Worked Example with Real Numbers

Let us put every idea together in one story, using the friendly zero-to-one-hundred ruler so the arithmetic fits in your head.

A food-delivery company has a gateway, an order service and a kitchen service. Ten thousand customers place orders in one hour. The gateway is the root and keeps twenty percent of traces. Remember the ruler: twenty percent kept means a threshold of eighty. The order and kitchen services follow the parent. A collector then applies a proportional reduction to fifty percent of what arrives.

1

The shared number is fixed for each order

Every order has a random value between zero and one hundred. About one in five of the ten thousand orders, roughly two thousand, have a number of eighty or more.

2

The root decision

The gateway keeps those roughly two thousand orders and records a threshold equal to a twenty percent keep rate. The other eight thousand are dropped at the root, and with parent-based followers their child spans are dropped too.

3

Each kept trace is complete

Because the order and kitchen services copy the parent decision, each of the two thousand kept orders has spans from all three services. There are no half traces.

4

The collector thins proportionally

Multiplying the incoming twenty percent by the collector’s fifty percent gives a new keep probability of ten percent, a threshold of ninety on the ruler. Only spans whose number is ninety or more survive: roughly one thousand orders. These orders were a subset of the two thousand, and all three of their spans survive together.

5

The backend estimates the truth

Each stored span now carries a keep probability of ten percent, so its adjusted count is ten. About one thousand kept orders multiplied by ten gives an estimate of ten thousand orders in the hour, which matches reality.

10,000
orders in the hour
~2,000
kept at the gateway (20%)
~1,000
kept after the collector (10%)
flowchart LR
    A["10,000 orders"] --> B["Gateway keeps R at or above 80: about 2,000"]
    B --> C["Collector keeps R at or above 90: about 1,000"]
    C --> D["Backend: each span counts as 10, estimate 10,000"]
        
FIG 7 — The numbers shrink at every stage, yet the estimate still recovers the true total

Notice what would have gone wrong with independent coin flips. If the collector had flipped a fresh fifty percent coin for each span, a kept order would often have lost its kitchen span or its gateway span, and the three-service story would have been broken for about three quarters of the surviving orders. The shared number is what makes the collector’s reduction remove whole orders instead of random pieces.

27Consistency Beyond a Single Trace

Sometimes the thing you care about is bigger than one trace, such as a user session or a batch job, and the shared number can be stretched to cover it.

Most of this tutorial treats one trace as the unit of consistency. But the explicit randomness value is just a number placed in a header by the root sampling decision, and nothing prevents the same number from being given to several different traces on purpose. When that happens, those traces rise and fall together: for any threshold, either all of them are above it or all are below it.

Whole user sessions

A website could hand the same random value to every request in one visit. A visitor whose number is high gets almost all their requests traced, and a visitor whose number is low gets almost none. When an investigator opens a complaint, the whole visit is present instead of scattered clicks.

Linked batch work

A nightly job that triggers many small tasks, each starting its own trace, can share one number so that either the whole run is visible or none of it is. The tasks still make their own decisions but they agree.

Translating other systems

When a decision has already been made by another consistent mechanism, such as a hash of an identifier, the hash can be turned into an explicit random value. Later stages can then keep recording the probability using the same threshold language while preserving the original decision.

!
Important limits

Only the root decision may set the explicit value, and sharing it across traces makes those traces correlated. That is the point, but it also means that statistics computed across them are no longer independent. A handful of heavy sessions with high numbers can swing totals, so be careful when estimating variability.

Simple Analogy

A teacher who gives every student in the same family the same lottery ticket will find that siblings are either all picked or all skipped. That might be exactly what a family photograph needs, but you would not use the number of photographed siblings as an independent measure of how popular the school is.

28Planning Cost and Capacity with Keep Rates

Because the keep rate is a plain number, you can plan budgets with simple multiplication before you change a single setting.

The cost of tracing grows with the amount of data you keep, which is the number of requests multiplied by the number of spans per trace, multiplied by the size of each span, multiplied by the keep rate. Everything except the keep rate is determined by your system. That makes the keep rate the main dial for cost, and consistent sampling makes the dial predictable: cut the rate in half and you cut the kept data roughly in half, without damaging trace completeness.

Illustrative scenarioRequests per daySpans per traceKeep rateSpans stored per day
Small service1 million10100%10 million
Same service, tuned1 million1010%1 million
Large platform500 million401%200 million
Large platform, wider net500 million405%1 billion

The numbers in this table are illustrative, not measurements of any real company, but the lesson is real: a handful of percentage points at a large scale means hundreds of millions of spans. Also notice that spans per trace matters as much as the keep rate. A trace that is twice as deep costs twice as much per kept request, so reducing needless spans in the application can be as valuable as lowering the keep rate.

A simple planning routine

1

Set a storage budget

Decide how many spans per day your backend can comfortably hold and pay for.

2

Measure unsampled volume

Estimate requests per day and average spans per trace, perhaps from a short period at a high keep rate.

3

Divide to find the keep rate

Budget divided by unsampled volume gives a starting keep rate. Round down to a clean value and write it as a threshold.

4

Verify with adjusted counts

After the change, compare the estimated total from adjusted counts with an independent request count. If they agree, your plan and your thresholds are both sound.

i
Seasonal traffic

A fixed keep rate means cost rises with traffic. Where traffic is spiky, consider raising the threshold temporarily, for example during a sale. Since the new threshold is recorded, the numbers on your dashboards stay correct throughout.

29Testing, Rollout and Troubleshooting

A careful rollout plus a small troubleshooting table will save you from most surprises.

How to test without touching production first

Before changing anything in production, run a test environment with synthetic traffic that touches every service. Use a high keep rate to gather plenty of data, then check four properties in turn. First, check completeness: nearly every kept trace should have all its expected spans. Second, check nesting: if you lower the keep rate at one stage, the surviving traces must be a subset of the earlier survivors. Third, check the recorded thresholds: every kept span should carry the threshold you expect at its stage. Fourth, check the totals: the sum of adjusted counts should match the number of synthetic requests you generated.

A cautious rollout order

1

Observe only

Turn on reporting of threshold and random flag coverage before changing any sampling rate, so you have a baseline.

2

Change the root first

Update the gateway or main entry point and watch that traces stay whole in the services behind it.

3

Add downstream stages one at a time

Introduce collector sampling in one pipeline, verify totals, then repeat for others.

4

Keep a rollback ready

Know how to return to the previous configuration quickly. Sampling changes are easy to reverse because they only affect what is kept from then on.

Symptom and likely cause

Symptom you seeLikely causeWhat to check
Many traces are missing spans from one serviceThat service makes its own independent decision or drops the trace stateIts sampler type and whether it forwards the OpenTelemetry entry
Dashboards under-report volume after a changeA stage dropped data without raising the recorded thresholdThe threshold on spans leaving each stage
Kept share is far from the configured rateThe identifier is not actually random, or the random flag is missingHow identifiers are generated and the random flag coverage
Many spans have no thresholdSome stage erases it as an unknown-probability stage or an inconsistencyParent-based stages receiving context with no threshold
Costs jump unexpectedlyUntrusted callers force spans to be keptEdge handling of incoming trace context
i
Debugging habit

When something looks wrong, pick one kept trace and walk it span by span, writing down the threshold and the source of randomness at each hop. The first hop where the story changes is almost always where the bug lives.

30How This Compares with Hash-Based Sampling

Many existing systems already reach consistency another way, by hashing the trace identifier, so it helps to see the similarities and differences.

A very common technique takes the trace identifier, runs it through a hash function, and compares the result with a keep rate. Because every service performs the same calculation on the same identifier, they reach the same answer. That is a consistent scheme, and it works well when every participant uses the same hash function and the same seed. Its weak spot is the word same: different languages, libraries or versions often implement the hash slightly differently, and even a small difference breaks agreement across a system.

The OpenTelemetry approach moves the agreement from the calculation to the data. Instead of every participant needing to run the same hash, they only need to read the same shared number and compare it with a threshold, a trivial operation that every language does identically. The threshold is then written into the trace state, which also gives the final backend the knowledge of the keep probability that a pure hash scheme normally lacks.

AspectHash-based consistent samplingShared randomness and threshold
What must everyone agree onThe exact hash function and seedOnly the meaning of the number and the comparison
Risk across languages and versionsHigher, because small implementation differences break agreementLower, because the work is an integer comparison
Keep probability known downstreamUsually not recordedRecorded as the threshold
Further thinning by collectorsPossible, but statistics are harder to keep honestBuilt in with raise-only thresholds
Using non-trace data such as logsNeeds extra conventionsRandomness and threshold can be recorded as attributes
Simple Analogy

Hash-based agreement is like two cooks who must each follow an identical recipe to produce an identical dish. Shared randomness is like handing both cooks the same finished ingredient and asking only whether it is above a line. The second method has far fewer ways to go wrong.

i
Bridging old and new

Existing hash-based decisions do not have to be thrown away. The hash can be mapped into a randomness value, so that systems that started with hashing can still describe their decisions with thresholds and fit into the same ecosystem.

31Status and Ecosystem Today

A tutorial should be honest about how settled the topic is, so here is where things stand.

The OpenTelemetry specification page on probability sampling carries the status Development, in the specification release numbered 1.61.0 at the time of writing. That label means the mechanism is documented and implemented in places, but its details are still allowed to evolve. Treat the concepts in this tutorial, the shared randomness and the rejection threshold, as the stable core, and check the current specification before depending on a fine detail such as an exact field name or default.

Spec

Probability sampling in trace state

Defines the shared randomness, the rejection threshold, the decision rule, and the downstream sampler behaviours described in this tutorial.

Spec

Trace state handling

Defines how the threshold and the explicit randomness value are written into the OpenTelemetry trace state entry.

Proposals

Threshold and composite sampler proposals

The enhancement proposals behind the threshold encoding and the composable sampler design that the specification references.

Collector

Probabilistic sampler processor

A collector processor that can apply consistent sampling to spans, and which also supports a hash-based mode whose decisions can be mapped onto the same randomness and threshold values. For logs, which have no trace state, it can record the randomness and threshold as attributes.

i
Stay current

Because the specification is still in development, check the official OpenTelemetry documentation and the release notes of your tracing libraries and collector before rolling out. Support and defaults differ by language and version.

32Frequently Asked Questions

Q1What is consistent probability sampling in one sentence?

It is a way for every service and collector to decide independently, but in agreement, whether to keep a span, by comparing one shared random number with a personal threshold, and then recording that threshold so the keep probability stays known.

Q2Do services need to talk to each other to agree?

No. The shared random number travels inside the normal trace context, and each sampler needs only its own configuration. There is no coordinator and no extra network call.

Q3Why is the threshold called a rejection threshold?

Because it measures how much of the number range is rejected. Spans whose random number falls below it are dropped, and spans at or above it are kept.

Q4Where does the random number come from?

Either from the last seven bytes of the trace identifier, when the library promises they were generated randomly, or from an explicit random value stored in the OpenTelemetry trace state entry.

Q5Can different services use different keep rates and still agree?

Yes. Their decisions nest: whatever the stricter sampler keeps, the more generous one also keeps. They do not produce identical sets, but they never contradict each other.

Q6Why can a collector only raise the threshold?

Lowering it would claim that an earlier stage was more generous than it truly was, which would make estimated counts too small. Raising it honestly records an extra round of thinning.

Q7What does a threshold of zero mean?

It means nothing is rejected, so every span is kept. It stands for a keep probability of one hundred percent.

Q8What happens if a span has no threshold?

Its keep probability is unknown, so it cannot be used for adjusted counts. It may still be stored and viewed like any other span.

Q9Can I keep all errors with this scheme?

Not by using the shared random number alone, since errors are not known at the start of a request. Rule-based and tail approaches can favour errors, provided they choose their own thresholds and record them honestly.

Q10Is zero percent sampling the same as dropping everything with a threshold?

No. Probability sampling is defined from a tiny positive probability up to one hundred percent. Dropping everything is valid configuration but is not a form of probability sampling and carries no threshold.

Q11Does this replace tail sampling?

No. They complement each other. Consistent probability sampling gives a cheap, correct baseline, and tail techniques can be layered on top for smarter choices.

Q12Is it production-ready?

The specification is marked as in development, and support varies by language and tooling. Many teams adopt the ideas gradually, starting with root-only ratio sampling and parent-based followers, and verify support in their own libraries first.

33Summary and Key Takeaways

We began with a courier company whose warehouses flipped private coins and ended up with an album full of fragments. Consistent probability sampling replaces those private coins with one shared lucky number on the parcel label and a personal cut-off line at every warehouse. Because everyone reads the same number, the decisions nest neatly, whole traces survive at the configured rate, and different services can still choose different rates. By recording the threshold that was applied, the system also remembers how many real spans each kept span stands for, so totals, rates and dashboards remain trustworthy even though most data was never stored. The scheme is cheap, needs no coordination, and fits naturally into the headers that already travel with every request, though its specification is still in development and deserves a fresh look before you commit to details.

Key Takeaways

  • One shared number — a 56-bit randomness value, taken from the trace identifier or an explicit value, is read by every participant.
  • One simple rule — keep a span when the random value is at or above the rejection threshold, otherwise drop it.
  • Nested decisions — stricter samplers always keep a subset of what more generous samplers keep, which is what consistency means.
  • Threshold recorded, probability known — kept spans carry their threshold in the trace state, and a missing threshold means unknown probability.
  • Raise, never lower — downstream samplers can only increase the threshold, either equalizing to a common rate or reducing proportionally.
  • Adjusted counts — one divided by the keep probability turns sampled spans back into honest totals.
  • Mind the edges — protect trust boundaries, plan migrations carefully, and monitor threshold coverage and trace completeness.