Consistent Probability Sampling in OpenTelemetry
A complete, plain-English guide to how every service in a system can agree on which traces to keep, using one shared random number and one simple threshold, so sampled traces stay whole and the numbers you compute from them stay honest.
Picture a courier company that moves parcels through five warehouses. Management wants a photo record of the journey of some parcels, but photographing everything would fill every hard disk in the company. So each warehouse manager decides on their own, by flipping a coin, whether to photograph the parcel in front of them. At the end of the month, management opens the album hoping to follow a parcel from the first warehouse to the last, and discovers that almost no parcel has photos from all five stops. Most have one photo, a few have two, and the story of any single journey is full of holes. The photos are not useless, but they cannot answer the question that matters: what happened to this parcel from start to finish?
This is exactly the situation that distributed tracing faces when many services each decide independently what to keep. Consistent Probability Sampling is the OpenTelemetry approach that fixes it. Instead of every warehouse flipping its own coin, every warehouse looks at the same lucky number written on the parcel label and compares it with its own personal cut-off line. Because everyone reads the same number, the decisions line up. In this tutorial we will build the idea from the ground up, one small step at a time, so that you can explain it to a teammate, reason about it in a design review, and recognise it when you see it in a trace header.
1Why Sampling Exists at All
Before we talk about doing sampling consistently, we need to be clear about why we sample in the first place.
Modern software is rarely one program. A single click on a shopping website may touch a web gateway, a login service, a catalogue service, a pricing service, a payment service, and several databases. To understand what happened during that click, engineers record a trace, which is the full story of the request, made of many small pieces called spans. Each span describes one step, such as one function call, one database query, or one network request, and records when it started, how long it took, and whether it succeeded.
Recording every span for every request is the dream of every engineer and the nightmare of every finance team. A busy system can create millions of spans every minute. Each span has to be built in memory, sent over the network, received by a collector, processed, stored, indexed, and later searched. Each of those steps costs processor time, network bandwidth, disk space, and money. At large scale, the bill for keeping everything can rival the bill for running the actual product.
Think of a national opinion poll. Nobody asks every citizen what they think about a new law, because that would take years and cost a fortune. Pollsters ask a carefully chosen small group and use the answers to describe everyone. Sampling in observability works the same way: you keep a small, representative part of your traces and use it to understand the whole.
The key promise of sampling is representativeness. If you keep one trace out of every hundred, and you keep them fairly, then what you see in the kept traces should resemble what happens in all of them. You can still find slow requests, still see typical error patterns, and still estimate how many requests of each kind happened. The word fairly is doing a lot of work in that sentence, and the rest of this tutorial is about what fairness means when many independent programs have to cooperate.
Sampling is a cost and quality lever, not a bug fix. Done well, you keep nearly all the insight for a small fraction of the cost. Done badly, you keep the cost of tracing and lose the ability to explain what happened.
2The Vocabulary You Need First
Every new term below comes with what it is, why it exists, where you meet it, and a picture to hold in your head.
Trace
The whole journey of one request through a system. It exists so humans can follow cause and effect across many programs. Like a parcel’s complete delivery history.
Span
One timed step inside a trace. A trace is a family tree of spans, with a root span at the top and children below. Like one stamp in a passport.
Sampling probability
The chance that a span is kept. A probability of 0.25 means roughly one span in four survives. It is usually written as a number between zero and one.
Sampler
The small piece of logic that decides keep or drop. It lives inside the tracing library of each program and, later, inside collectors.
Head sampling
A decision made when a span begins, before anyone knows how the request will end. Cheap and fast, but it cannot see errors that come later.
Tail sampling
A decision made after spans finish, usually in a collector, so the decision can use what actually happened, such as slowness or failure.
Adjusted count
How many real spans one kept span stands for. If one in a hundred is kept, each kept span counts as one hundred. It is simply one divided by the probability.
Consistent decision
A set of decisions that never contradict each other in a harmful way. The precise meaning is given in Chapter 8.
A word of caution about the word “head”
The word head sampling is used in two slightly different ways in the industry. Sometimes it means the exact moment a tracing library creates a new span and decides whether to record it. Other times it describes a whole style of tracing setup in which every child follows its parent’s decision, so the choice made at the very start of the trace governs everything after it. When someone says they use head sampling, it is worth asking which meaning they have in mind. Tail sampling, similarly, may mean thinning individual spans as they pass through a collector, or it may mean waiting to assemble a whole trace and then deciding about the entire thing at once.
Consistent probability sampling is not a rival of head or tail sampling. It is a rule for how probability-based decisions are made and recorded, and it can be used by head samplers and by downstream samplers alike.
3The Problem: Independent Coin Flips Break Traces
To appreciate the cure, let us feel the pain of the disease with real numbers.
Suppose a request travels through five services, and every service keeps a span with a probability of fifty percent, deciding by its own private coin flip. For the trace to be complete, all five coins must come up heads. The chance of that is one half multiplied by itself five times, which is about three percent, or roughly one trace in thirty-two. The other thirty-one traces are partial: they have some spans but not others, which means they are full of missing pieces and broken parent-child links.
It gets dramatically worse as the keep rate falls. Real systems rarely keep half of their traces. If each of the five services keeps ten percent, the chance that all five happen to keep the same request is one in a hundred thousand. You would pay to store spans from almost every request, yet nearly every stored trace would be unusable.
| Keep rate per service | Services in the request path | Chance the whole trace survives | What you get |
|---|---|---|---|
| 50% | 5 | About 3% (1 in 32) | Mostly fragments |
| 10% | 5 | About 0.001% (1 in 100,000) | Almost no whole traces |
| 10% | 10 | One in ten billion | Effectively none |
| 10% with consistent sampling | Any number | 10% | One in ten traces is whole |
flowchart LR
A["Gateway: coin flip says KEEP"] --> B["Login service: coin flip says DROP"]
B --> C["Catalogue service: coin flip says KEEP"]
C --> D["Payment service: coin flip says DROP"]
D --> E["Database: coin flip says KEEP"]
There is a second, quieter problem. Even when you do not care about complete traces, you often want to count things from sampled data, for example to estimate how many payments were attempted. If different services keep spans at different rates and the data does not record those rates, any total you compute will be wrong in an unknowable way. A trustworthy count needs to know the keep probability that applied to each span.
The two pains in one sentence
Independent sampling produces broken stories (incomplete traces) and untrustworthy arithmetic (unknown probabilities). Consistent probability sampling is designed to cure both with a single mechanism.
4The Core Idea in One Page
The whole mechanism rests on two ingredients and one comparison.
The first ingredient is a shared random number, called the randomness value. It is created once, when the trace begins, and every participant that touches the trace can read it. The second ingredient is a personal cut-off line, called the rejection threshold. Each participant chooses its own cut-off according to how much data it wants to keep. The comparison is a single question: is the shared number at or above my cut-off? If yes, keep the span. If no, drop it.
Imagine a school where, on the first day, every student is handed a ticket with a random number from zero to one hundred. For a photo day, the art club says “we photograph students whose ticket is 90 or higher” and the sports club says “we photograph students whose ticket is 50 or higher”. Nobody has to talk to each other. Yet every student the art club photographs is also photographed by the sports club, because 90 or higher always means 50 or higher. The ticket is the shared random number, and the club rules are the thresholds.
A trace begins and a random number is fixed
The first service creates the trace and makes sure a 56-bit random value exists for it, either hidden inside the trace identifier or written explicitly into the trace headers.
Each sampler turns its keep rate into a threshold
A sampler configured to keep ten percent computes the matching cut-off once and reuses it. Keeping everything means a cut-off of zero.
The comparison decides
Random number at or above the threshold means keep. Anything lower means drop. There is no extra coin flip and no extra communication.
The threshold is recorded for kept spans
The cut-off that was applied is written into the trace headers, so later stages and the final storage know the exact keep probability for that span.
Notice how little is required. There is no central coordinator, no shared database, and no network call to ask what others decided. The only shared thing is a number that travels with the request anyway, inside the trace context. This is what makes the idea practical: it adds almost no cost to the request, yet it gives every participant the power to make decisions that fit together.
5The Randomness Value: The Shared Lucky Number
Everything depends on this number being truly random and truly shared, so let us look at where it comes from.
The randomness value, often written as R, is a 56-bit number. A bit is a single zero or one, so 56 bits give about seventy-two quadrillion possible values. That is a very large range, which matters because the range limits how finely you can express a keep rate. With 56 bits, you can sample at probabilities as small as two raised to the power of minus fifty-six, far smaller than anyone realistically needs.
OpenTelemetry supports two places where this number can live.
The trace identifier itself
Every trace has a 16-byte identifier. The last seven bytes, which are exactly 56 bits, can serve as the randomness value, provided the identifier was generated randomly. This costs nothing extra on the wire.
An explicit value in the trace state
A random 56-bit number can be written into a sub-key called rv inside the OpenTelemetry part of the trace state header. It is used when the identifier cannot be trusted to be random.
Why the trace identifier can be used
The W3C Trace Context standard, in its second level, defines a flag that a tracing library can set to promise that the trace identifier’s last seven bytes were generated randomly. This is called the trace random flag. When the flag is present, any downstream participant can safely read those seven bytes as the shared lucky number without any additional header. When it is absent, a participant cannot be sure, so it should not assume randomness and may rely on an explicit value instead.
A trace identifier is like a parcel tracking number. If the post office promises that the last seven digits are assigned by a fair lottery machine, you can use those digits as your lucky number. If the post office assigns numbers in order, like one, two, three, then those digits are not lottery numbers, and you would rather write a fresh lottery number on the label.
Who is allowed to create or change the number
The rule is strict and simple: only the root sampling decision, the very first one in a trace, may create or change the explicit randomness value. Everyone after that must pass the same value along unchanged. If a middle service replaced the number, then services before and after it would be comparing their thresholds against different numbers, and the neat agreement would be destroyed. A consistent system is only as consistent as its most careless participant.
Using identifiers that are not random, such as ones built from timestamps or counters, as a source of randomness. Sampling would then be biased: some kinds of requests would be kept much more often than others, and any statistics computed from the kept data would be skewed.
Sharing luck across several traces
Because the randomness value can be explicit, the same value can be given to several independent traces on purpose. For example, all traces belonging to one user session could receive the same number, so that a session is either mostly kept or mostly dropped as a unit. The root decision is the only place where this is allowed to happen.
6The Rejection Threshold: Your Personal Cut-Off Line
If the randomness value is the lucky number, the threshold is the line you draw on the number ruler to separate keep from drop.
The rejection threshold, often written as T, is also a 56-bit number, and it is derived directly from the keep probability a sampler wants. The name tells you its meaning: it is the count of values, out of all the possible ones, that will be rejected. If you want to keep a quarter of spans, you reject three quarters of the number range, so the cut-off sits three quarters of the way up the ruler. Everything below that line is dropped, and everything at or above the line is kept.
In words, the formula is short: take one minus the keep probability, and multiply it by the size of the whole range, which is two raised to the power fifty-six. The extreme cases are easy to remember. Keeping everything gives a threshold of zero, because nothing is rejected. Keeping a tiny sliver gives a threshold very close to the top of the range.
Think of a doorway with a height bar, like the one at a fairground ride. A tall bar lets very few people in, and a bar set at floor level lets everyone in. The threshold is the height of that bar. The randomness value is each visitor’s height. The rule is simple: if you are at least as tall as the bar, you may enter.
Valid range and the “never” trap
Sampling probability in OpenTelemetry is valid from two raised to the power of minus fifty-six up to exactly one. Zero is deliberately not part of the range. The reason is conceptual: a rule that never keeps anything is not a probability sampler at all, because it keeps no data from which anything could be estimated. A sampler that drops everything is a perfectly legitimate configuration choice, but it is not a form of probability sampling and it does not carry a threshold.
How a threshold looks when written down
Thresholds are written as hexadecimal numbers, which is a compact way of writing big numbers using the digits zero to nine and the letters a to f. Trailing zeros are removed to save space, which also means the writer can choose how precise to be. Here is how common keep rates look when written with four digits of precision.
| Keep one in | Keep probability | Written threshold | What it tells you |
|---|---|---|---|
| 1 (everything) | 100% | 0 | Nothing is rejected |
| 2 | 50% | 8 | Half the range is rejected |
| 4 | 25% | c | Three quarters rejected |
| 8 | 12.5% | e | Seven eighths rejected |
| 10 | 10% | e666 | Not a power of two, so digits repeat |
| 100 | 1% | fd70a | Cut-off very near the top |
| 1000 | 0.1% | ffbe77 | More leading f digits as keep rate shrinks |
Two patterns are worth noticing. Powers of two, like one half or one quarter, produce short, clean thresholds. Other rates, like one in ten, never divide evenly into binary, so the digits repeat and the writer has to decide where to stop. And as the keep rate gets smaller, the threshold starts with more and more f characters, which is just the number creeping closer to the top of the range.
The specification suggests that tracing libraries use four digits of precision by default. That is plenty for sampling, since nobody needs to distinguish a keep rate of ten percent from ten percent plus a few parts per million.
7The Decision Rule, Worked Through by Hand
The rule is one comparison, and it is worth practising it on a friendly small ruler before trusting it on a giant one.
The entire decision is this: if the randomness value is greater than or equal to the threshold, keep the span; otherwise drop it. To make the arithmetic easy, let us shrink the giant 56-bit range into a simple ruler from zero to one hundred. A keep probability of fifty percent means a threshold of fifty. A keep probability of twenty percent means a threshold of eighty, because eighty percent of the ruler is rejected. A keep probability of ninety percent means a threshold of ten.
Now imagine four requests arriving, each with its own random number: 12, 55, 83 and 97. We have three samplers, with different keep rates. Look at which requests each one keeps.
| Request random number | Sampler A keeps 90% (threshold 10) | Sampler B keeps 50% (threshold 50) | Sampler C keeps 20% (threshold 80) |
|---|---|---|---|
| 12 | Keep | Drop | Drop |
| 55 | Keep | Keep | Drop |
| 83 | Keep | Keep | Keep |
| 97 | Keep | Keep | Keep |
Read the table column by column and then row by row. Sampler C keeps the fewest requests, and every request it keeps is also kept by B and by A. Sampler B keeps more, and everything it keeps is also kept by A. There is no request that C keeps while B drops. This staircase pattern, in which the more selective sampler always keeps a subset of what the less selective sampler keeps, is the mathematical heart of the whole system.
flowchart TD
S["Span arrives with shared random number R"] --> Q{"Is R at or above my threshold T?"}
Q -->|"Yes"| K["Keep the span and record T"]
Q -->|"No"| D["Drop the span"]
The decision involves one comparison of two integers. There is no random number generation at decision time, no lookup, and no network call. Even a service handling enormous traffic can afford to do this for every span.
Does the fairness hold?
If the random number is truly uniform, meaning every value is equally likely, then the fraction of requests with a number at or above the threshold equals the keep probability. For the ninety-percent sampler above, ninety out of every hundred requests will have a number of ten or more. The rule therefore delivers exactly the keep rate the operator asked for, on average, while making the decisions line up perfectly across different samplers.
8The Consistency Guarantee, Stated Precisely
Now that we have seen it in action, we can say exactly what consistent means.
The specification defines a consistent sampling decision like this: if a span is kept by a sampler using probability p1, then every span belonging to the same trace is also kept by any sampler using a probability p2 that is greater than or equal to p1. In plain language, the keep decisions are nested, like a set of Russian dolls. The group kept by the strictest sampler fits inside the group kept by the next strictest sampler, which fits inside the group kept by the most generous one.
Imagine water rising in a tank with marks on the wall. A mark at ten litres is passed before a mark at fifty litres. Any moment when the water is above the fifty-litre mark that is also above the ten-litre mark is impossible; the reverse is what always holds: whenever the water reaches the higher mark, it has already passed every lower mark. Thresholds behave the same way along the number ruler.
What this buys you in practice
Whole traces
If every service in a request path uses the same rule, the whole trace is kept or the whole trace is dropped. The cost of a trace that is kept at rate p is that you really do get complete traces at rate p.
Different rates, still coherent
A busy backend can use a low keep rate and a small gateway can use a high one. The backend’s spans are always a subset of the gateway’s, so what you keep never contradicts itself.
Safe further thinning
A collector can drop even more data later by raising the threshold, and the result remains consistent with everything that was decided earlier.
Known probabilities
Because the threshold is recorded, the exact keep probability of each kept span can be reconstructed and used for counting.
graph TD
A["All spans"] --> B["Kept at 50 percent: threshold 8"]
B --> C["Kept at 25 percent: threshold c"]
C --> D["Kept at 10 percent: threshold e666"]
D --> E["Kept at 1 percent: threshold fd70a"]
One more point deserves emphasis. Consistency does not require every service to use the same keep rate. It only requires that they all compare against the same random number. That freedom is what lets teams tune cost per service while still ending up with traces that make sense when they are stitched back together.
9How the Information Travels: The Trace State Header
Numbers are only useful if they reach the next participant, so we need a place to carry them.
Distributed tracing already passes a small packet of context from service to service with every request. One part of that packet, defined by the W3C Trace Context standard, is the trace state, a list of vendor-specific entries. Each entry has a short key naming the vendor or project and a value that the owner is free to define. OpenTelemetry reserves the key ot for itself, and inside that value it defines small named sub-keys. Two of them matter for sampling.
th (threshold)
Records the rejection threshold that applied when a span was sampled. It exists so later stages and the final backend know the keep probability. It is only written for spans that were kept.
rv (randomness value)
Records an explicit 56-bit random value, used when the trace identifier cannot be trusted to be random. Once written by the root, every later participant passes it on unchanged.
Three details that confuse beginners
Only kept spans carry a threshold
A dropped span is never exported, so there is nobody to read its threshold. The header only needs to describe spans that survive.
Trailing zeros are removed
A threshold for a one percent keep rate, written to five digits, appears as fd70a4, not fd70a400000000. Fewer digits means less header weight and a way to express how precise the value is.
Missing threshold means unknown probability
If a kept span has no threshold, it means we do not know what probability applied, perhaps because a sampler outside this scheme made the decision. Such spans can be stored but cannot safely be used for counting.
A threshold of zero is written as the single digit zero and means one hundred percent, keep everything. It does not mean the span was dropped. Mixing up a low threshold with a low keep rate is one of the most frequent reading mistakes.
Because the information lives in an ordinary header that already travels with every request, no new network path is introduced. Services that do not understand the OpenTelemetry entry simply pass it along, which is the usual behaviour for trace state entries, so mixed systems keep working even when only some participants are upgraded.
10Sampling Stages: Two Moments, Two Dimensions
A span can be judged more than once in its life, and understanding when helps you place each sampler in the picture.
The specification describes two kinds of stages. The first is parent and child sampling, which happens inside the tracing library while a request is still running. These decisions follow the direction of cause and effect: the first span decides, and its children usually follow. The second is downstream sampling, which happens after spans have finished and are travelling through a collection pipeline of agents and gateways. A finished span has therefore been judged exactly once by a parent and child sampler and zero or more times by downstream samplers.
flowchart LR
subgraph SDK["Inside the application: parent and child stage"]
R["Root span decides"] --> C1["Child span follows"] --> C2["Grandchild follows"]
end
subgraph PIPE["Collection pipeline: downstream stage"]
AG["Agent collector"] --> GW["Gateway collector"] --> ST["Storage backend"]
end
C2 --> AG
Why the two stages are treated differently
Inside the application, decisions are made live, while the request is in flight, and they can only use information that exists at that moment. Downstream, spans are historical records. A collector can look at a finished span, notice that it was slow or failed, and make a smarter choice, but it must also respect a rule that does not apply inside the application: it may only raise the threshold, never lower it. We will see why in Chapter 13.
Think of a newspaper. Reporters in the field decide in the moment what to write down. Later, editors in the office look at everything that came in and cut it down to fit the page. An editor can remove a story, but cannot add a quote the reporter never wrote. In the same way, a downstream sampler can thin the data further but can never resurrect what an earlier stage threw away.
Combining both stages
A very common design keeps a modest share at the application level, for instance ten percent of traces, and then lets a collector pipeline apply extra thinning or special rules. Because all stages read the same random number and record their thresholds, the final kept data still has a precisely known probability for each span.
11Architecture and Components: Who Does What
Before diving into the finer rules, let us map the cast of characters and the job each one performs.
Consistent probability sampling is not one single program. It is a shared agreement among several components, each doing a small, well-defined job. Seeing all of them on one page makes the later rules much easier to follow.
Tracing library (SDK)
Lives inside each application. Creates spans, makes sure randomness exists for the trace, runs the sampler, and writes the threshold into outgoing context.
Sampler
The decision maker inside the library. Compares randomness with its threshold and returns keep or drop, along with the threshold to record.
Context propagator
Carries the trace identifier, flags and trace state from one service to the next, so the shared number and recorded thresholds travel with the request.
Collector agent
A small process near the application that receives spans, can apply downstream sampling, and forwards what remains.
Collector gateway
A central collection point that sees data from many sources and often applies stricter or smarter sampling before storage.
Storage and analysis backend
Stores kept spans along with their thresholds and uses them to show traces and to estimate counts and rates.
flowchart LR
subgraph APP["Application process"]
SP["Span creation"] --> SM["Sampler: compare R with T"]
SM --> EX["Exporter"]
end
SM --> PR["Propagator writes context with th and rv"]
PR --> NEXT["Next service in the request path"]
EX --> AGT["Collector agent"]
AGT --> GWY["Collector gateway"]
GWY --> BE["Backend: store spans with thresholds"]
Think of an airport. Check-in counters (the libraries) tag every bag. Security lines (collectors) can remove bags that look unnecessary but cannot change the tag. The baggage hall (the backend) reads the tags to know which flight each bag belongs to and how it was handled. The system works because everyone trusts the same tag format.
12The Base Cases: Root, Parent-Based and Probability Samplers
Real configurations are built out of a few simple building blocks, each behaving differently with respect to the shared number.
Within the parent and child stage, the specification identifies three primary base cases. Learning these three gives you the vocabulary to read almost any sampler configuration.
Root sampling
The very first decision in a trace. There is no earlier threshold to respect, so it always decides on its own. It is also the only place allowed to create or modify the explicit randomness value. If the trace identifier is already trusted as random, nothing extra needs to be written.
Parent-based sampling
A child span copies the decision of its parent. The parent and child end up with matching thresholds, which is how a whole trace inside one application stays together. This is the most common choice for non-root spans.
Consistent probability sampling
A sampler that applies its own fixed probability regardless of what the parent decided. Always-on and always-off behaviours belong to this family, as does the ratio-based sampler that keeps a configured fraction. Such a sampler ignores the parent’s threshold, which can lead to incomplete traces if the rates are mixed carelessly, but when randomness is shared the results still nest consistently.
Choosing among them
| Situation | Typical choice | Why |
|---|---|---|
| The entry point of a request, such as an API gateway | Root with a ratio | Sets the keep rate for the whole trace |
| Internal services receiving a call from a trusted caller | Parent-based | Follows the entry point so traces stay complete |
| A service that must always be traced, such as a payment step | Always-on probability | Gives complete visibility, accepting the cost |
| A noisy background job with no value in tracing | Always-off | Removes cost, but it is not probability sampling and carries no threshold |
Putting a ratio-based sampler on every service as if it were the root. Each service then makes its own independent keep decision. With shared randomness the decisions nest, but different rates still produce traces with missing branches. Reserve independent samplers for places where partial traces are acceptable.
13Downstream Sampling: Raising the Bar After the Fact
Collectors can thin data further, but they must follow one rule above all others: the threshold can only go up.
Why can a downstream sampler raise a threshold but never lower it? Because lowering would be a lie. A span that arrives with a recorded threshold has already survived a stage with a particular keep probability. Pretending that the stage had been more generous than it really was, by lowering the recorded number, would claim that other spans had been kept when they were in fact thrown away earlier. Any count computed from such a claim would be too small. Raising the threshold, on the other hand, honestly records that an extra round of thinning happened.
The specification names two standard styles of downstream probability sampler.
The equalizing sampler
An equalizing sampler wants every span leaving it to carry the same threshold. If an incoming span already has a higher threshold than the sampler’s own, the span cannot be equalized, so it passes through unchanged. If the incoming threshold is lower, the sampler applies its own threshold: spans whose random number is at or above it are kept and marked with the new threshold, and the rest are dropped. The effect is to bring mixed data to a common keep rate, which makes later analysis simpler.
The proportional sampler
A proportional sampler wants to reduce traffic by a fixed factor regardless of what arrives. It multiplies the incoming keep probability by its own configured probability to compute a new, stricter threshold for each incoming span, and then applies the comparison. If the multiplied probability would fall below the smallest representable one, the span is not kept.
| Incoming keep rate | Equalizing sampler set to 10% | Proportional sampler set to 20% |
|---|---|---|
| 100% (threshold 0) | Outgoing rate 10% | Outgoing rate 20% |
| 50% | Outgoing rate 10% | Outgoing rate 10% |
| 10% | Outgoing rate 10%, nothing more dropped | Outgoing rate 2% |
| 5% | Passes through unchanged at 5% | Outgoing rate 1% |
An equalizing sampler is like a teacher who says “every student’s essay will be cut down to one page”, leaving short essays alone and trimming long ones. A proportional sampler is like a teacher who says “every essay will lose half its length”, so a long essay stays longer than a short one but both shrink by the same ratio.
Choose equalizing when you want uniform data for analysis. Choose proportional when you need a guaranteed reduction in traffic volume, for example to protect a downstream storage budget.
14The Rules That Keep the Numbers Honest
A handful of rules, if followed by every participant, protect both whole traces and trustworthy statistics.
The specification lists general requirements for every stage that touches the sampling threshold. They look dry at first, but each one blocks a specific way things can go wrong. Here they are in plain language.
Whoever changes the threshold must record the change
If a stage raises the effective threshold, it must re-encode the trace state so that the new value is visible. Otherwise the recorded probability would not match reality.
Do not let the rule depend on the shared number
A conditional stage, such as one that keeps errors more often, must not base its condition on the randomness value or anything derived from it. It may use independent randomness. Peeking at the shared number would break the even spread that the arithmetic relies on.
Erase the threshold when probability is unknown
A stage that produces spans of unknown probability, such as a parent-based sampler that receives context with no threshold, must remove the threshold in its output instead of inventing one.
Check for obvious inconsistency
When it is a cheap test, a stage should verify that a sampled span really satisfies the rule that its random number is at or above its recorded threshold, and erase the threshold if it does not. This catches bugs and bad inputs early.
Pass the explicit randomness value forward unchanged
If randomness came from an explicit value, the same value must appear in the outgoing trace state. Only the root decision may alter it.
Problem
A rule-based stage wants to keep slow requests more often, so it uses the shared random number itself to decide how much extra to keep.
Why It’s Harmful
The shared number is supposed to be spread evenly in every slice of the data. Once a rule filters data by that number, the surviving spans no longer form an even spread, and the recorded probability no longer matches the real chance of being kept. Counts computed from the data become biased without any visible warning.
Correct Approach
Let the rule choose its own threshold based on the condition, such as slow or failed, and compare that threshold with the shared number. Where extra randomness is needed, draw it from a separate, independent source.
15Adjusted Count: Turning Samples Back into Totals
This is the payoff that makes the recorded threshold so valuable: you can estimate what you did not keep.
The adjusted count of a kept span is one divided by its keep probability. If a span was kept at a probability of one percent, then it stands for roughly one hundred real spans, so its adjusted count is one hundred. If it was kept at fifty percent, its adjusted count is two. To estimate how many requests of some kind happened, you add up the adjusted counts of the kept spans of that kind, rather than counting spans one by one.
Suppose a school survey asks every tenth student a question and eighteen of the students asked say yes. You would estimate that about one hundred and eighty students in the whole school would say yes, because each answer represents ten students. Adjusted count is that multiplication, done automatically for every span.
Mixed probabilities are fine, as long as they are recorded
A backend might hold spans kept at fifty percent from a quiet service and spans kept at one percent from a very busy one. A naive row count would give the quiet service far too much weight. Summing adjusted counts corrects for this automatically, because each span carries a number that reflects how many others it stands for. That is why the recorded threshold matters even for traces nobody ever opens: it powers the numbers on dashboards.
Turning traces into metrics
Using adjusted counts to derive metrics such as request rates and error rates from sampled spans is sometimes called span-to-metrics. The rules in Chapter 14 exist so that these derived numbers are reliable. If a stage secretly changed its keep rate without updating the threshold, every rate computed afterwards would be quietly wrong.
Adjusted count is not defined for kept spans that carry no threshold. Never fill in a guess. Leave such spans out of count calculations, or report them separately as spans of unknown weight.
Because thresholds are rounded to a limited number of hexadecimal digits, the recorded probability can be slightly different from the target. With four digits the difference is tiny, but it is the recorded value, not the intended one, that should be used to compute the count.
16Composable and Rule-Based Samplers
Real teams do not want a single flat keep rate; they want rules like “keep all errors” and “keep very little of the health check traffic”.
A related OpenTelemetry proposal introduces composite samplers, which combine several smaller sampling rules into one decision. Each rule can be simple, such as “requests to the health endpoint keep at a very low rate” or “requests for checkout keep at a high rate”, and a composition picks the matching rule for each span. The consistent probability building blocks described in this tutorial are the base pieces from which such compositions are assembled.
Rule 1: protect valuable traffic
Give important operations, such as payments or sign-ups, a generous keep rate. Their threshold is low, so a large share of them are kept and the shared number still lines everything up.
Rule 2: starve noisy traffic
Give high-volume, low-value operations such as health checks a very high threshold, so only a trickle is kept. Their recorded thresholds let you still estimate their true volume.
Rule 3: use a sensible default
Everything that matches no special rule falls into a default rate. This is the safety net that ensures no part of the system is left without a decision.
Think of a library that keeps every copy of rare manuscripts, a few copies of popular novels, and a single copy of old phone directories. Each shelf has its own policy, yet all policies are applied with the same cataloguing system, so the librarian can always estimate how many books of each type the library really owned.
The key point is the rule from Chapter 14: every rule may choose its own threshold, but none may use the shared random number to decide which rule applies. The category a span belongs to is decided by attributes like the operation name, never by the random number. As long as that holds, rule-based sampling keeps all the benefits of consistency.
17Data Flow and Lifecycle: One Request from Start to Storage
Let us follow a single request through a realistic system, step by step, and watch the random number and the threshold at work.
A customer taps the pay button in a mobile app. The request enters through a gateway, moves to an order service, then to a payment service, and finally the finished spans flow through a collector to storage. Assume the gateway keeps ten percent of traces as the root sampler, the order and payment services use parent-based sampling, and the collector applies a further proportional reduction.
sequenceDiagram
participant G as Gateway
participant O as Order service
participant P as Payment service
participant C as Collector
participant S as Storage
G->>G: Make sure random value R exists
G->>G: Compare R with threshold e666
G->>O: Call with context carrying th and R
O->>O: Parent-based: copy the parent decision
O->>P: Call with the same context
P->>P: Parent-based: copy the parent decision
P-->>C: Export kept spans with th
O-->>C: Export kept spans with th
G-->>C: Export kept spans with th
C->>C: Proportional thinning raises th
C->>S: Forward survivors with updated th
Birth of the trace
The gateway creates the trace identifier. If its library promises that the identifier is random, the last seven bytes already serve as the shared number. Otherwise it writes an explicit value.
The root decision
The gateway compares the shared number with its ten percent threshold. If the number is at or above it, the span is kept and the threshold is recorded for propagation.
Propagation
The context travels with each outgoing call. Downstream services read the trace identifier, the flags and the OpenTelemetry trace state entry.
Child decisions
Parent-based samplers copy the parent’s outcome and keep the threshold unchanged. Whole-trace consistency inside the application chain comes for free.
Export and collection
Finished, kept spans leave each application and arrive at the collector, each carrying the recorded threshold.
Downstream thinning
The collector may apply another round of sampling. Because every span of the same trace shares the same random number, a stricter threshold removes the same trace’s spans together, keeping trace completeness.
Storage and analysis
The backend stores spans along with their final thresholds, and computes adjusted counts so that dashboards show estimated true volumes, not sampled volumes.
18Migrating from Older Ratio Samplers
Most systems already run some form of ratio sampling, so the practical question is how to move without breaking traces along the way.
Early OpenTelemetry ratio-based samplers were meant to be consistent, but the exact way they turned a trace identifier into a decision was never fully pinned down when those samplers became stable. In practice, different language libraries, and even different versions of the same library, could compute the decision differently for the same identifier. Because of that, the original guidance said to use such a sampler only at the root of a trace, together with a parent-based sampler for everything else.
The modern specification pins down the behaviour and ties it to the W3C Trace Context randomness guarantee. During the transition, the guidance is to keep following the older advice until every tracing library in the system has been upgraded. You can tell that the upgrade is complete when all spans in the system carry the trace random flag. To help, the ratio-based sampler issues a warning the first time it assumes randomness for a context where the flag is not set.
Inventory your libraries
List every service and its tracing library version. Identify which ones can set the random flag and which still cannot.
Keep root-only ratio sampling
Until all libraries are upgraded, apply the ratio sampler only at trace roots and rely on parent-based sampling elsewhere.
Upgrade and watch the flag
As libraries upgrade, monitor how many spans carry the random flag. A rising share is your progress meter.
Turn on consistent behaviour widely
Once all spans carry the flag, you can safely rely on the shared randomness in more places, including downstream samplers.
The danger window is the period when some libraries produce decisions the new way and others the old way. In that window, the same request can be kept by one service and dropped by another. Keeping root-only decisions during the window prevents that.
19Security and Trust Considerations
Whenever data travels in a header that outsiders can influence, it is worth asking what a misbehaving sender could do with it.
The trace context travels in headers of ordinary requests. At the edge of your system, where requests come from the public internet or from third parties, those headers are untrusted input. A caller could send a crafted trace state with a chosen random value or a fake threshold. The consequences are mostly about data quality and cost, but they deserve thought.
Forcing everything to be kept
A caller who supplies a very high random value, or a context claiming the span is sampled, could make parent-based samplers keep spans, raising your cost.
Hiding from sampling
A caller who supplies a very low random value could try to keep their traffic out of your records, which matters if tracing supports audits.
Poisoned statistics
False thresholds distort adjusted counts, making dashboards report wrong volumes.
Header bloat
Unbounded or malformed trace state entries can inflate request size or cause parsing problems.
Problem
Edge services accept trace headers from callers they do not control.
Decision
At trust boundaries, treat incoming sampling information as a hint and start a new root decision for requests from untrusted sources, replacing outside trace state with your own. Internally, trust the context that your own services produce.
Consequence
Outsiders cannot dictate your sampling cost or distort your numbers, at the price of losing end-to-end traces that begin outside your organisation, unless you explicitly agree to share them with trusted partners.
Also remember that traces may contain sensitive attributes such as user identifiers or request details. Sampling reduces the amount of such data stored, but it is not a privacy control. Sensitive values still need to be scrubbed or avoided in the spans that survive.
20Monitoring, Logging and Health Metrics
A sampling setup that nobody watches will drift, so build a small dashboard that tells you whether it is behaving.
Because the whole benefit of this scheme is trustworthy numbers, the most important monitoring checks compare what the data says with what you expect. Here are the checks that give the most value for the least effort.
Share of spans with a threshold
Track the fraction of stored spans that carry a threshold. A drop usually means some service or stage is erasing it or not writing it.
Share of spans with the random flag
Track how many spans carry the trace random flag. Anything below full coverage points to libraries that still need upgrading.
Effective keep rate versus configured rate
Compare the observed fraction of kept traces with the configured probability. Large, persistent gaps suggest biased randomness or misconfiguration.
Estimated total versus an independent count
Compare the sum of adjusted counts for a key operation with a trustworthy count from another source, such as a load balancer metric. Close agreement is the best proof that your thresholds are honest.
Trace completeness
Measure the share of kept traces that have no missing parents. A fall in completeness is the earliest sign that decisions are no longer lining up.
When a stage erases or rewrites a threshold because of an inconsistency, log that event at a low volume with a counter. These events are rare when everything works, and a sudden burst is an excellent early warning.
21Design Patterns and Anti-Patterns
Patterns capture what tends to work; anti-patterns capture what tends to hurt, and both are cheaper to learn from a page than from an outage.
Patterns that work
Decide once at the edge
Make the root decision at the entry of the system and let parent-based samplers follow. Whole traces are kept or dropped, and the keep rate is easy to reason about.
Thin gradually along the pipeline
Keep a moderate share in applications, then use collectors to apply stricter reductions as data approaches expensive storage. Each stage records its change.
Rule-based rates by importance
Assign generous rates to valuable operations and tiny rates to noisy ones, relying on recorded thresholds to restore true volumes in analysis.
Anti-patterns to avoid
Problem
Every service applies its own ratio sampler with a private random source, such as a fresh random number at each service.
Why It’s Harmful
This is the original coin flip problem. Traces are fragmented and the chance of a whole trace falls exponentially with the number of services.
Correct Approach
Derive the decision from the shared random number carried with the trace, and let non-root services follow the parent unless there is a deliberate reason not to.
Problem
A downstream stage keeps spans using its own criteria and leaves the recorded threshold untouched.
Why It’s Harmful
The stored threshold now claims a higher keep probability than the one that really applied, so adjusted counts are too low and dashboards under-report.
Correct Approach
Every stage that changes the effective keep probability must update the threshold. If the new probability cannot be expressed, erase the threshold so the span is treated as having unknown weight.
Problem
Treating a non-random identifier as a source of randomness because it is convenient.
Why It’s Harmful
Some requests are systematically kept more than others, which produces biased samples and misleading statistics, and cannot be fixed afterwards.
Correct Approach
Use identifiers only when the library promises random generation, and supply an explicit random value at the root otherwise.
22Best Practices and Common Mistakes
A short checklist is often more useful than a long theory, so here are the habits that separate smooth rollouts from painful ones.
Best practices
- Make the root decision at one clear place, usually the entry gateway.
- Use parent-based sampling for internal services unless you have a specific reason not to.
- Pass trace state through every hop without dropping the OpenTelemetry entry.
- Use around four digits of precision for thresholds.
- Keep a dashboard comparing estimated totals with an independent source.
- Document each stage’s role: root, parent-based, equalizing or proportional.
Common mistakes
- Letting several services create their own randomness for the same trace.
- Lowering thresholds downstream, which wrongly implies a higher earlier keep rate.
- Counting spans directly and ignoring adjusted counts.
- Confusing a low threshold with a low keep rate.
- Trusting outside callers’ trace headers without checking.
- Mixing old and new ratio samplers during a migration without root-only safeguards.
If you can answer three questions for every trace you store, your setup is healthy: where did the shared random number come from, which stage last changed the threshold, and what keep probability does the stored threshold say.
23Real-World Scenarios
Examples help the ideas settle, so here are four situations that teams commonly meet.
An online shop with a flash sale
Traffic jumps tenfold for an hour. The team lowers the root keep rate at the gateway so cost stays flat. Because thresholds are recorded, the dashboards still show the true surge in orders, and traces that are kept remain complete for investigating slow checkouts.
A bank with strict storage budgets
Applications keep a generous share, and an equalizing collector brings everything down to one common rate before storage. Analysts then work with uniform data, and auditors can see exactly which rate applied.
A streaming service with a noisy health check
Health checks make up most of the traffic. A rule gives them a tiny keep rate, while playback requests keep a large one. Estimated volumes for both are recovered from their thresholds, so capacity planning stays accurate.
A company with several teams and languages
Teams use different libraries and versions. By following the migration steps and watching the random flag coverage, the platform group moves everyone to consistent sampling without breaking traces mid-rollout.
24Advantages, Disadvantages and Trade-offs
No design is free, so let us weigh what consistent probability sampling gives and what it asks in return.
Advantages
- Complete traces at the configured keep rate, not just fragments.
- Different services can use different rates without contradicting each other.
- Collectors can thin further without losing correctness.
- Recorded thresholds allow accurate estimates of true volumes.
- The decision is one cheap comparison with no coordination or extra network calls.
- Works across languages and vendors when everyone follows the same rules.
Disadvantages and trade-offs
- Needs genuinely random identifiers or an explicit random value at the root.
- Requires every participant to propagate and respect the trace state entry.
- Cannot make content-based choices from the shared number, so keeping errors needs tail techniques.
- Mixed old and new libraries create a risky transition period.
- Adds a small amount of header content and some operational learning.
| Approach | Complete traces | Can favour errors or slow requests | Accurate volume estimates | Extra infrastructure |
|---|---|---|---|---|
| Independent random decisions | Rarely | No | No, rates unknown | None |
| Consistent probability sampling | Yes, at the configured rate | Not by itself | Yes, via recorded thresholds | None beyond trace state |
| Tail sampling on whole traces | Yes, for kept traces | Yes | Only if rates are recorded carefully | Collector tier that buffers traces |
The comparison shows that these approaches are not enemies. Consistent probability sampling is a solid, cheap baseline. Tail techniques, which can look at the finished trace and favour errors or slowness, can be layered on top, as long as they record the probabilities they apply.
25The Math Under the Hood, Without the Fear
You do not need advanced mathematics to trust this scheme; you only need to see why one comparison gives the right keep rate.
Start with a fact about fair randomness: if a number is chosen uniformly from a range, then every equal-sized slice of the range is equally likely to contain it. Cut the 56-bit range at the threshold. The slice below the line holds a fraction of the range equal to the rejection share, and the slice at or above the line holds the keep share. Since the random value lands in each slice with a probability equal to that slice’s share, the chance of landing in the keep slice is exactly the keep probability. That is the entire proof, and it is why the formula for the threshold uses one minus the keep probability.
Imagine throwing a dart at a long strip of paper, blindly, with every spot equally likely to be hit. If you colour the right-hand quarter of the strip red, a quarter of your throws will land in the red zone. The threshold is the place where the colour changes, and the strip is the number range.
Why 56 bits is plenty
Fifty-six bits give roughly seventy-two quadrillion distinct values. Even a keep rate of one in a million leaves tens of billions of values in the keep slice, so the slice is never too narrow to behave like a smooth probability. The specification limits keep probabilities to a range that begins at two raised to the power of minus fifty-six, which is the smallest slice that contains at least a single value.
Rounding and the small gap between target and actual
Because thresholds are written with a limited number of hexadecimal digits, the keep probability that a stored threshold really represents can differ slightly from the one the operator typed. The specification gives helpful examples with four digits of precision. A target of one in ten becomes a stored threshold whose real keep probability is about 0.100006, and its adjusted count is about 9.9994 instead of exactly ten. A target of one in three has a real probability of about 0.333328 and an adjusted count of about 3.00005. These gaps are tiny, but they show why the stored value, not the intended one, should be used when computing counts.
| Target keep rate | Stored threshold (4 digits) | Real keep probability | Adjusted count of one kept span |
|---|---|---|---|
| 1 in 2 | 8 | Exactly 0.5 | Exactly 2 |
| 1 in 3 | aaab | About 0.333328 | About 3.00005 |
| 1 in 10 | e666 | About 0.100006 | About 9.9994 |
| 1 in 100 | fd70a | About 0.0100002 | About 99.998 |
For everyday use the difference is invisible. It matters only if you compare very precise totals, in which case you should always rely on the recorded threshold.
Why powers of two are special
Rates like one half, one quarter and one eighth fit perfectly into binary, so their thresholds are short and exact. An earlier line of thinking about consistent sampling restricted probabilities to powers of two for exactly this reason: it makes the arithmetic simple. The newer approach keeps the same shared-number idea but allows nearly any probability by using a full-width threshold, which removes the awkward restriction without giving up consistency.
26A Full Worked Example with Real Numbers
Let us put every idea together in one story, using the friendly zero-to-one-hundred ruler so the arithmetic fits in your head.
A food-delivery company has a gateway, an order service and a kitchen service. Ten thousand customers place orders in one hour. The gateway is the root and keeps twenty percent of traces. Remember the ruler: twenty percent kept means a threshold of eighty. The order and kitchen services follow the parent. A collector then applies a proportional reduction to fifty percent of what arrives.
The shared number is fixed for each order
Every order has a random value between zero and one hundred. About one in five of the ten thousand orders, roughly two thousand, have a number of eighty or more.
The root decision
The gateway keeps those roughly two thousand orders and records a threshold equal to a twenty percent keep rate. The other eight thousand are dropped at the root, and with parent-based followers their child spans are dropped too.
Each kept trace is complete
Because the order and kitchen services copy the parent decision, each of the two thousand kept orders has spans from all three services. There are no half traces.
The collector thins proportionally
Multiplying the incoming twenty percent by the collector’s fifty percent gives a new keep probability of ten percent, a threshold of ninety on the ruler. Only spans whose number is ninety or more survive: roughly one thousand orders. These orders were a subset of the two thousand, and all three of their spans survive together.
The backend estimates the truth
Each stored span now carries a keep probability of ten percent, so its adjusted count is ten. About one thousand kept orders multiplied by ten gives an estimate of ten thousand orders in the hour, which matches reality.
flowchart LR
A["10,000 orders"] --> B["Gateway keeps R at or above 80: about 2,000"]
B --> C["Collector keeps R at or above 90: about 1,000"]
C --> D["Backend: each span counts as 10, estimate 10,000"]
Notice what would have gone wrong with independent coin flips. If the collector had flipped a fresh fifty percent coin for each span, a kept order would often have lost its kitchen span or its gateway span, and the three-service story would have been broken for about three quarters of the surviving orders. The shared number is what makes the collector’s reduction remove whole orders instead of random pieces.
27Consistency Beyond a Single Trace
Sometimes the thing you care about is bigger than one trace, such as a user session or a batch job, and the shared number can be stretched to cover it.
Most of this tutorial treats one trace as the unit of consistency. But the explicit randomness value is just a number placed in a header by the root sampling decision, and nothing prevents the same number from being given to several different traces on purpose. When that happens, those traces rise and fall together: for any threshold, either all of them are above it or all are below it.
Whole user sessions
A website could hand the same random value to every request in one visit. A visitor whose number is high gets almost all their requests traced, and a visitor whose number is low gets almost none. When an investigator opens a complaint, the whole visit is present instead of scattered clicks.
Linked batch work
A nightly job that triggers many small tasks, each starting its own trace, can share one number so that either the whole run is visible or none of it is. The tasks still make their own decisions but they agree.
Translating other systems
When a decision has already been made by another consistent mechanism, such as a hash of an identifier, the hash can be turned into an explicit random value. Later stages can then keep recording the probability using the same threshold language while preserving the original decision.
Only the root decision may set the explicit value, and sharing it across traces makes those traces correlated. That is the point, but it also means that statistics computed across them are no longer independent. A handful of heavy sessions with high numbers can swing totals, so be careful when estimating variability.
A teacher who gives every student in the same family the same lottery ticket will find that siblings are either all picked or all skipped. That might be exactly what a family photograph needs, but you would not use the number of photographed siblings as an independent measure of how popular the school is.
28Planning Cost and Capacity with Keep Rates
Because the keep rate is a plain number, you can plan budgets with simple multiplication before you change a single setting.
The cost of tracing grows with the amount of data you keep, which is the number of requests multiplied by the number of spans per trace, multiplied by the size of each span, multiplied by the keep rate. Everything except the keep rate is determined by your system. That makes the keep rate the main dial for cost, and consistent sampling makes the dial predictable: cut the rate in half and you cut the kept data roughly in half, without damaging trace completeness.
| Illustrative scenario | Requests per day | Spans per trace | Keep rate | Spans stored per day |
|---|---|---|---|---|
| Small service | 1 million | 10 | 100% | 10 million |
| Same service, tuned | 1 million | 10 | 10% | 1 million |
| Large platform | 500 million | 40 | 1% | 200 million |
| Large platform, wider net | 500 million | 40 | 5% | 1 billion |
The numbers in this table are illustrative, not measurements of any real company, but the lesson is real: a handful of percentage points at a large scale means hundreds of millions of spans. Also notice that spans per trace matters as much as the keep rate. A trace that is twice as deep costs twice as much per kept request, so reducing needless spans in the application can be as valuable as lowering the keep rate.
A simple planning routine
Set a storage budget
Decide how many spans per day your backend can comfortably hold and pay for.
Measure unsampled volume
Estimate requests per day and average spans per trace, perhaps from a short period at a high keep rate.
Divide to find the keep rate
Budget divided by unsampled volume gives a starting keep rate. Round down to a clean value and write it as a threshold.
Verify with adjusted counts
After the change, compare the estimated total from adjusted counts with an independent request count. If they agree, your plan and your thresholds are both sound.
A fixed keep rate means cost rises with traffic. Where traffic is spiky, consider raising the threshold temporarily, for example during a sale. Since the new threshold is recorded, the numbers on your dashboards stay correct throughout.
29Testing, Rollout and Troubleshooting
A careful rollout plus a small troubleshooting table will save you from most surprises.
How to test without touching production first
Before changing anything in production, run a test environment with synthetic traffic that touches every service. Use a high keep rate to gather plenty of data, then check four properties in turn. First, check completeness: nearly every kept trace should have all its expected spans. Second, check nesting: if you lower the keep rate at one stage, the surviving traces must be a subset of the earlier survivors. Third, check the recorded thresholds: every kept span should carry the threshold you expect at its stage. Fourth, check the totals: the sum of adjusted counts should match the number of synthetic requests you generated.
A cautious rollout order
Observe only
Turn on reporting of threshold and random flag coverage before changing any sampling rate, so you have a baseline.
Change the root first
Update the gateway or main entry point and watch that traces stay whole in the services behind it.
Add downstream stages one at a time
Introduce collector sampling in one pipeline, verify totals, then repeat for others.
Keep a rollback ready
Know how to return to the previous configuration quickly. Sampling changes are easy to reverse because they only affect what is kept from then on.
Symptom and likely cause
| Symptom you see | Likely cause | What to check |
|---|---|---|
| Many traces are missing spans from one service | That service makes its own independent decision or drops the trace state | Its sampler type and whether it forwards the OpenTelemetry entry |
| Dashboards under-report volume after a change | A stage dropped data without raising the recorded threshold | The threshold on spans leaving each stage |
| Kept share is far from the configured rate | The identifier is not actually random, or the random flag is missing | How identifiers are generated and the random flag coverage |
| Many spans have no threshold | Some stage erases it as an unknown-probability stage or an inconsistency | Parent-based stages receiving context with no threshold |
| Costs jump unexpectedly | Untrusted callers force spans to be kept | Edge handling of incoming trace context |
When something looks wrong, pick one kept trace and walk it span by span, writing down the threshold and the source of randomness at each hop. The first hop where the story changes is almost always where the bug lives.
30How This Compares with Hash-Based Sampling
Many existing systems already reach consistency another way, by hashing the trace identifier, so it helps to see the similarities and differences.
A very common technique takes the trace identifier, runs it through a hash function, and compares the result with a keep rate. Because every service performs the same calculation on the same identifier, they reach the same answer. That is a consistent scheme, and it works well when every participant uses the same hash function and the same seed. Its weak spot is the word same: different languages, libraries or versions often implement the hash slightly differently, and even a small difference breaks agreement across a system.
The OpenTelemetry approach moves the agreement from the calculation to the data. Instead of every participant needing to run the same hash, they only need to read the same shared number and compare it with a threshold, a trivial operation that every language does identically. The threshold is then written into the trace state, which also gives the final backend the knowledge of the keep probability that a pure hash scheme normally lacks.
| Aspect | Hash-based consistent sampling | Shared randomness and threshold |
|---|---|---|
| What must everyone agree on | The exact hash function and seed | Only the meaning of the number and the comparison |
| Risk across languages and versions | Higher, because small implementation differences break agreement | Lower, because the work is an integer comparison |
| Keep probability known downstream | Usually not recorded | Recorded as the threshold |
| Further thinning by collectors | Possible, but statistics are harder to keep honest | Built in with raise-only thresholds |
| Using non-trace data such as logs | Needs extra conventions | Randomness and threshold can be recorded as attributes |
Hash-based agreement is like two cooks who must each follow an identical recipe to produce an identical dish. Shared randomness is like handing both cooks the same finished ingredient and asking only whether it is above a line. The second method has far fewer ways to go wrong.
Existing hash-based decisions do not have to be thrown away. The hash can be mapped into a randomness value, so that systems that started with hashing can still describe their decisions with thresholds and fit into the same ecosystem.
31Status and Ecosystem Today
A tutorial should be honest about how settled the topic is, so here is where things stand.
The OpenTelemetry specification page on probability sampling carries the status Development, in the specification release numbered 1.61.0 at the time of writing. That label means the mechanism is documented and implemented in places, but its details are still allowed to evolve. Treat the concepts in this tutorial, the shared randomness and the rejection threshold, as the stable core, and check the current specification before depending on a fine detail such as an exact field name or default.
Probability sampling in trace state
Defines the shared randomness, the rejection threshold, the decision rule, and the downstream sampler behaviours described in this tutorial.
Trace state handling
Defines how the threshold and the explicit randomness value are written into the OpenTelemetry trace state entry.
Threshold and composite sampler proposals
The enhancement proposals behind the threshold encoding and the composable sampler design that the specification references.
Probabilistic sampler processor
A collector processor that can apply consistent sampling to spans, and which also supports a hash-based mode whose decisions can be mapped onto the same randomness and threshold values. For logs, which have no trace state, it can record the randomness and threshold as attributes.
Because the specification is still in development, check the official OpenTelemetry documentation and the release notes of your tracing libraries and collector before rolling out. Support and defaults differ by language and version.
32Frequently Asked Questions
It is a way for every service and collector to decide independently, but in agreement, whether to keep a span, by comparing one shared random number with a personal threshold, and then recording that threshold so the keep probability stays known.
No. The shared random number travels inside the normal trace context, and each sampler needs only its own configuration. There is no coordinator and no extra network call.
Because it measures how much of the number range is rejected. Spans whose random number falls below it are dropped, and spans at or above it are kept.
Either from the last seven bytes of the trace identifier, when the library promises they were generated randomly, or from an explicit random value stored in the OpenTelemetry trace state entry.
Yes. Their decisions nest: whatever the stricter sampler keeps, the more generous one also keeps. They do not produce identical sets, but they never contradict each other.
Lowering it would claim that an earlier stage was more generous than it truly was, which would make estimated counts too small. Raising it honestly records an extra round of thinning.
It means nothing is rejected, so every span is kept. It stands for a keep probability of one hundred percent.
Its keep probability is unknown, so it cannot be used for adjusted counts. It may still be stored and viewed like any other span.
Not by using the shared random number alone, since errors are not known at the start of a request. Rule-based and tail approaches can favour errors, provided they choose their own thresholds and record them honestly.
No. Probability sampling is defined from a tiny positive probability up to one hundred percent. Dropping everything is valid configuration but is not a form of probability sampling and carries no threshold.
No. They complement each other. Consistent probability sampling gives a cheap, correct baseline, and tail techniques can be layered on top for smarter choices.
The specification is marked as in development, and support varies by language and tooling. Many teams adopt the ideas gradually, starting with root-only ratio sampling and parent-based followers, and verify support in their own libraries first.
33Summary and Key Takeaways
We began with a courier company whose warehouses flipped private coins and ended up with an album full of fragments. Consistent probability sampling replaces those private coins with one shared lucky number on the parcel label and a personal cut-off line at every warehouse. Because everyone reads the same number, the decisions nest neatly, whole traces survive at the configured rate, and different services can still choose different rates. By recording the threshold that was applied, the system also remembers how many real spans each kept span stands for, so totals, rates and dashboards remain trustworthy even though most data was never stored. The scheme is cheap, needs no coordination, and fits naturally into the headers that already travel with every request, though its specification is still in development and deserves a fresh look before you commit to details.
Key Takeaways
- One shared number — a 56-bit randomness value, taken from the trace identifier or an explicit value, is read by every participant.
- One simple rule — keep a span when the random value is at or above the rejection threshold, otherwise drop it.
- Nested decisions — stricter samplers always keep a subset of what more generous samplers keep, which is what consistency means.
- Threshold recorded, probability known — kept spans carry their threshold in the trace state, and a missing threshold means unknown probability.
- Raise, never lower — downstream samplers can only increase the threshold, either equalizing to a common rate or reducing proportionally.
- Adjusted counts — one divided by the keep probability turns sampled spans back into honest totals.
- Mind the edges — protect trust boundaries, plan migrations carefully, and monitor threshold coverage and trace completeness.