If you only have two minutes, this chapter gives you the whole idea. The remaining chapters explain why it works and how to use it well.

TraceIdRatioBased is a built-in OpenTelemetry sampler. A sampler is the part of the SDK that decides, when a new span starts, whether its trace should be recorded and exported or thrown away. TraceIdRatioBased makes that decision using two things only: the trace ID, a long random number that names the trace, and a ratio between zero and one that you configure.

In simple terms, it treats the trace ID as a number, checks whether that number falls inside the lowest fraction of all possible numbers, and keeps the trace if it does. With a ratio of 0.25, about a quarter of all trace IDs fall inside that range, so about a quarter of traces are kept. The decision depends only on the ID and the ratio, which means the same trace gets the same answer everywhere.

Before learning how to keep fewer traces, it helps to see why anyone would throw data away on purpose.

A trace is the story of one request as it moves through your system. It is made of spans, where each span is one timed step, such as “check the cart” or “query the database”. A single request in a system of many small services can easily create twenty or fifty spans.

Now multiply. If a service handles one thousand requests per second and each creates thirty spans, that is thirty thousand spans every second, which is about 2.6 billion per day. Each span carries a name, timestamps, identifiers, and attributes, so it may weigh a kilobyte or more. At one kilobyte each, that is roughly 2.6 terabytes of trace data per day from a single modest service. Storing it, indexing it, and searching it costs real money, and sending it uses network and processor time inside your applications.

Most traces look alike

The second half of the story is that most of that data is boring. If ninety-nine requests out of a hundred are healthy and fast, their traces teach you almost the same lesson each time. Keeping one in ten of them still shows you what normal looks like, and costs a tenth as much. The interesting traces, which are the slow ones and the broken ones, are a small minority, and you would like to keep as many of those as you can.

The trade you are making

Sampling trades completeness for cost. You give up the ability to look up any individual request, and in return you pay less and stay fast. A good sampling plan makes that trade deliberately, so you know what you gave up and what you kept.

A short list of words appears in every sampling conversation. Learn these once and the rest of the guide reads easily.

Two ideas hiding in the table

First, sampling is decided per trace, not per span. The goal is to keep whole stories, not random sentences from them. If you kept a random half of the spans from every trace, no trace would be readable. Good sampling keeps a trace in full or drops it in full.

Second, a sampled trace is a trace that is recorded and sent onward. An unsampled trace may still be created in a lightweight form so that its identifiers can travel to other services, but its details are not kept. This difference between “still travels” and “is kept” becomes important when you meet the ParentBased wrapper later.

The sampler works at the very beginning of a span’s life. Seeing the whole path shows how much work sampling saves.

flowchart LR
    CODE["Application code starts a span"] --> TR["Tracer"]
    TR --> SAM{"Sampler decides"}
    SAM -- "drop" --> NOOP["Span is not recorded or exported"]
    SAM -- "record and sample" --> SP["Span records data"]
    SP --> PROC["Span processor"]
    PROC --> EXP["Exporter"]
    EXP --> BE["Collector or backend"]

Reading the picture

Your application code asks the tracer to start a span. Before anything else happens, the tracer asks the sampler a question: “Should this trace be recorded?” If the answer is no, the span becomes an almost empty placeholder. It records no attributes, creates no events, and never reaches the processor or the exporter. If the answer is yes, the span records its data as usual and flows down the pipeline toward the backend.

This position at the very front is what makes sampling powerful for cost. A dropped trace saves everything downstream: the memory for attributes, the work in the span processor, the effort of packing and compressing, the network trip, and the storage at the end. Dropping data late, for example in the backend, saves only storage.

What the sampler can see

When it decides, the sampler is handed a small set of facts: the parent context if any, the trace ID, the span name, the span kind such as server or client, the attributes known at the start, and any links. It does not see how long the span will take, whether it will fail, or what its children will do, because none of that has happened yet. This is the heart of what “head sampling” means.

Every sampler, built in or custom, gives one of a small set of answers. Understanding them explains how unsampled traces behave.

What the sampler also returns

Besides the decision, a sampler can return a few extra attributes to add to the span, and it can update the trace state, which is a small bag of vendor-specific key-value pairs that travels with the trace. Simple samplers like TraceIdRatioBased usually add nothing extra. Newer, more advanced samplers use the trace state to record the sampling probability so that backends can later correct their counts.

The sampled flag

The decision turns into a single bit in the trace context called the sampled flag. When a service calls another service, it sends this flag along in a header, and the receiving service reads it. In the common W3C Trace Context format, the header is called traceparent and ends with two hexadecimal digits of flags, where the last bit means sampled. This is how a decision made once at the front door can be respected by every service behind it.

OpenTelemetry ships with a small family of samplers. Meeting the whole family makes it clear what TraceIdRatioBased is for.

How they combine

The first three are leaf samplers, which make a decision by themselves. ParentBased is a composite sampler, which wraps other samplers. A typical production setup combines them: ParentBased, with TraceIdRatioBased as the choice for new traces, and with the default behavior of following the parent for continuing traces. Many SDKs use a parent-based always-on arrangement as their default when you set nothing.

The important detail about the parent

TraceIdRatioBased ignores the parent completely. It looks at the trace ID and the ratio, nothing else. If a service receives a request whose upstream already decided “sampled”, a bare TraceIdRatioBased sampler may still say “drop”, if the ID happens to fall outside its ratio, which can leave holes in the trace. This is why the ParentBased wrapper exists, and why it deserves its own chapters.

The idea is small enough to explain with one picture of a number line.

A trace ID is just a big number

A trace ID in OpenTelemetry is sixteen bytes long, which is 128 bits. It is usually written as thirty-two hexadecimal characters. Hexadecimal is a way of writing numbers using the digits 0 to 9 and the letters a to f. Underneath, it is simply a very large number. The ID generator inside the SDK picks it at random, so across many traces the numbers are spread evenly over the whole range, much like random lottery tickets.

A ratio is a slice of the number line

Picture the range of all possible IDs as a long ruler. A ratio of 0.1 means “take the first tenth of the ruler”. The sampler converts your ratio into a threshold, which is a particular position on the ruler. If a trace’s ID, turned into a position, lands before the threshold, the trace is kept. If it lands after, the trace is dropped.

flowchart LR
    R["Ratio 0.25 chosen by you"] --> TH["Threshold at one quarter of the ruler"]
    ID["Trace ID read as a number"] --> CMP{"Is the ID below the threshold?"}
    TH --> CMP
    CMP -- "yes" --> KEEP["Record and sample"]
    CMP -- "no" --> DROP["Drop"]

Why this keeps about the right amount

Because IDs are random and evenly spread, any slice of the ruler holds about the same share of traces as its share of the ruler. A quarter of the ruler holds about a quarter of the traces. This is the same reasoning that makes lottery numbers fair: no ticket is more likely than another, so a fixed slice catches a predictable fraction.

The result is a decision that is cheap, because it is one comparison of two numbers. It needs no memory of earlier traces, no shared counter, and no talk with other services.

Here is exactly what happens inside the sampler each time a span starts. The steps are few, and each is fast.

Why only part of the ID

Using the lower 64 bits is a practical choice. A 64-bit number fits a standard integer type in most languages, so the comparison is fast and simple. It is also common for systems to generate IDs with a random lower half, even when other parts carry extra meaning. Some trace ID formats, such as those that place a timestamp in the first part, would give uneven results if the sampler read that part. Reading the lower bytes avoids this trap. The details differ by implementation and are covered by the language SDKs, so do not assume your SDK uses exactly these bytes without checking.

The description string

Samplers have a human-readable description that shows up in debugging output. For this sampler, it contains the name and the ratio, for example TraceIdRatioBased{0.100000}. Seeing this in logs is a quick way to confirm the ratio your service actually uses, which is handy when environment variables may have been overridden.

The decision depends only on the ID and the ratio. That single fact is behind almost every advantage of this sampler.

Agreement without conversation

Suppose a request passes through five services, and every one of them uses TraceIdRatioBased with a ratio of 0.1. Each service sees the same trace ID, so each one reaches the same answer. There is no need for the services to ask one another. If the first service decided to keep the trace, the second, third, fourth, and fifth would also decide to keep it, even if the flag was lost on the way. This gives you whole traces instead of fragments.

Repeatable behavior

Because there is no dice, the sampler behaves the same way every time. If you replay a trace ID through the sampler, you get the same answer. This is good for testing and reasoning. You can write a test that feeds known IDs to the sampler and checks the expected decisions.

No shared state

Many other sampling approaches need counters or coordination, such as a service that tracks how many traces were kept in the last second. TraceIdRatioBased keeps no state at all. Two copies of a service running on two machines make consistent choices without talking, and a restarted service behaves exactly like it did before.

flowchart TB
    ID["Trace ID 5b8e... same in every service"] --> A["Service A with ratio 0.1"]
    ID --> B["Service B with ratio 0.1"]
    ID --> C["Service C with ratio 0.1"]
    A --> DA["Same decision"]
    B --> DB["Same decision"]
    C --> DC["Same decision"]

Where agreement stops

The agreement holds only when the services use the same ratio, the same way of reading the ID, and the same ID. If two services use different ratios, or use different language SDKs that read the ID differently, they can disagree. This is one reason to rely on the parent’s flag as the main way of keeping traces whole, and to treat the deterministic agreement as a useful backup.

The fairness of the whole scheme depends on one assumption. If it fails, the numbers you get can be quietly wrong.

The sampler assumes that trace IDs are spread evenly over their range. OpenTelemetry’s default ID generator produces random IDs, so this is normally true. Problems appear when something else creates the IDs. A custom generator might include a counter or a timestamp. A different tracing system, an older one, or a gateway might create IDs with patterns. If the part of the ID that the sampler reads is not random, the share of traces kept can drift far from your ratio.

An example of the failure

Suppose a custom generator builds IDs whose last bytes come from a counter that increases by one per trace. Then the number read by the sampler also increases steadily. With a ratio of 0.1, the sampler’s threshold is fixed, so as the counter passes it, the sampler would keep nothing for a long time, or everything for a long time, depending on where the counter began. Your carefully chosen ratio would produce a very different outcome from the one you expected.

The standards are catching up

The W3C Trace Context standard recommends that trace IDs be random and, in its newer revisions, describes which part of the ID is guaranteed to be random so that samplers can rely on it. OpenTelemetry’s specification has also been evolving toward sampling methods that use an explicit random value carried in the trace state, so that the assumption does not depend on how the ID was made. Because these details are still moving, check your SDK’s current documentation for the status of the older TraceIdRatioBased and any newer probability-based samplers before you rely on fine points.

The ratio is a number between zero and one. What happens at the ends, and outside them, is worth knowing before an odd value reaches production.

What the values at the ends tell you

A ratio of one is a handy way to say “keep everything” while still using the same configuration style as other environments. A ratio of zero is a quiet way to turn off tracing for a service while keeping trace identifiers moving. With zero, the sampler says drop, so no data is kept, yet the context still travels so that downstream services can follow the flag.

Mistakes with percentages

A common mix-up is writing a percentage where a ratio is expected. A setting of 10 is read as ten, which is above one, and therefore means “keep everything”. A person who meant ten percent sees a flood of data and a large bill. Another version is writing 0.1 when the system expects a percentage, such as some Collector settings, which would mean a tenth of a percent. Always check the unit named in the setting.

Very small ratios

Ratios like 0.0001, which is one in ten thousand, are valid and are used for enormous systems. The sampler still needs a decent amount of traffic to behave predictably. At one hundred traces per day, a ratio of 0.0001 would keep a trace only about once every hundred days, which is probably not what anyone wants. Pick small ratios only when traffic is huge.

TraceIdRatioBased is a head sampler. That label explains its biggest strength, which is cheapness, and its biggest weakness, which is blindness.

Head sampling means the decision is made at the head, or start, of the trace. At that moment the system knows almost nothing about what will happen. It does not know whether the request will be slow, whether it will fail, or whether the customer is important. TraceIdRatioBased uses only the ID, so it is blind to all of these, by design.

The strength: it is cheap and saves everything

Because the decision happens first, unsampled traces cost almost nothing. No data is collected, no memory is used, and nothing is sent. The savings extend through the whole pipeline. This is why head sampling is so popular for very large systems.

The weakness: it treats all traces alike

Suppose one request in a thousand fails with a rare, serious error. With a ratio of 0.1, only about one tenth of those failing traces is kept. Nine out of ten rare failures leave no trace at all. The sampler did not choose to ignore errors. It simply could not see them, because they had not happened yet at the moment of the decision.

flowchart LR
    START["Request begins"] --> HEAD{"Head sampler decides now"}
    HEAD -- "kept" --> RUN1["Request runs and is fully traced"]
    HEAD -- "dropped" --> RUN2["Request runs and is not traced"]
    RUN1 --> OUT1["Outcome is known only afterwards"]
    RUN2 --> OUT2["A failure here leaves no trace"]

Who should accept this trade

Head sampling by ratio suits teams that mainly want to understand normal behavior, spot broad trends, and keep costs predictable. If you must keep every error trace, or every very slow trace, you need a different tool, called tail sampling, which comes later in this guide. Many real systems combine both.

In real systems, TraceIdRatioBased almost never works alone. It is placed inside a wrapper that respects decisions made earlier in the trace.

The ParentBased sampler asks a first question: does this span have a parent? If it does, the sampler copies the parent’s decision. If the parent was sampled, this span is sampled. If the parent was not, this span is not. Only when there is no parent, which means this is the root of a new trace, does it fall back to a different sampler, called the root sampler. You choose TraceIdRatioBased as that root sampler.

flowchart TB
    START["Span starts"] --> HASP{"Does it have a parent context?"}
    HASP -- "no, this is a root" --> ROOT["Use the root sampler: TraceIdRatioBased"]
    HASP -- "yes" --> REM{"Is the parent remote or local?"}
    REM -- "remote and sampled" --> RS["Sample"]
    REM -- "remote and not sampled" --> RN["Drop"]
    REM -- "local and sampled" --> LS["Sample"]
    REM -- "local and not sampled" --> LN["Drop"]

Why this wrapper matters

Without it, every service in a chain applies its own ratio rule to every span, even when an earlier service already made a decision. With it, the decision is made once, at the root, and every later span simply follows. The trace is either kept whole or dropped whole. This fits the most important goal of sampling, which is to keep complete stories.

The default behavior of the wrapper

The wrapper has separate settings for four cases: remote parent sampled, remote parent not sampled, local parent sampled, and local parent not sampled. A remote parent is a span in another service, and a local parent is a span in the same program. The usual defaults are to follow the parent in all four cases, which is what you want almost always.

A table makes the four parent cases and the root case easy to see together. Keep it handy when debugging odd results.

An example with three services

Take a request that goes from a gateway to an orders service, and then to a payments service. All three use ParentBased with a root ratio of 0.1. The gateway receives a request with no trace context. It is a root, so the ratio rule applies. Suppose the ID falls below the threshold. The gateway samples the trace and sends the flag on. The orders service sees a remote parent with the flag on, so it samples. Payments does the same. The trace is whole.

In another request, the ID falls above the threshold. The gateway drops the trace and sends the flag as off. The other two services see the flag off and drop their spans. Nobody wastes effort, and no half-trace appears.

A request that arrives with context already attached

Now imagine that the gateway receives a request from a mobile application that already carries a trace context with the flag on. The gateway is no longer a root. It has a remote parent that is sampled, so it samples, regardless of its own ratio. This is a feature when the caller is trusted, and a risk when it is not, as the security chapter explains.

You usually choose the sampler through settings rather than by writing logic. Knowing the names saves guessing.

The environment variables

OpenTelemetry defines two standard environment variables for samplers. One, named OTEL_TRACES_SAMPLER, picks the sampler by a name. The other, named OTEL_TRACES_SAMPLER_ARG, gives it an argument such as the ratio. The names that matter here are shown below.

The setting you probably want

For most services, the right choice is the parent-based version of the ratio sampler, with the ratio as the argument. This matches the wrapper pattern from the previous chapters. The default when you set nothing is generally the parent-based always-on arrangement, which keeps everything that starts here and follows what arrives. That surprises people who expected sampling to be off by default, and who later discover large volumes.

Setting it in code instead

All SDKs let you choose a sampler when creating the tracer provider, using the same building blocks: a ratio sampler wrapped by a parent-based sampler. Code settings are useful when the choice depends on the environment or on logic. Settings from environment variables are easier for operations teams, since they change without a rebuild. If both exist, check your SDK’s rule for which wins.

The ratio is a business decision wearing a technical coat. A few simple multiplications turn it into something you can argue about.

The volume formula

The number of spans you keep per second is the number of requests per second, times the average spans per trace, times the ratio. For example, one thousand requests per second, with twenty spans per trace, at a ratio of 0.05, keeps one thousand times twenty times 0.05, which is one thousand spans per second. At a ratio of one, you would keep twenty thousand. The ratio scales cost directly.

Cost drivers to multiply by

How to pick

Begin with a budget. If you can afford to store five hundred spans per second, and your traffic would produce twenty thousand, the ratio is five hundred divided by twenty thousand, which is 0.025. Then ask whether 0.025 gives you enough examples to answer your questions. If you rarely need individual traces and mostly want trends, a very low ratio is fine. If your engineers hunt individual bugs every day, you may need a higher ratio, or a smarter strategy such as tail sampling for errors.

The following story is invented to show the arithmetic from start to finish. The company and figures are not real.

Step 1: the full volume

At full capture, the volume is 2,000 requests times 25 spans, which is 50,000 spans per second at peak. That is fifty times the budget.

Step 2: the ratio

To fit within 1,000 spans per second, the team needs a ratio of 1,000 divided by 50,000, which is 0.02. They configure parent-based with a root ratio of 0.02 on the entry services.

Step 3: what they get

Out of 2,000 requests per second, about 40 traces are kept each second at peak. Over an hour of peak traffic, that is about 144,000 traces, which is plenty to study normal behavior, to compare versions, and to find slow endpoints. At night, with 100 requests per second, the same ratio keeps only about 2 traces per second, still adequate for trends.

Step 4: what they lose

A rare error happens about 30 times per hour. At a ratio of 0.02, the team expects to keep 0.6 of those traces per hour, which means a typical hour holds no trace of the error at all. This is the head-sampling gap. The team decides to add tail sampling for errors in a Collector, as a second layer, and keeps the 0.02 ratio for ordinary traffic.

Sampling is a game of chance, and a little probability tells you what to expect. No advanced mathematics is needed.

Will I see at least one example?

If each trace is kept with probability p, the chance that a particular trace is not kept is one minus p. If an event happens n times, the chance that none of its traces is kept is that value multiplied by itself n times. The chance that at least one is kept is one minus that.

The lesson is that low ratios are good at showing common patterns and poor at showing rare ones. If a problem happens only a few dozen times per day, a one percent ratio may well miss it entirely.

How close is the kept share to the ratio?

Because the process is random-like, the actual share fluctuates slightly. For one million traces at a ratio of 0.1, you expect about 100,000 kept. The typical wobble, called a standard deviation, is about 300, which is only 0.3 percent of the expected number. So with large traffic, the share is very close to the ratio. With small traffic, such as one thousand traces, the wobble is about ten out of a hundred expected, which is ten percent, a much bigger relative swing. Do not be alarmed by noisy numbers on small volumes.

Real organizations rarely have one tidy setting everywhere. What happens when two services in the same trace use different ratios?

With the parent-based wrapper

If every service uses the parent-based wrapper, only the root’s ratio matters for traces that begin at that root. Services deeper in the chain follow the flag, so their own ratios are used only when they happen to start a new trace themselves, such as when a background job begins with no parent. This is clean and predictable, and it is the recommended arrangement.

Without the wrapper

If services use a bare ratio sampler, each applies its own ratio to the same trace ID. A service with ratio 0.5 and another with 0.1 will often disagree. Because the rule is “keep if the ID is below the threshold”, and a lower ratio means a lower threshold, the 0.1 service keeps only traces that the 0.5 service also keeps, assuming both read the ID in the same way. So the traces from the 0.1 service form a subset of the other’s. In traces kept by the 0.5 service but not the 0.1 service, you will see spans from the first and a hole where the second should appear.

flowchart LR
    subgraph REQ["One request with trace ID X"]
      A["Service A ratio 0.5"] --> B["Service B ratio 0.1"]
    end
    A --> DA{"ID below 0.5 threshold?"}
    B --> DB{"ID below 0.1 threshold?"}
    DA -- "yes" --> KA["A spans kept"]
    DB -- "no" --> DR["B spans dropped, hole in trace"]

When differences are intentional

Sometimes you want a downstream service to keep more of its own traces, for example a service under investigation. The parent-based wrapper blocks this when traces arrive from upstream, because the flag wins. If you really need a different rate in the middle, consider a separate, deliberate mechanism, such as a custom sampler or a Collector rule, and accept the possible partial traces knowingly.

The worst outcome of bad sampling is not less data. It is data that misleads. Partial traces are the main example.

A partial trace is a trace missing some of its spans. In a trace viewer, you see a child span whose parent never arrived, or a gap where a service should appear. The viewer may draw the tree with a broken top, or may label missing parents as unknown. A reader might wrongly conclude that a service was never called, or that time vanished between two steps.

Common causes

Why partial traces are worse than none

Missing traces simply mean less data, and you know to be careful. A partial trace appears legitimate and may be used for a conclusion. An engineer might blame a service for a delay that actually happened in an unseen service. Backends try to cope by showing placeholders or waiting for late spans, but they cannot invent what was never sent.

The cure

Use the parent-based wrapper everywhere, make sure context travels through every hop including queues, keep all services on the same sampling plan, and watch for drops downstream of the sampler. The next chapter explains the context part in more detail.

The decision made at the root reaches the other services through a small piece of text attached to each call. Following it once makes the whole system clear.

sequenceDiagram
    participant U as User
    participant G as Gateway
    participant O as Orders service
    participant P as Payments service
    U->>G: Request with no trace context
    Note over G: New root, TraceIdRatioBased decides
    G->>O: Call with traceparent flag sampled
    Note over O: Remote parent sampled, so sample
    O->>P: Call with traceparent flag sampled
    Note over P: Remote parent sampled, so sample
    P-->>O: Response
    O-->>G: Response
    G-->>U: Response

What is inside the header

In the common W3C Trace Context format, the header named traceparent has four parts separated by dashes: a version, the trace ID, the ID of the calling span, and a set of flags. The last bit of the flags is the sampled flag. When a service makes a call, its OpenTelemetry instrumentation fills in this header automatically. When a service receives a call, the instrumentation reads it and passes the information to the sampler as the parent context.

Where propagation breaks

Propagation can fail in several ordinary ways. A proxy might strip unknown headers. A message queue might not carry headers unless the instrumentation copies the context into the message properties. A background job might start in a new thread without the context. A service using a different propagation format, such as an older one, might not understand the header. Each failure makes the next service see “no parent” and behave as a new root, which then uses the ratio, producing a new trace or a disconnected fragment.

Unsampled traces still carry their ID

Even when a trace is dropped, its context still travels so that downstream services can respect the decision. The flag says “not sampled”, the ID is still present, and the downstream spans are created in a lightweight form. This is why dropped traces still cost a little: a header on every call and a tiny object per span.

Sampling affects any number you compute from traces. If you forget this, your charts will quietly understate reality.

Suppose you derive a chart of requests per second from the stored spans. If only two percent of traces are kept, the chart shows two percent of the real traffic. A dashboard that says “40 requests per second” when the true figure is 2,000 will mislead anyone who forgets about sampling. Latency percentiles computed from a sample are less affected, if the sample is fair, but rare slow requests are underrepresented, and the high percentiles are noisier.

Three ways to keep numbers honest

Advanced approach: carry the probability

More recent OpenTelemetry designs record the sampling probability in the trace state of each sampled span, so a backend can correct its counts per span even when ratios differ across services or change over time. These designs are still maturing, so treat this as a direction rather than a given, and check the current status in your tools.

Logs and traces are often linked by the trace ID. Sampling can break that link in a way that surprises people.

OpenTelemetry log records can carry the trace ID and span ID of the span that was active when the log was written. This lets a viewer jump from a log line to its trace. But if the trace was not sampled, the trace does not exist in the backend. The log line still holds the ID, the link is still shown, and clicking it finds nothing.

How teams handle this

• Log the sampled flag too. Many setups include the trace flags in the log record, so a viewer can show the link only for sampled traces, and avoid dead links.
• Accept the dead link. For many teams, a link that sometimes leads nowhere is a small price and is explained in a short note in the log viewer.
• Choose logs differently. Some teams keep every log line but sample traces, since logs for errors are valuable on their own. They set logging levels so the key lines are always kept.
• Make interesting traces sticky. A tail-sampling component keeps all error traces, so error logs usually have a working trace link.

A note about sampled logs

The sampler described here decides about traces, not logs. Logs are not dropped by TraceIdRatioBased. If you need to reduce log volume, use other means, such as log levels, filters, or a Collector processor. Mixing up the two leads to surprise on both sides: traces missing when logs exist, and logs flowing when traces were reduced.

Newcomers often confuse the sampler with the span processor, since both sit in the same pipeline and both affect how much data leaves your program.

They work together

Sampling reduces how many spans reach the processor. The Batch processor then packs those spans efficiently. If you sample too little and send too much, the processor’s queue fills and begins to drop spans in an uncontrolled way, which is the worst kind of sampling, because it ignores the trace structure and can produce partial traces. A good setup uses the sampler to cut volume on purpose, so the processor and exporter never have to drop anything by accident.

Why order matters

Sampling must come first. If you let everything through and then drop spans at the processor, you pay the cost of creating them, and you lose the property of keeping or dropping whole traces. Deliberate trace-level sampling is both cheaper and cleaner than accidental span-level dropping.

Sampling does not have to happen inside your application. The OpenTelemetry Collector can sample too, and the two places complement each other.

The Collector is a separate program that receives telemetry, changes it, and forwards it. It has a component often called the probabilistic sampler processor, which keeps a configured percentage of traces based on a hash of the trace ID. It follows the same idea as TraceIdRatioBased: a deterministic function of the ID, so the same trace gets the same decision. It has its own settings, such as a percentage rather than a ratio, and an optional seed for the hash. Settings and names have changed across releases, so read the documentation of the version you run.

Why sample in the Collector

Choosing between them

Sampling in the SDK saves effort at the source and is the best way to cut application and network costs. Sampling in the Collector gives central control and saves backend costs. Many organizations do the broad cut in the SDK with a modest ratio and then apply targeted rules in the Collector. If you use both for the same purpose, remember that the rates multiply. An SDK ratio of 0.5 followed by a Collector percentage of 20 keeps about ten percent of traces, not twenty.

flowchart LR
    APP["Application SDK with ParentBased and TraceIdRatioBased"] --> COL["Collector"]
    COL --> PS["Probabilistic or tail sampling processor"]
    PS --> BE["Backend"]
    APP -. "dropped here costs nothing downstream" .-> X["Unsampled spans"]
    PS -. "dropped here still cost application and network" .-> Y["Unkept spans"]

If TraceIdRatioBased is blind to outcomes, tail sampling is the opposite. It waits until it has seen the whole trace and then decides.

A tail sampler holds the spans of each trace in memory for a short time, usually inside a Collector, until it believes the trace is complete. Then it applies rules: keep every trace containing an error, keep every trace slower than a limit, keep traces from a certain customer, and keep a small ratio of everything else. This lets you keep nearly all of the interesting traces while discarding most of the boring ones.

flowchart TB
    IN["Spans arrive from many services"] --> BUF["Buffer spans by trace ID"]
    BUF --> WAIT["Wait for the decision time"]
    WAIT --> RULES{"Apply rules to the whole trace"}
    RULES -- "has an error" --> KEEP["Keep"]
    RULES -- "slower than limit" --> KEEP
    RULES -- "none of the above" --> RATIO{"Random share"}
    RATIO -- "chosen" --> KEEP
    RATIO -- "not chosen" --> DROP["Drop"]

The price of seeing the whole story

The scaling difficulty

Because the sampler needs every span of a trace in one place, a group of Collectors must route spans by trace ID, so that all spans of one trace go to the same instance. This is often done with a first layer of Collectors that forwards by trace ID to a second layer that does the tail sampling. It works, but it is more to build, run, and watch than a ratio in an SDK.

Why TraceIdRatioBased still has a place

Tail sampling cannot reduce the cost of creating and sending spans, since all of them must reach the Collector first. Head sampling with a ratio reduces that cost at the source. This is why many large systems pair a modest head ratio, to tame volume, with a tail sampler, to rescue the interesting traces from what remains. The next chapter explains this combination and its catch.

The two styles can work as a team, but only if you understand how they interact. Otherwise one quietly cancels the other.

The catch

If the SDK uses a ratio of 0.1, then ninety percent of traces are dropped at the source. They never reach the Collector, so the tail sampler cannot rescue an error trace among them. The tail sampler can only choose among the traces it receives. Therefore, the overall chance of keeping an error trace is at most the head ratio, however clever the tail rules are.

Three common arrangements

Making the arrangement explicit

Write down, in plain words, the overall plan: what share of ordinary traces you keep, what you do to keep errors, and where each step runs. Include the multiplication of rates. A plan that lives only in people’s heads will drift when someone changes one piece without knowing the others.

A single ratio treats every trace alike and scales with traffic. Sometimes you want something a little smarter.

Different ratios for different routes

A health-check endpoint called every second adds no insight and creates huge volume. A checkout endpoint is precious. A custom sampler can look at the span name or attributes known at start, such as the route, and apply different ratios: very low for health checks, high for checkout, and moderate for the rest. Since the sampler sees attributes only at the start, it can use only information that exists then, such as the route and the method, not the result.

Rate limiting instead of ratios

A ratio keeps cost proportional to traffic. During a traffic spike, your cost rises with it. A rate-limiting sampler instead keeps at most a fixed number of traces per second, so cost stays flat. It needs state, namely a counter or a token bucket, so it is not deterministic by ID and does not agree across services without extra coordination. Some tracing systems offer remote-controlled sampling, where a central service tells each application what rates to use. The built-in OpenTelemetry list is shorter, and these ideas typically come from extensions or vendors, so check what your tools provide.

Choosing what to build

Do not build a custom sampler because it sounds powerful. Start with a parent-based ratio and see whether it meets the need. Many problems that look like they need a custom sampler are better solved by excluding noisy endpoints from instrumentation, or by sending errors through a tail sampler.

Sampling looks like a cost tool, but it touches security in three ways: who can force sampling, what is kept, and what the sampled data reveals.

Outsiders forcing your decision

With the parent-based wrapper, a service obeys the sampled flag of an incoming request. If your public gateway accepts a trace context from the internet, any caller can set the flag to “sampled” on every request. A malicious or buggy client could then force you to keep all of its traffic, raising your volume and cost, and could even attempt to flood the system. Callers can also pick trace IDs they choose, which is why ID-based rules should not be treated as a secret or a protection.

Defenses at the edge

• Ignore or reset untrusted context. At the public edge, start a fresh trace and do not accept the caller’s flag, or accept it only from known, trusted callers.
• Use different settings for edge and internal services. The parent-based wrapper can be set to ignore remote parents at the edge.
• Limit volume. A rate limit or a Collector-level cap protects the backend from a flood of forced samples.

What sampled data reveals

A sampled trace can contain sensitive information in its attributes, such as user identifiers, query text, or addresses. Sampling reduces how much of this is stored but does not remove it from the traces that remain. Keep sensitive values out of spans, and mask them in a processor or Collector, regardless of the ratio.

Sampling is not anonymization

Some people think that keeping a small share of traces makes the data safe. It does not. One sampled trace can still expose one customer’s details. Apply the same access control to sampled data as to any other telemetry.

A sampler works silently. To trust it, you need a few numbers that tell you it is doing what you intended.

Where to get the data

Request counts come from regular metrics, not from traces. Kept traces come from the backend or from Collector metrics that count spans received and exported. Orphan counts are sometimes shown by the backend. If not, sample some traces by hand each week. Drops come from the SDK’s diagnostic output where available, and from Collector metrics about refused data.

Alerting suggestions

Alert if the observed share drifts far from the ratio for a long time, since it suggests a broken ID source, a mixed configuration, or a change in another sampler. Alert if pipeline drops continue for several minutes. Do not alert on small differences in short windows, since randomness alone causes them, as the earlier math showed.

Understanding the common ways a sampling setup fails helps you spot them early.

The quiet failures are the worst

Almost none of these produce an error message. The system continues to run, the dashboards continue to draw, and the only sign is that the data is not what you assumed. This is why the monitoring ideas of the previous chapter matter. A short weekly check of the observed share and a quick look at a few traces catch most of these problems early.

These patterns are the repeated habits of teams that run sampling calmly.

Why the written plan matters

Sampling decisions are easy to make and easy to forget. A new engineer sees a ratio of 0.02 in a configuration file and has no idea why. A one-page plan explains that 0.02 came from a budget of one thousand spans per second at peak, and that error traces are rescued by a Collector rule. This turns a mystery number into a decision that can be reviewed when conditions change.

Each of these looks reasonable at first and causes pain later.

This is the checklist to keep near your configuration files.

Best practices

• Use the parent-based wrapper with TraceIdRatioBased as the root sampler in services that are called by others.
• Choose the ratio from a budget, using the volume formula, and write the reasoning down.
• Use unsampled metrics for counts, rates, and alerts.
• Make sure context passes through every hop, including proxies, queues, and background jobs.
• Keep services on the same SDK family and the same ID generation where possible, and test mixed environments.
• Rescue errors with a tail sampler or a higher ratio on critical routes if rare failures matter.
• Control the sampled flag at the public edge.
• Monitor the observed share and pipeline drops.
• Revisit the ratio when traffic or budget changes.

Common mistakes and their cures

These stories are invented to illustrate common situations. They are not reports about real companies.

When the number of kept traces does not match your ratio, work through the path of a decision from start to finish.

flowchart TB
    START["Kept traces do not match the ratio"] --> A{"Does the sampler description show the expected ratio?"}
    A -- "no" --> A1["Fix the environment variable or code setting and restart"]
    A -- "yes" --> B{"Is the sampler parent-based?"}
    B -- "no" --> B1["Switch to parent-based so roots decide"]
    B -- "yes" --> C{"Do requests arrive with a sampled flag from callers?"}
    C -- "yes" --> C1["Check which callers set it and whether to trust them"]
    C -- "no" --> D{"Are trace IDs random?"}
    D -- "no" --> D1["Fix the ID generator or the source of IDs"]
    D -- "yes" --> E{"Is another sampler or limit downstream?"}
    E -- "yes" --> E1["Account for Collector or backend sampling and drops"]
    E -- "no" --> E2["Compare against unsampled request counts and use more data"]

Explaining the checklist

Start by confirming what the sampler actually is. The description string tells you the ratio the running process uses. If it differs from your intention, the setting did not take effect, or a different setting won. Next, check whether the service is a root or a follower. A follower copies its parent and ignores its own ratio, so its kept share matches the caller’s, not its own.

Then check whether callers send a sampled flag, and whether you should trust them. Then check the trace IDs, since non-random IDs skew results. Finally, check whether another layer, such as a Collector percentage or a backend limit, is removing data after the SDK sampled it. If none of these explains the gap, compare against unsampled request counts, and use a larger window of data, because small samples fluctuate widely.

TraceIdRatioBased sampling keeps a fixed fraction of traces by comparing the trace ID, a random number, with a threshold made from your ratio. Because the decision depends only on the ID and the ratio, it is cheap, needs no shared state, and gives the same answer in every service that uses the same ratio. It is a head sampler, so it decides before the outcome is known and cannot favor errors or slow requests. In practice, you use it inside a parent-based wrapper so that the root decides and everyone else follows, and you complement it with unsampled metrics for honest numbers and, where needed, a tail sampler to rescue the traces that matter most.