Before choosing a sampling strategy, you need to feel the size of the problem. The numbers grow faster than most people expect.

What a trace is, in plain words

When a customer taps “Place Order” in a shopping app, that one tap may touch a dozen services. One service checks the login, another reads the cart, another charges the card, another reserves stock, another sends an email. A trace is the full story of that single request as it travels through all those services. Each step inside the story is called a span. A span records what happened, how long it took, and whether it succeeded or failed.

Traces are powerful because they answer the hardest question in distributed systems: “Why was this one request slow, or why did it fail?” Logs tell you what one service saw. Metrics tell you averages. Only a trace shows the whole journey in one picture.

The arithmetic of volume

Now do the math for a large platform. Suppose it handles one million requests every second, and each request creates thirty spans, and each span is about five hundred bytes when sent over the network. These are illustrative round numbers, but they are realistic for big consumer platforms.

A petabyte is one thousand terabytes. At 1.3 petabytes per day, the storage alone would cost a fortune, the network links between data centers would be crowded with telemetry, and no human could ever search through it quickly. If you keep only one request in a hundred, the same system produces about thirteen terabytes per day. That is still big, but it is the kind of big that a real company can afford.

The hidden costs beyond storage

Storage is the cost people notice first, but it is rarely the only one. Every span has to be created, filled with attributes, converted into a wire format, and sent. That uses CPU and memory inside your application. The data then crosses the network, and in cloud environments, data that crosses between availability zones (separate data center buildings inside one region) is billed per gigabyte. The data then lands in a collector, which uses more CPU and memory. Finally it is indexed in a backend, which uses even more compute, and every search over it costs money as well.

There is also a human cost. When a dashboard shows ten million traces, nobody can find the one that matters. A smaller, well-chosen set is easier to explore and easier to trust.

Sampling is not the same as filtering or aggregation

Three different tools reduce telemetry, and beginners often mix them up. Filtering drops data by a fixed rule, for example never recording the health-check requests that load balancers send every few seconds. Aggregation squeezes many events into a few numbers, such as counting requests per minute. Sampling keeps a subset of complete, detailed events and drops the rest. A mature design uses all three, each where it fits best.

Keep that last sentence in mind. The rest of this guide is really about one question: how do you shrink the data by 99 percent and still keep the traces you will wish you had during the worst night of the year?

A short vocabulary lesson. Once these words are clear, everything else in the guide becomes easy to follow.

Propagation: how services share a story

For many services to contribute to one trace, they must agree on the trace ID. The first service creates it, and every call it makes to another service carries it along inside an HTTP header (a small note attached to a web request). Passing this note from service to service is called context propagation. The standard format for the note is defined by the W3C, the same organization that standardizes the web, and the header is named traceparent.

The sampling decision

A sampling decision is the verdict for a trace: keep or drop. The sampling rate (or probability) is the fraction kept. A rate of 10 percent means one trace in ten is kept on average. Two timing choices exist for when the verdict is made.

• Head sampling decides at the very beginning of the trace, when the first span is created, before anyone knows how the request will turn out.
• Tail sampling decides at the end, after most or all spans have arrived, so the verdict can depend on what actually happened.

Three outcomes inside the SDK

Inside the SDK, a sampler returns one of three answers. Drop means the span is not recorded at all. Record only means the span is recorded locally, so processors can look at it, but it is not marked as sampled and is normally not exported. Record and sample means the span is recorded, marked as sampled, and exported. Most people only ever deal with drop and record and sample, but the middle option exists for special cases such as computing local metrics from spans.

One more subtle point: even when a span is dropped, the SDK still creates a lightweight placeholder with a valid span context. That placeholder lets the trace ID and the “not sampled” verdict continue travelling to downstream services. Dropping is cheap, but it is not absolutely free.

Understanding the mechanics explains why some designs work smoothly and others produce broken, half-empty traces.

The sampled flag in the traceparent header

The traceparent header has four parts separated by dashes: a version number, the trace ID, the ID of the parent span, and a set of trace flags. One bit inside those flags is the sampled flag. When it is on, it tells every downstream service “the head of this trace decided to keep it”. When it is off, it says “this trace was dropped”. A second header, tracestate, carries optional extra key-value pairs that individual vendors or tools can use.

The built-in samplers

OpenTelemetry SDKs ship with a small set of samplers that you combine like building blocks.

The parent-based sampler is the most important one to understand. If a request arrives with a parent that was sampled, the new span is kept. If a request arrives with a parent that was not sampled, the new span is dropped. Only when a request has no parent at all (a brand-new trace) does the root sampler get to make a fresh choice. The default in OpenTelemetry SDKs is parent-based with always-on as the root, which means that, out of the box, nothing is sampled away. You only start dropping data when you configure it, usually through environment variables that name a sampler and an optional ratio.

flowchart LR
    A["Incoming request with traceparent header"] --> B{"Parent span context present?"}
    B -- "No" --> C["Root sampler decides"]
    B -- "Yes, sampled flag on" --> D["Keep: record and export span"]
    B -- "Yes, sampled flag off" --> E["Drop: do not record span"]
    C -- "Keep" --> D
    C -- "Drop" --> E
    D --> F["Child calls inherit the decision and forward the header"]
    E --> F

Read the diagram from left to right. A request arrives. If it already belongs to a trace, the service simply follows the flag. If it is the start of a trace, the root sampler rolls the dice once. Whatever the outcome, the decision is written into the header and forwarded, so every later service repeats the same verdict without any coordination or shared database.

Why the ratio sampler needs care

The ratio-based sampler looks at the trace ID, which is a random number, and keeps the trace if that number falls below a threshold. Because the decision depends only on the trace ID, two services using the same ratio will reach the same answer for the same trace. That is the idea of a consistent decision. The trouble is that early versions of the specification did not describe the exact calculation, so different language SDKs could disagree on edge cases. Newer specification work defines explicit randomness and threshold values that can be carried in the trace state, making decisions comparable across languages and across services that use different rates. Check what your SDK versions support before relying on cross-service consistency.

Head sampling is the simplest, cheapest, and most widely used approach. Its weakness is that it decides while knowing almost nothing.

How it works

When the first span of a trace is created, usually at an edge service such as an API gateway (the front door that receives traffic from outside), the sampler is asked for a verdict. It sees only a small amount of information: the span name, its kind, some initial attributes, and the trace ID. It cannot see whether the request will fail, how long it will take, or whether it belongs to an important customer. After the verdict, it writes the sampled flag into the outgoing header, and every downstream service follows.

Why teams love it

The savings begin at the source. A dropped trace creates no full spans, no attribute data, no network traffic, and no collector work. The sampler is stateless, meaning it remembers nothing between requests, so it needs no memory or coordination and scales perfectly. If you double your servers, you double your sampling capacity without thinking about it. Operationally it is the easiest design to run, debug, and explain.

Why teams outgrow it

The blind spot is outcomes. If you keep one percent of traces at random, you also keep only one percent of the failing ones, and one percent of the slow ones. The very traces you most want are treated exactly like the boring ones.

Scale changes this picture in an interesting way. Suppose a platform handles one million requests per second and one in a thousand fails. That is one thousand failures every second, and a one percent head sample still keeps ten failing traces every second, which is plenty to spot a common bug. The danger is for rare problems. A bug that hits one request in ten million happens once every ten seconds in absolute terms, but a one percent sample catches it once every thousand seconds. And for a smaller, quieter service with perhaps ten requests per second, a one percent rate keeps one trace every ten seconds, which may be too little to learn anything. The same percentage behaves very differently at different traffic levels, so one global rate is almost always a mistake.

Making head sampling a little smarter

Although the head sampler is blind to outcomes, it is not blind to everything. You can write a custom sampler that looks at the attributes available at span creation. For example, it can keep all traces for internal test accounts, keep a higher fraction for a new feature that was just released, and keep a tiny fraction for noisy, low-value endpoints. Some teams also use a remote sampling configuration, where a central service tells each SDK what rate to use per service or per endpoint, so rates can change without redeploying code. This costs very little at runtime and improves usefulness a lot, but it still cannot see the outcome.

Tail sampling trades money and complexity for intelligence. It lets you say “keep every trace that failed” or “keep every trace slower than two seconds”.

How it works

In tail sampling, applications record and send all spans, usually to an OpenTelemetry Collector running a component called the tail sampling processor. The processor groups incoming spans by trace ID and holds them in memory. It starts a timer for each new trace. When the timer ends, called the decision wait, it assumes the trace is complete enough and runs a list of policies over it. If any policy says keep, the whole trace is passed on to the backend. If none do, the trace is discarded.

The kinds of policies

Policies are evaluated together, and a trace is kept if any policy votes to keep it, unless you build combinations that say otherwise. A common recipe is: keep all errors, keep all slow traces, keep all traces for a few special customers, and keep a small random baseline of everything else.

sequenceDiagram
    participant SvcA as Service A
    participant SvcB as Service B
    participant TS as Tail Sampling Collector
    participant BE as Tracing Backend
    SvcA->>TS: Span 1 of trace T1
    SvcB->>TS: Span 2 of trace T1 with error status
    Note over TS: Buffer spans in memory and start the decision wait timer
    SvcB->>TS: Span 3 of trace T1
    Note over TS: Timer ends, evaluate every policy on the full buffered trace
    TS->>BE: Keep trace T1 because the error policy matched
    SvcA->>TS: Late span 4 of trace T1
    TS->>BE: Forward late span using the remembered decision

The diagram shows the life of one trace. Spans from different services arrive at different moments. The collector keeps them together and does nothing until the timer ends. Then it makes the verdict with full knowledge. The last two lines show a real-world nuisance: a span that arrives after the decision. Collectors can remember recent verdicts in a small cache so late spans follow the same fate, but that memory is limited, and very late spans may be treated as new, incomplete traces.

The memory bill

Buffering is the heart of the cost. Consider a stream of 100,000 new traces per second with a decision wait of 30 seconds. At any moment about three million traces are waiting in memory. With thirty spans per trace and five hundred bytes per span on the wire, that is roughly 45 gigabytes of raw data, and the in-memory form of spans is often several times larger than their wire form. This memory must be spread across many collector machines, and the plan must survive restarts and traffic spikes.

One collector can tail sample a small system. A large system needs a fleet, and a fleet creates a new puzzle: how do all spans of one trace end up in the same place?

The routing puzzle

Imagine a trace that touches five services. Each service runs on many servers, and each server sends its spans to whichever collector is nearest or least busy. Without special care, the five spans of one trace land on five different collectors. Each collector sees only a fragment, so none of them can judge whether the trace contained an error or was slow. The decision becomes meaningless.

The answer is a two-layer design. The first layer receives everything and forwards each span to a second layer using the trace ID as the routing key. A rule such as “trace ID modulo number of machines” guarantees that every span with the same trace ID reaches the same second-layer machine. In OpenTelemetry, a component called the load balancing exporter does this job, and it can discover the second-layer machines through a fixed list, DNS records, Kubernetes service discovery, or AWS Cloud Map.

flowchart TB
    subgraph Apps["Application Tier"]
        S1["Service A with SDK"]
        S2["Service B with SDK"]
        S3["Service C with SDK"]
    end
    subgraph Agents["Tier 0: Agent Collectors on each node"]
        AG1["Agent Collector 1"]
        AG2["Agent Collector 2"]
    end
    NLB["Internal Network Load Balancer"]
    subgraph T1["Tier 1: Routing Collectors"]
        R1["Routing Collector A with load balancing exporter"]
        R2["Routing Collector B with load balancing exporter"]
    end
    subgraph T2["Tier 2: Tail Sampling Collectors"]
        TS1["Tail Sampler 1"]
        TS2["Tail Sampler 2"]
        TS3["Tail Sampler 3"]
    end
    BE["Tracing Backend and Storage"]
    MB["Metrics Backend"]
    S1 --> AG1
    S2 --> AG1
    S3 --> AG2
    AG1 --> NLB
    AG2 --> NLB
    NLB --> R1
    NLB --> R2
    R1 -->|"route by trace ID hash"| TS1
    R1 -->|"route by trace ID hash"| TS2
    R2 -->|"route by trace ID hash"| TS2
    R2 -->|"route by trace ID hash"| TS3
    R1 -->|"request, error, duration metrics from all spans"| MB
    R2 -->|"request, error, duration metrics from all spans"| MB
    TS1 --> BE
    TS2 --> BE
    TS3 --> BE

Walk through the figure from top to bottom. Applications send spans to a small agent collector on their own machine, which absorbs bursts and adds helpful labels such as the machine name. Agents forward to a network load balancer that spreads work across the routing collectors. Routing collectors do the trace-ID routing and also create metrics from every span before any dropping happens. Tail sampling collectors hold the buffers, apply policies, and send the winners to the backend.

The trade-offs hidden inside this design

Random sampling sounds easy until two services use different rates, or until someone tries to count requests from a sample.

Consistent probability sampling

Suppose service A samples at 50 percent and service B at 10 percent. If each makes independent random choices, then B will keep roughly 10 percent of the traces that A kept, and the combined result is a trace that exists in A but is missing in B most of the time. A consistent design fixes this by using one shared random number per trace, and each service compares that number against its own threshold. A trace kept at 10 percent is then guaranteed to also be kept by any service sampling at 50 percent. Lower-rate decisions become subsets of higher-rate decisions, so traces never have holes caused by mismatched rates.

OpenTelemetry’s newer probability sampling specification works this way. It defines a randomness value (derived from the trace ID or carried explicitly) and a threshold value, and it records the threshold in the trace state so that anyone reading a span can tell how likely that span was to be kept. The collector’s probabilistic sampler processor also offers modes that implement these ideas. Support differs by language and version, so treat this area as one to verify during upgrades.

Why weights matter

Here is a subtle but critical point. If you keep 1 trace in 100, then each kept trace stands for about 100 real ones. If you later count the kept traces and report “we served 5,000 requests”, you have under-reported by a factor of one hundred. The number you want is the count multiplied by the inverse of the probability, called the adjusted count. When the rate varies, each trace carries its own weight, and totals are computed by adding weights, not by counting rows.

This becomes more important with biased sampling. If you keep all errors but only 1 percent of successes, then your kept data contains far more errors than reality. Anyone who computes an “error rate” from the kept traces will see a number that looks catastrophic. Correct answers need either the stored weights or metrics computed before sampling.

The most common sampling disaster is not a missed trace. It is a dashboard that quietly lies because it was built from sampled data.

The lying dashboard

Many teams build their first charts, such as requests per second, error percentage, and 99th percentile latency (the time that 99 out of 100 requests beat), directly from stored spans. This works fine while every span is kept. The day sampling is switched on, every number changes. Request counts fall by 99 percent, the error rate jumps if errors are favored, and percentile latencies shift because slow traces are over-represented. Alerts that used to be accurate turn noisy or silent overnight.

The fix: measure first, sample second

The robust design is to calculate metrics from the complete stream of spans before any dropping happens. OpenTelemetry provides a component called the span metrics connector, which watches spans as they pass through a collector and produces counters and histograms for request count, error count, and duration, grouped by service, operation, and status. Because it runs in the early, routing layer of the collector fleet, it sees all the data, and its numbers stay exact no matter how aggressively traces are sampled later. Traces then become the place you go to explain a number, and metrics become the place you go to see it.

Cardinality deserves its own warning. Cardinality is the number of different label combinations a metric can have. If you attach a customer ID or a full web address with random numbers in it as a label, each new value creates a new time series, and the metrics backend can run out of memory or send you a shocking bill. Metrics labels should be few and bounded: service, operation name, status class, and perhaps a region.

A fixed percentage is easy to explain but terrible at handling traffic that rises, falls, and spikes. Dynamic strategies aim for a steady amount of data instead of a steady percentage.

Why fixed percentages struggle

Say you fix the rate at 1 percent. On a quiet night, you keep a trickle. On a sales day with ten times the traffic, you suddenly store ten times the data, and the bill and the backend load rise with it. Meanwhile an obscure endpoint called twenty times a day gets no traces at all, because one percent of twenty is nearly nothing. Percentages follow traffic, but budgets and learning needs do not.

Target-rate sampling

The alternative is to say “keep about N traces per second per service or per endpoint”. A busy endpoint gets a tiny probability so it stays near N. A quiet endpoint gets a probability near 100 percent so it also reaches N or as close as possible. This is called rate-limited or adaptive sampling. It shares the budget fairly across endpoints, and it guarantees that quiet but important paths always have some traces.

flowchart LR
    A["Collectors count spans per service and operation"] --> B["Strategy calculator compares counts with the target rate"]
    B --> C["New sampling probabilities are published"]
    C --> D["SDKs poll for the strategy and apply new probabilities"]
    D --> A

The loop works like a thermostat. Sensors count what is being kept, a calculator compares it with the goal, new settings go out, and the next round of measurements shows whether the room has warmed up. The Jaeger tracing project, which originated at Uber, popularized this pattern: its SDKs can poll a central strategy endpoint, and its adaptive mode calculates per-service, per-operation probabilities from observed throughput. OpenTelemetry SDKs can use a remote sampler in this style, and AWS X-Ray has its own version, where a rule gives a reservoir of one trace per second plus a fixed percentage of further requests (five percent by default). Amazon’s distribution of OpenTelemetry can talk to this X-Ray rule system.

The price of dynamic behavior

Rate limiting as a safety net inside tail sampling

Dynamic thinking also matters inside tail sampling. A policy like “keep every error” is wonderful on a normal day and dangerous in an outage, because when a database fails, ninety percent of traces become errors, and the “keep all errors” rule floods the backend at exactly the moment you need it to stay healthy. The remedy is to give the error policy a budget, for example by combining it with a rate limit through a composite policy, so errors get priority but cannot exceed a ceiling. A partial view of a disaster that you can actually query beats a complete view that crashes your backend.

Every sampling design is a bundle of compromises. This chapter lays them out side by side so you can see which dials you are really turning.

The eleven dials

Notice that dials interact. Choosing tail sampling (dial 2) forces you into stateful collectors (dial 3), makes metrics-before-sampling much more attractive (dial 6), and adds delay (dial 7). Choosing adaptive rates (dial 8) makes weights unavoidable (dial 5). Good design is not picking the best end of each dial, because there is no best end. It is choosing a coherent set of positions that fit your traffic, team, and budget.

The right answer for ten requests per second is the wrong answer for a million. Match the design to the traffic, not to the trend.

flowchart TD
    Q1{"Must every error and slow trace be kept?"}
    Q1 -- "No" --> H["Parent based head sampling with a probability root sampler"]
    Q1 -- "Yes" --> Q2{"Can your team run stateful collector fleets?"}
    Q2 -- "No" --> V["Use a managed tail sampling service or biased head sampling"]
    Q2 -- "Yes" --> T["Two tier collectors with tail sampling policies"]
    H --> M["Always create metrics before sampling"]
    V --> M
    T --> M

A hybrid worth knowing

Large organizations often use a hybrid. The SDK applies a parent-based sampler with an “always on” root in most services, so nothing is lost at the source, while a very noisy service, such as a high-volume cache lookup, gets its own low head rate. All traffic flows to the collector fleet, where tail sampling decides. The advantage is simplicity in applications. The disadvantage is that you pay to move and buffer all that data.

A second hybrid reverses the order: a head sampler cuts volume by a large factor first, for example to ten percent, and tail sampling then picks the best of what remains. This is cheaper, but you must accept a hard limit: ninety percent of errors were thrown away before the smart stage ever saw them. Tail sampling can only choose among traces that survived the head. Teams that choose this path should say out loud that they are accepting partial error coverage, and should lean on metrics for exact error counts.

Sampling systems fail in surprising, quiet ways. Most failures do not crash anything. They just make your data wrong. Knowing the common ones lets you design around them.

Scenario 1: The incident that floods the backend

A database fails, and nearly every trace now contains an error. The “keep all errors” policy keeps nearly everything, and the backend receives ten or fifty times its normal load. Queries slow down, ingest queues back up, and engineers cannot even open the traces they need. Recovery: prevent it by giving error policies a cap, and by having a “break glass” configuration that shifts to a strict rate limit in emergencies. After the incident, review whether the budget was right.

Scenario 2: Late spans and long traces

Some work does not finish quickly. A request that puts a message on a queue may be processed minutes later by another service. A batch job may produce spans for an hour. If the decision wait is thirty seconds, then the tail sampler judges the trace before the later spans arrive. The result is a trace that looks short and healthy, gets dropped, and takes the important late error with it. Recovery: use span links (a way to connect related but separate traces) so queue-based work forms its own trace, set the decision wait with typical durations in mind, and consider a longer wait only for pipelines that need it, since a longer wait costs memory.

Scenario 3: A tail sampler dies

The process crashes or is restarted during a deployment. Everything it was holding in memory disappears, including the traces it would have kept. Recovery: accept that tail sampling is best effort. Use rolling deployments that replace one machine at a time, and schedule restarts at quiet hours. Keep metrics outside the sampler so numbers survive. For the highest-value traces, such as payment failures, consider an extra copy of that narrow slice that bypasses the buffer.

Scenario 4: Scaling events split traces

When the sampling fleet grows during a traffic peak, the routing math changes. Spans of traces already in flight begin going to a new machine while their earlier spans sit on the old one. Neither side sees the full trace, and both may drop it. Recovery: prefer scaling in small steps, use stable routing (consistent hashing, which moves only a small slice of keys when the machine list changes), and pre-scale before known peaks instead of reacting during them.

Scenario 5: Back pressure moves upstream

A slow backend makes the tail sampling layer slower. Its queues fill, and then the routing layer’s queues fill, and finally the agents start dropping or blocking. If applications wait for the collector, a telemetry problem can slow down real customer requests. Recovery: make telemetry paths non-blocking, use bounded queues with drop-oldest or drop-newest rules, and always rank customer traffic above telemetry in any resource contention.

Scenario 6: The silent drift

Someone changes a policy, a ratio, or a service name, and the share of kept traces shifts without anyone noticing. A month later, a chart shows something strange, and no one can explain it. Recovery: keep sampling configuration in version control, review changes like code, and monitor the effective sampling rate as a first-class metric (the next chapters explain how).

Sampling is usually discussed as a cost topic, but it touches security in three real ways: who can influence your decisions, what the sampler can see, and what must never be dropped.

Untrusted callers can set the sampled flag

The traceparent header comes from the caller. If your public service obeys it blindly, an outsider can send millions of requests with the sampled flag turned on and force you to record and pay for every one. This is both a cost attack and a pressure attack on your telemetry pipeline. Defense: at the edge, ignore incoming sampling flags from untrusted sources and make a fresh decision with your own root sampler, while continuing to honor flags from your own internal services. Many gateways and proxies can strip or overwrite these headers.

The sampler can see sensitive data

Tail sampling policies read span attributes, such as customer tier or an error message. Those attributes may hold personal data. The collector fleet is therefore a place where sensitive information is held in memory, and it should be protected like any other production data store: encrypted connections between tiers, restricted access to machines, and careful logging that never prints span contents. Where possible, remove or hash sensitive attributes in an earlier stage, before they spread into buffers, but be careful to do this after any policy that needs them.

Some data must never be sampled

Sampling is a statistical tool. Audit trails, financial records, and security events often have legal requirements that say “keep all of it”. Using a sampled tracing pipeline for those records is a mistake, because a dropped trace is a lost record. Defense: keep compliance data in a separate pipeline with its own retention rules, and treat tracing as a debugging aid, not as the system of record.

Fairness and bias

If rates are tuned by service or by customer, a heavy user or a loud service may consume the budget and starve the others. Per-tenant caps help here. Likewise, if policies keep “interesting” traces, remember that interesting is a judgment, and any judgment can reflect blind spots. A small uniform baseline, kept alongside the clever policies, acts as a check against your own assumptions.

A sampler that no one watches is a sampler that is slowly going wrong. Treat the sampling pipeline as a production service with its own dashboards and alerts.

What to watch

The OpenTelemetry Collector exposes its own internal metrics, and the tail sampling processor adds counters for traces sampled and not sampled, decision latency, traces dropped too early, and late spans. Exact metric names depend on the version, so confirm them in the documentation for the release you run.

Which alerts are worth paging for

Page a human only for conditions that threaten data or customers: collectors refusing data, buffers at their limit, or the effective rate collapsing to zero. Send quieter notifications for drift conditions: the effective rate changing by more than a set percentage, error-class traces suddenly exceeding their budget, or spans from a service disappearing. The goal is to learn about a bad sampler from an alert and not from an engineer who cannot find a trace in the middle of an outage.

Experienced teams keep arriving at the same good habits and the same expensive mistakes. Learn from their scars.

Patterns that work

Anti-patterns to avoid

The same trade-offs show up in famous systems. Seeing them in the wild makes the abstract ideas stick.

Traces rarely live alone. Teams jump between charts, logs, and traces, and sampling can break those jumps if you are not careful.

Dangling links

Many systems write the trace ID into every log line, so an engineer can click from a log message to the matching trace. When sampling drops the trace, the log line still carries an ID that leads nowhere. Engineers see “trace not found” and assume tracing is broken. This is not a bug in the tool, but it is a real experience cost of sampling.

There are three ways to reduce the pain. First, include the sampled flag in the log line, so the viewer can tell “this trace was dropped on purpose” from “something is broken”. Second, let log pipelines keep logs for sampled traces longer than logs for dropped traces. Third, if logs are tail sampled too, use the same decision for both, so the log and the trace are kept or dropped together.

Exemplars: pointing from numbers to examples

An exemplar is a sample trace reference attached to a metric data point. A chart of slow requests per minute can include a link to one real slow trace from that minute. For this to work, the linked trace must have been kept. A good design therefore makes sure that the traces chosen as exemplars are sampled ones, since linking a number to a dropped trace creates the same dangling-link problem.

Sampling and other signals

Because metrics are exact and traces are sampled, a healthy workflow goes in one direction: notice a problem in a metric, narrow it down by service and operation, and then open a kept trace to see why. If your sampling keeps errors and slow requests, there will usually be a matching trace waiting. If it does not, the right reaction is to adjust the sampling policy, not to blame the engineer.

Switching on sampling in one big step is risky. A phased plan keeps surprises small and gives you evidence at each stage.

Words about trade-offs are useful, but numbers make them real. This chapter compares four designs for the same imaginary platform, so you can see how each dial changes the bill.

The imaginary platform

Suppose a company serves 200,000 requests per second. Each request creates 20 spans, and each span is 500 bytes on the wire, so one trace is about 10 kilobytes. That means 2 gigabytes of span data leave the applications every second, or about 172.8 terabytes per day. Assume that 0.2 percent of requests fail (400 failures per second), that 1 percent are slow enough to matter, and that the team wants a 1 percent random baseline of ordinary traffic. These figures are made up for teaching, but the method works with your own numbers.

How the numbers were built

For design C, the tail sampler keeps all errors (0.2 percent), all slow traces (1 percent), and a random baseline (1 percent), which adds up to roughly 2.2 percent of 172.8 terabytes, or about 3.8 terabytes. Memory comes from the trace arrival rate times the decision wait: 200,000 traces per second times 30 seconds is six million traces held at once, and six million times 10 kilobytes is 60 gigabytes of raw data. Because the in-memory form is often three times larger or more, plan for hundreds of gigabytes across the fleet. For design D, only one trace in five enters the system, so every figure is one fifth as large, and the stored amount is 2.2 percent of 34.6 terabytes, close to 0.8 terabytes.

What the table does not show

Data transfer is the hidden line. In design C, 2 gigabytes per second cross the network at least once, and often twice or three times, because spans move from apps to agents, from agents to routing collectors, and from routing collectors to sampling collectors. If a hop crosses availability zones and a cloud provider charged one cent per gigabyte for it (check current prices, because they differ and change), that single hop would cost about 1,728 dollars per day, or roughly 630,000 dollars per year. Placing collector tiers in the same zone as the traffic they serve, and compressing telemetry on the wire, can remove a large share of that bill.

Reading the lesson

No design wins every column. Design B has the smallest bill and the weakest error coverage. Design C has the strongest coverage and the heaviest machinery. Design D sits between them and is often chosen by teams who accept losing some errors in exchange for a much smaller fleet, as long as exact error counts come from metrics created before sampling. The right choice depends on what a missed trace costs you. If one missed payment failure costs more than the fleet, choose C. If tracing is a convenience and metrics carry the alerting, B or D is likely enough.

Real platforms are messier than diagrams. Here are the situations that break simple designs and how teams handle them.

Many regions, many clusters

A request may start in one region and call a service in another. If each region runs its own tail sampling fleet, then the spans of that request are split between two fleets, and neither sees a complete trace. There are three common answers. The first is to accept regional traces: each region judges what it sees, and cross-region calls become linked but separate traces. The second is to route by trace ID to a single global fleet, which gives whole traces but pays for cross-region data transfer and adds delay. The third is to make the sampling decision at the head (where the request enters) and carry it in the flag, so each region obeys it without needing to see the whole trace. Many companies choose the third for cross-region traffic and the first for everything else.

flowchart LR
    U["User request"] --> E["Edge Gateway in Region 1 makes the root decision"]
    E --> S1["Services in Region 1"]
    S1 -->|"traceparent with sampled flag"| S2["Services in Region 2"]
    S1 --> F1["Tail Sampling Fleet in Region 1"]
    S2 --> F2["Tail Sampling Fleet in Region 2"]
    F1 --> B["Tracing Backend"]
    F2 --> B

In the figure, the gateway in region one makes the first choice. Region two receives the flag and follows it. Each fleet then applies its own policies to what it sees. The backend can still join the pieces by trace ID when someone searches, even though no single fleet held the whole trace. The price is that outcome-based rules in region two cannot see what happened in region one.

Long-running and asynchronous work

Queues, schedulers, and batch jobs stretch a trace across minutes or hours. Waiting that long in a tail sampler is unaffordable, so teams usually treat each unit of asynchronous work as its own trace and connect it to the original request with a span link. The sampling decision for the new trace is then made when the work begins. A useful trick is to copy a “this origin was important” attribute onto the linked trace, so a policy can keep the follow-up work whenever the original request was interesting.

Short-lived processes and serverless functions

Functions that live for a few hundred milliseconds may end before their telemetry is sent. Spans that were never flushed are lost, whatever the sampler decided. Head sampling helps here, because it reduces how much needs to be flushed, and a local collector extension or agent next to the function can accept data quickly and send it onward. Whatever you choose, test that a sampled span actually reaches the backend after the function freezes.

Mixed propagation formats

Older systems may use other header formats, such as the B3 headers that came from Zipkin. Each format carries the sampling decision a little differently, and converting between them at gateways can lose or misread flags. If your company has a mixture, test a trace that crosses every boundary and confirm that a “dropped” verdict stays dropped and a “kept” verdict stays kept. Standardizing on the W3C format reduces this risk over time.

Multi-tenant platforms

If one telemetry pipeline serves many teams or customers, a single noisy tenant can fill the buffers or the backend budget. Give each tenant its own quota and its own tail sampling policies, and report each tenant’s effective rate to them, so they understand why some traces are missing. Fairness is not only a technical choice here. It is also a service promise that should be written down.

Service meshes and sidecars

In a service mesh, sidecar proxies create spans for every network hop, sometimes doubling the span count. Their sampling rate is configured separately from the application SDKs. If the mesh samples at 1 percent and the application at 100 percent of parents, the result depends on who starts the trace. Decide which layer is the owner of the root decision, and configure the other layer to follow it.

A sampling design is not finished when it is deployed. It needs owners, rules for changing it, and a habit of checking that it still serves the people who depend on it.

Who owns the decision

Sampling sits between teams. Application owners care about seeing their traces, the platform team cares about cost and stability, and finance cares about the bill. Without a clear owner, rates drift upward after every incident (“keep more, just in case”) and downward after every invoice. Name a small group that owns the sampling policy, publishes it, and accepts requests for exceptions with a reason and an end date.

Questions to ask before every change

Testing a change safely

The safest way to test a new rule is to run it on a small slice first. Apply the new policy to one service or one percent of traffic, compare the kept volume with your forecast, and watch the sampler’s own metrics for surprises such as growing buffers or rising decision delay. Where possible, run the new policy in a side pipeline that only counts what it would have kept, without exporting anything, and compare that count with the real pipeline. If the counts match your expectations, promote the rule to everyone.

Regular review

Set a calendar reminder to review the sampling system every quarter. Look at which policies never match anything, which services are always at their budget, which traces engineers actually opened, and which complaints of “I could not find the trace” were real. Retire rules that no longer help, and add rules where gaps appeared. Sampling that is never reviewed becomes a collection of old guesses.

Documentation people can find

Write a short public page that answers four questions for any engineer: what is kept, what is dropped, how to ask for an exception, and how to force-keep a trace while debugging. A common technique is a special request header or attribute that a policy recognizes and always keeps, with a rate limit and an access check so it cannot be abused. That small feature saves many hours of frustration and many escalations.

Myths that cause bad decisions

Myth: “Sampling at one percent means we see one percent of the problems.” It means you see one percent of the traces, which is a different thing. A problem that affects thousands of requests every second will appear in many sampled traces. A problem that affects one request a day may never appear. Think in absolute counts of affected requests, not in percentages alone.

Myth: “Tail sampling is always better than head sampling.” Tail sampling is better at catching specific outcomes, but it costs memory, machines, network traffic, and delay, and it does nothing for application overhead. For many systems, a well-tuned head sampler plus accurate metrics gives most of the value at a small fraction of the cost.

Myth: “If a trace is dropped, the request was not recorded anywhere.” Logs and metrics still describe that request. Sampling removes only the detailed trace. This is exactly why metrics created before sampling matter, and why logs should say whether their trace was kept.

Myth: “We can fix a bad sampling rate after the incident.” Sampling decisions cannot be replayed. A trace that was dropped last night is gone. If you are about to launch something risky, raise the rates before the launch, not after the first failure, and lower them again when the risk has passed.

Myth: “The sampler is just a setting, so anyone can change it.” Every change shifts the cost, the visibility, and the trustworthiness of statistics. Treat it as production configuration with review, a rollback plan, and a clear owner.

Myth: “A random sample is always fair.” Random sampling is fair across requests, but not across users, endpoints, or teams. A heavy endpoint generates most of the sampled traces, and a rare endpoint generates almost none. Fairness usually needs per-endpoint or per-tenant budgets on top of randomness.

A short runbook for a bad day

When the sampling system itself misbehaves, a calm checklist beats improvisation. First, check whether customer traffic is affected. If telemetry is slowing real requests, shed telemetry load immediately by lowering head rates or disabling the heaviest exporters, because customers always come first. Second, look at the effective sampling rate and the buffer fullness charts to see whether the problem is too much input, too little capacity, or a stuck backend.

Third, decide what to protect. During a large incident, the most valuable data is usually a modest set of error traces from the affected services, plus accurate metrics. Switch to the emergency configuration that caps every keep-all policy, and confirm that the span metrics are still flowing. Fourth, communicate: tell the incident channel that some traces are being dropped on purpose and how to find the ones that are kept. Engineers who know the rules will stop wasting time searching for missing traces.

Finally, after the incident, write down what the sampling system did, what it dropped, and what you wish it had kept. Turn each wish into a concrete policy change with an owner. Over many incidents, this habit is what slowly turns a generic sampling setup into one that fits your company’s real failures.

Sampling in OpenTelemetry is the art of keeping the traces you will need while paying for only a fraction of the data. There is no free option. Head sampling gives low cost and simplicity but cannot see outcomes. Tail sampling gives outcome-aware choices but needs memory, routing, and operational discipline. Consistency keeps traces whole, weights keep statistics honest, and metrics created before sampling keep dashboards truthful. The best systems combine these pieces deliberately, watch them closely, and revisit the choices as traffic grows.