Before we adapt anything, we need to understand the problem that sampling solves: too much telemetry, too little budget.

The firehose problem

A modern application is rarely one program. It is often dozens or hundreds of small services that call each other. One click on a shopping site may touch a web gateway, a login service, a cart service, a pricing service, a stock service, a payment service, and several databases. Each step can produce a record describing what happened. When thousands of users click at the same time, the number of records becomes enormous.

A trace is the story of one request as it travels through all of those services. A span is one chapter of that story, for example “the cart service talked to its database for 12 milliseconds”. A single trace may hold five spans or five hundred. A busy system can produce hundreds of thousands of spans every second.

Three kinds of cost

Keeping every trace is expensive in three separate ways, and it helps to name them because different sampling strategies fight different costs.

Most traces are boring

Here is the key insight that makes sampling worthwhile. In a healthy system, about 99 out of every 100 requests look almost the same: fast, successful, and uninteresting. The valuable traces are the slow ones, the failed ones, and the unusual ones. If you could keep all the valuable traces and only a small taste of the boring ones, you would lose very little knowledge and save a huge amount of money.

Why a fixed rule is not enough

The simplest sampling rule is “keep 1 percent”. It is easy, but traffic is never steady. At 3 a.m. a service may handle 50 requests a minute, and 1 percent means you keep a trace every two minutes, which is too thin to see anything. During a sale the same service may handle 500,000 requests a minute, and 1 percent still floods you with 5,000 traces a minute. A rule that is right at one moment is wrong at another. Adaptive sampling exists because the world moves and the rule must move with it.

A short, friendly refresher on the pieces of OpenTelemetry that sampling touches.

What OpenTelemetry is

OpenTelemetry, often shortened to OTel, is an open standard and a set of tools for producing and moving telemetry. It is vendor neutral, which means you can create your data once and send it to many different backends. It is maintained in the open under the Cloud Native Computing Foundation, and it is one of the most active projects in that foundation.

The building blocks

Traces, metrics, and logs side by side

OpenTelemetry handles three kinds of telemetry, and sampling affects them very differently. Metrics are numbers measured over time, such as requests per second. They are small, they can be combined cheaply, and most teams keep them complete. Logs are lines of text describing events. They can be large, and teams often filter them by severity. Traces are the detailed stories of individual requests. They are the largest and the most expensive per request, which is why sampling is mostly discussed in the context of traces.

The three kinds can be linked together. A log line can carry the trace identifier of the request that produced it, and a metric can point to an example trace. This linking is the reason sampling needs care: if a log points to a trace that the sampler dropped, the link leads nowhere. A good design keeps the linked traces for the events that matter most.

Trace identity and the sampled flag

Every trace has a unique trace identifier. When service A calls service B, it passes that identifier along, together with the identifier of the calling span and a small set of flags. One of those flags says whether the trace is being sampled. This follows the W3C Trace Context standard. There is also an optional extra field called trace state, where vendors and the OpenTelemetry sampling specification can pass small notes along the chain.

The Collector pipeline in one picture

The Collector is where much of the more advanced sampling lives, so it is worth seeing its shape. Data enters through receivers, passes through a chain of processors, and leaves through exporters. Sampling processors sit in the middle of that chain.

flowchart LR
  A["Applications with OTel SDK"] --> B["Receiver (OTLP)"]
  B --> C["Processors (batch, filter, sampling)"]
  C --> D["Exporter (OTLP)"]
  D --> E["Tracing backend"]
        

Every adaptive design is built on top of two older ideas. You must understand both to see why adaptation is hard.

Head sampling: decide at the start

In head sampling, the decision is made at the very beginning of the trace, when the first span, called the root span, is created. The sampler looks only at the information available at that moment, such as the trace identifier, the name of the operation, and a few starting attributes. Once it says yes or no, that decision is written into the trace context and followed by every other service.

The big advantage is cost. If the answer is no, the application barely does any work for that trace: no spans are recorded, nothing is sent, nothing is stored. The big weakness is blindness. At the start of a request, nobody knows whether it will become slow or fail. A head sampler cannot say “keep the slow ones” because slowness has not happened yet.

Tail sampling: decide at the end

In tail sampling, every span is recorded and sent to a Collector. The Collector gathers the spans of each trace, waits until the trace looks complete, and then decides. Now it can look at the whole story: how long it took, whether any span had an error, how many spans there were, and which services were involved. It can say “keep all errors, keep everything slower than two seconds, keep a small taste of the rest”.

The strength is smart choice. The weakness is cost and complexity. All spans must travel to the Collector before any are dropped. The Collector must hold spans in memory while it waits. And every span of a trace must arrive at the same Collector instance, or the decision will be made on an incomplete picture.

The simple arithmetic of probability sampling

Probability sampling is easy to picture. Choose a number between zero and one, called the sampling probability. A probability of 0.1 means “keep about one trace in ten”. The sampler turns each trace identifier into a number, compares it with the probability, and keeps the trace if the number falls under the line. Because the trace identifier is effectively random, about the right share of traces passes the test, and because the identifier never changes, the same trace always gets the same answer wherever it is evaluated.

This is why the trace identifier matters so much. If every service derives its decision from the same identifier, they all agree without needing to talk to each other. The decision becomes a pure calculation, which is fast, cheap, and repeatable. It also means you can predict your costs: if you receive a million traces an hour and keep one percent, you will store about ten thousand.

The weakness of a fixed probability is exactly what we said earlier. It follows traffic. The number you pick on a quiet Tuesday is wrong on a busy Friday, and nobody wants to edit it by hand every hour.

Fixed versus dynamic

Both head and tail sampling can be fixed or dynamic. A fixed head sampler might always keep 10 percent. A fixed tail sampler might always keep errors plus 5 percent of everything else. A dynamic version of either one changes its numbers over time based on what it observes. That second word, dynamic, is the subject of the rest of this guide.

The words are used loosely across the industry, so let us pin them down.

A working definition

Dynamic sampling, also called adaptive sampling, is any sampling approach in which the keep-or-drop probability is not a single constant but is continuously recalculated from live observations of the traffic. The system watches how many traces are flowing, what they look like, and how much data it is allowed to keep, and then it adjusts the sampling rate to hit a goal.

The goals an adaptive sampler can chase

The three ingredients of every adaptive sampler

No matter which product or design you meet, adaptive sampling always has three parts working together.

How adaptive sampling differs from nearby ideas

Several neighbouring techniques are often confused with adaptive sampling. Separating them makes later chapters easier to follow.

In real systems you will usually combine several of these. A health-check filter removes pure noise, a priority rule protects errors, a rate cap guards against disasters, and an adaptive controller handles everything in between.

A tour of the real building blocks, from SDK samplers to Collector processors to remote control channels.

SDK samplers

Every OpenTelemetry SDK includes a small family of built-in samplers. They are the head sampling tools. Two are trivial: one that always keeps and one that never keeps. A third keeps a fixed fraction of traces by looking at the trace identifier, so the same trace identifier always produces the same answer. A fourth, called the parent-based sampler, is a wrapper that says “if my parent already decided, follow my parent; otherwise ask this other sampler”. Parent-based behaviour is what keeps traces whole across services, so almost every production setup uses it.

Probability samplers and the trace state

A subtle problem appears when you sample in more than one place. If service A keeps 50 percent and service B keeps 50 percent independently, you do not get 50 percent of whole traces. You get a messy mix of broken ones. To fix this, the OpenTelemetry sampling specification describes consistent probability sampling. Each trace carries a random value and each sampler carries a threshold. A trace is kept when its random value passes the threshold. Because every service uses the same random value, a lower rate is always a subset of a higher rate. A service that keeps 10 percent keeps traces that every 50 percent service would also have kept.

Just as important, the threshold that was used is written into the trace state. That note tells later stages and the backend exactly how likely it was that this trace was kept, so they can count correctly later. We return to this in the chapter on honest math.

Consistent probability sampling in more depth

Consistent probability sampling is one of the quieter but more important ideas in the OpenTelemetry sampling work, so it deserves a slower look. The goal is simple: when many stages each make their own keep-or-drop choice, the combined result should still be easy to reason about.

The trick is to separate two numbers. One is a randomness value that belongs to the trace. It is either carried along explicitly or taken from the random part of the trace identifier, depending on the version of the propagation rules in use. The other is a threshold that belongs to the sampler. A trace is kept when its randomness value is above the sampler’s threshold. A higher keep rate means a lower threshold, and a lower keep rate means a higher threshold.

Because the randomness value is shared by everybody, the kept sets are nested like Russian dolls. Every trace kept at 1 percent is also kept at 10 percent, and every trace kept at 10 percent is also kept at 50 percent. This is the property that lets you chain a coarse early stage with a finer later stage without shredding traces.

Finally, when a stage decides to keep a trace, it writes the threshold it used into the trace state. If a later stage uses an even higher threshold, it updates that note. At the end of the chain, the backend can read the final threshold and know the exact probability of survival, and therefore the exact weight each trace deserves. That note is what turns sampled traces from a loose pile of examples into a statistically meaningful sample.

Collector processors for sampling

Which of these are “dynamic”?

The tail sampling processor has a rate limiting policy and a composite policy that can divide a total budget among several sub-policies. Those give you budget-style adaptation inside the Collector. The remote sampling extension, which follows the model made popular by the Jaeger tracing system, lets a central place change what SDKs do. Fully automatic per-key rate adjustment, where an algorithm continuously retunes probabilities from live traffic, is typically supplied by a tracing backend, by a dedicated sampling proxy, or by custom code built around the processors above. Features and names change between releases, so always confirm details against the current OpenTelemetry documentation before you design around them.

Remote configuration channels

Changing rates at runtime needs a way to talk to running software. Two channels matter. One is the remote sampling approach where SDKs poll a central endpoint for their strategy at regular intervals. The other is OpAMP, the Open Agent Management Protocol, a standard being developed in the OpenTelemetry community for managing fleets of agents and Collectors, including pushing new configuration. Both let a control brain steer many data-plane workers.

flowchart TD
  C["Control brain (rate calculator)"] -->|"new strategies"| R["Remote sampling endpoint"]
  C -->|"new config"| O["OpAMP server"]
  R -->|"SDKs poll"| S1["Service A SDK"]
  R -->|"SDKs poll"| S2["Service B SDK"]
  O -->|"push or fetch"| G["Gateway Collectors"]
  S1 --> T["Telemetry stream"]
  S2 --> T
  G --> T
  T -->|"observed volume"| C
        

Now we open the engine and look at the moving parts, in words and pictures rather than code.

Step one: choose a key

An adaptive sampler rarely works on all traffic as one blob. It splits traffic into groups using a key, for example “service name plus operation name plus status”. Each key gets its own counter and its own probability. This is what allows a busy “view product” endpoint to be sampled hard while a rare “refund order” endpoint is sampled lightly, or even kept in full.

Step two: count in a time window

The sampler counts how many traces each key produced during a recent time window, for example the last 30 seconds. Counting is done with a smoothed average, often called an exponential moving average, which gives recent seconds more weight than old ones. This way a sudden spike is noticed quickly, but a single odd second does not make the system overreact.

Step three: compute the rate

Suppose the goal is to keep about 100 traces per second across the whole service. If the sampler estimates that 10,000 traces per second are arriving, the overall keep probability must be about 1 percent. If traffic falls to 500 per second, the probability rises to 20 percent. The sampler recalculates this every few seconds.

Step four: share the budget fairly

If you simply gave every key the same percentage, the busiest key would still fill most of the budget. A fairer idea is to give each key a share that grows more slowly than its traffic. A common approach scales the number of kept traces with the square root, or with the logarithm, of each key’s volume. The effect is that a key with a hundred times more traffic gets only about ten times more samples, so rare keys keep a meaningful share.

Step five: stamp the probability

Every kept trace must carry a note saying how likely it was to be kept. If a trace was kept at a 1 percent rate, that single trace stands in for about one hundred similar traces. This number is often called the adjusted count, and it is simply one divided by the probability. Without this stamp, any chart you draw from sampled data will lie to you.

Common algorithm families

Behind the friendly word “adaptive” there are several quite different ways to calculate the next probability. Knowing the families helps you read vendor documentation and decide what you actually need.

These families can be mixed. A common recipe uses a priority layer for errors, a moving-average layer for fairness, and a budget-pacing layer as a safety net. The important point is that every family follows the same loop you just saw: observe, decide, apply, and record.

Cold start and warm-up

A freshly started sampler knows nothing. It has no history of traffic, so its first estimates are poor. If it guesses too low, it may drop traces it should have kept. If it guesses too high, it may flood the pipeline for a few seconds. Good designs handle this “cold start” on purpose. They begin with a safe default probability, they let the moving average fill up before trusting it, and they keep the first few minutes of rates conservative. After a restart, the same care applies, because every restart wipes the memory of the controller unless its state is stored somewhere.

The feedback loop

flowchart LR
  A["Incoming traces"] --> B["Count per key in time window"]
  B --> C["Smooth with moving average"]
  C --> D["Compare with goal"]
  D --> E["Compute new probability per key"]
  E --> F["Keep or drop new traces"]
  F --> G["Stamp probability on kept traces"]
  F --> B
        

The quality of an adaptive sampler depends less on its maths and more on the signals and keys you feed it.

Signals that describe a trace

An adaptive sampler can only react to things it can see. In head sampling, that means information known at the start. In tail sampling, it also includes the outcome. Choosing the right signals is a design decision, and different signals protect different goals.

The danger of too many keys

Every key costs memory because the sampler must remember counters for it. If you build keys from values that almost never repeat, such as user identifiers, request identifiers, or full web addresses with query strings, the number of keys explodes. This problem is called high cardinality. It wastes memory, slows the sampler, and makes every individual key so rare that the algorithm cannot learn a stable rate.

A worked example of key design

Let us design keys for an imaginary online bookshop with six services. The team first tries the key “service plus full URL”. Within a day the sampler tracks tens of thousands of keys, because every book page and every order has its own address. Memory climbs, and most keys have seen only one request, so no rate can be learned.

The team then moves to “service plus route template plus status class”. Now a book page becomes one route template, an order lookup becomes another, and status class groups results into success, client error, and server error. The number of keys falls to a few hundred, each with plenty of traffic to learn from.

The lesson is that key design is a negotiation between precision and stability. Add detail only when you can show that a missing detail hides an important group, and remove detail when you see keys that rarely receive traffic.

Combining a base rule with exceptions

Mature systems rarely use one sampler alone. They layer rules in priority order. First comes a “must keep” layer for errors, very slow traces, and flagged debugging sessions. Next comes a fairness layer that gives rare keys a guaranteed minimum. Last comes the adaptive layer that fills the remaining budget from everything else. Thinking in layers keeps each rule simple and easy to explain to the team.

There is no single correct place to put an adaptive sampler. Here are the three shapes you will meet most often.

Pattern A: SDK head sampling with central control

Each application’s SDK runs a head sampler. A central service decides the rates and publishes them as strategies. The SDKs poll for updates every minute or so and switch to the new strategy. Because the decision happens at the start, dropped traces cost almost nothing. The limitation is the same as any head design: the sampler cannot see errors or latency that have not happened yet.

flowchart LR
  subgraph Apps["Applications"]
    A1["Service A SDK sampler"]
    A2["Service B SDK sampler"]
  end
  CTRL["Central strategy service"] -->|"poll every minute"| A1
  CTRL -->|"poll every minute"| A2
  A1 -->|"kept traces only"| COL["Collector"]
  A2 -->|"kept traces only"| COL
  COL --> BE["Backend"]
  BE -->|"volume statistics"| CTRL
        

Pattern B: Two-tier Collector with tail sampling

Applications send everything to a first layer of Collectors, often running next to the applications as agents. That layer forwards spans to a second layer, the gateway, using a routing step that sends every span of the same trace to the same gateway instance. The gateway holds traces briefly, applies tail policies, and exports the survivors. The routing step is the secret that makes tail sampling work when you have more than one gateway.

flowchart LR
  S["Services"] --> AG["Agent Collectors"]
  AG --> LB["Load balancing by trace ID"]
  LB --> GW1["Gateway 1 with tail sampling"]
  LB --> GW2["Gateway 2 with tail sampling"]
  LB --> GW3["Gateway 3 with tail sampling"]
  GW1 --> BE["Backend"]
  GW2 --> BE
  GW3 --> BE
        

Pattern C: Hybrid, a coarse head filter then adaptive tail

Pure tail sampling sends everything over the network, which can be costly. A hybrid design applies a gentle head sampler first, perhaps keeping a quarter of traces, to cut the firehose. It then uses tail sampling on what remains, with adaptive rates to meet the budget and always-keep rules for errors. The catch is that the head stage may already have dropped some interesting traces, so teams often exempt known-critical services from the head filter.

Where should the sampler physically run?

Besides choosing a pattern, you must choose where each component runs. Three placements are common, and each shifts cost and risk to a different place.

Many clusters and many regions

Large organisations rarely run one cluster. If services in different regions take part in the same trace, the spans of that trace can start in different places. A tail-sampling design then faces a choice: send every span to a single central region, which costs bandwidth and adds delay, or make decisions locally and accept that some cross-region traces will be incomplete.

A common compromise is to keep most traces within their home region, since most requests never cross a region border, and to use a lightweight head decision for the few that do. Teams should measure what share of traces really cross regions before paying for a complicated design. Often the answer is small enough that a simple approach is good enough.

Follow one trace from the first click to the moment the sampler keeps or drops it.

A trace through a tail-sampling pipeline

Let us walk through Pattern B with a single shopping request. The story shows where time is spent, where memory is used, and where decisions happen.

sequenceDiagram
  participant U as User
  participant G as Web gateway service
  participant P as Payment service
  participant A as Agent Collector
  participant T as Tail sampling gateway
  participant B as Backend
  U->>G: Click pay
  G->>P: Call with trace context
  P-->>G: Error returned
  G-->>U: Failure shown
  G->>A: Send spans
  P->>A: Send spans
  A->>T: Route by trace ID
  T->>T: Buffer and wait for late spans
  T->>T: Apply policies and current rates
  T->>B: Keep trace and stamp probability
        

The lifecycle in plain steps

When things go wrong in the pipeline

A tail-sampling pipeline is a chain of machines, and chains break. Planning for the breaks is part of the design, not an afterthought. Four failure stories are worth knowing.

First, a gateway instance restarts. Everything it was holding in memory is gone, which means the traces that were waiting for a decision vanish. The loss is usually small, but during a rolling update it can repeat several times, so schedule updates for calm periods and update one instance at a time.

Second, the gateway runs short of memory. A well-built system starts discarding the oldest or least interesting traces on purpose instead of crashing. You want this behaviour to be visible in your metrics so that you know when it happens.

Third, the set of gateway instances changes because of scaling. When an instance is added or removed, the routing layer sends some traces to a different instance halfway through their life. Those traces may be judged on partial information. This is a known cost of scaling tail sampling, and it is a good reason to scale in small, planned steps.

Fourth, clocks disagree. Services on different machines may record slightly different times, which can make a trace look longer, shorter, or even out of order. Keep machine clocks synchronised and avoid latency rules so tight that a small clock difference changes the outcome.

Queues, back-pressure, and overload

Between each pair of stages in a pipeline sits a queue that absorbs short bursts. When the next stage is slow, the queue grows. If it grows too long, something must give: the pipeline can refuse new data, slow down the sender, or drop spans. Each choice has a cost, and the right one depends on how much you value completeness against how much you value the health of your applications.

Most teams choose to protect the applications first. The Collector should refuse or drop data rather than push pressure back into a customer-facing service. When the pipeline does drop data under overload, it is much better to drop whole traces than random spans, because a trace with missing spans is confusing and a missing trace is simply absent. Counting these overload drops separately from deliberate sampling drops tells you whether a gap in the data was a design decision or a symptom of trouble.

The waiting-period dilemma

The waiting period is a trade-off with no perfect answer. If it is too short, long traces are judged before their slow spans arrive, and the sampler may drop a trace that would have been marked slow. If it is too long, memory use rises and traces appear in your backend later than people expect. Most teams measure the real length of their longest common traces and set the wait just above it, then watch a metric that counts spans arriving after their trace was already decided.

Sampling saves money but can quietly poison your charts. This chapter explains how to avoid that.

The problem of biased counts

Suppose you keep 1 percent of ordinary requests and 100 percent of failed requests. If you count the traces in your backend, errors will look enormously more common than they really are. The sample is deliberately unfair, and any statistic computed from it without correction will be wrong.

Adjusted counts

The fix is to give each kept trace a weight equal to one divided by its keep probability. An ordinary trace kept at 1 percent counts as a hundred. A failed trace kept at 100 percent counts as one. When you add up the weights instead of the raw rows, the totals match reality. This is why every adaptive sampler must record the probability it used. If the rate changes every few seconds, the weights change too, and only a per-trace stamp can keep the sums correct.

Compute rate, error, and duration metrics before sampling

The safest way to protect your dashboards is to compute headline metrics from all the data, before any trace is dropped. In a Collector, a component that turns spans into metrics can sit ahead of the sampling stage so request counts, error rates, and latency distributions reflect every request. Sampled traces then play their proper role: they are the evidence you open when a metric looks wrong, not the source of the metric itself.

flowchart LR
  IN["All incoming spans"] --> SM["Span to metrics conversion"]
  IN --> SP["Sampling stage"]
  SM --> MB["Metrics backend with full counts"]
  SP --> TB["Trace backend with kept traces"]
        

A worked example with weights

Let us put numbers on the idea. Imagine that during one minute a service handles 60,000 requests. Of those, 59,400 succeed and 600 fail. The sampler keeps every failure, so 600 failed traces are stored, each with weight one. It keeps 1 percent of successes, so about 594 successful traces are stored, each with weight one hundred.

If you count stored rows, you see 600 failures and 594 successes, which suggests that almost half of all requests fail. That reading is badly wrong. Now add up the weights instead. Failures contribute 600 times one, which is 600. Successes contribute 594 times one hundred, which is 59,400. The weighted view says that 600 out of 60,000 requests failed, which is exactly one percent, and matches reality.

This small calculation is the reason adaptive sampling and honest dashboards must travel together. A backend that understands weights can do the correction for you. A backend that does not should never be used to calculate rates from sampled traces.

Statistical honesty about small samples

Adjusted counts give the right average, but they do not give certainty. If a key was sampled at 0.1 percent and only three traces were kept, the estimate “three thousand requests” has a wide margin of error. Adaptive samplers help here because they raise rates for quiet keys, which gives those keys enough samples for reliable estimates. Still, tell your readers when a number comes from a thin sample.

Rate changes and the staircase effect

Because the rate changes over time, a graph of kept traces per second is a poor picture of real traffic. It may stay flat while real traffic triples, because the sampler is working hard to keep the output steady. Always plot weighted counts, not raw kept counts, when you want to see demand.

Adaptive sampling is powerful, but it is not free. A balanced view helps you decide when it is worth the effort.

Adaptive versus fixed, side by side

When adaptive sampling is not worth it

Adaptive sampling is not a badge of maturity. For many teams, a simpler approach is the better engineering choice. Consider staying with fixed sampling when your volume is small and your bill is comfortable, when traffic is steady enough that a fixed percentage works all year, when your team is small and cannot afford another system to operate, or when your questions can be answered from metrics and logs and traces are only an occasional extra.

A good test is to ask what decision the extra complexity will improve. If nobody can name a question that the adaptive design will answer and the fixed design cannot, wait. Systems that are simple are easier to debug at three in the morning, and that has real value.

A system that decides what you remember must itself be protected, measured, and trusted.

Security considerations

Compliance and privacy

Sampling is sometimes mistaken for anonymisation. It is not. A trace that is kept is kept in full, with every attribute it carries. If a rule says “always keep traces for this customer”, that customer’s data is being stored more heavily than others, which may matter for privacy reviews. Document your keep rules the same way you document any other data processing.

Governance: who owns the sampling policy?

Sampling rules decide what the organisation will be able to see during its worst day, so they should not be an accident of whoever edited a file last. Healthy teams treat the sampling policy as a shared product with a named owner, usually the platform or observability team, and with input from the engineers who rely on the traces.

Write the policy in plain language: which traces are always kept, which services receive guaranteed floors, what the total budget is, and what the fallback behaviour is. Store the machine-readable version in the same place as other configuration, require review for changes, and keep a history so that you can answer the question “what were the rules on the day of the incident?”

Finally, make it easy for engineers to ask for exceptions. If a team is launching a risky feature, they should be able to request a temporary higher rate through a simple process that expires on its own. Without an easy path, teams invent their own workarounds, and the central design slowly loses control.

Monitoring the sampler

An adaptive sampler is a control system, and control systems need instruments. The following signals tell you whether it is healthy.

Capacity planning for tail sampling

Because tail sampling holds traces in memory, you can estimate the machine size you need with simple arithmetic. Take the number of new traces arriving each second, multiply by the waiting period in seconds, and you get the number of traces held at any moment. Multiply that by the average size of a trace and you have the raw memory need. Then add a generous safety margin, because spans carry extra overhead inside the program and traffic is never perfectly smooth.

Here is an illustrative example. Suppose 2,000 new traces arrive every second, each with about 30 spans of roughly one kilobyte, and the gateway waits 30 seconds. That means about 60,000 traces are held at once, and at around 30 kilobytes each the raw buffer is close to 2 gigabytes. With overhead and headroom, a team would plan for several times that amount, spread over more than one instance.

The same arithmetic shows why shortening the wait or applying a gentle head filter first has such a large effect: both reduce the number of traces in the buffer directly. It also shows why a sudden growth in span count per trace, perhaps after a new feature adds many database calls, can quietly double your memory need.

Teams keep rediscovering the same good ideas and the same expensive mistakes. Here they are, named.

Patterns worth copying

Anti-patterns to avoid

A practical checklist, grouped by the stage of your journey.

Before you start

• Name your questions. Write down the three or four questions traces must answer, such as “why was this checkout slow” or “which release broke payments”. Sampling rules should protect those answers.
• Measure your current volume. Know spans per second, spans per trace, and the cost per million spans. Without a baseline you cannot prove improvement.
• Decide your budget. Pick a target in traces per second or in cost per day. An adaptive sampler needs a goal to chase.

While you build

• Use parent-based decisions everywhere so traces stay whole.
• Layer your rules. Must-keep first, fairness second, adaptive third.
• Stamp the probability on every kept trace so counts can be corrected.
• Keep keys small and stable. Prefer route templates and status classes.
• Add safety limits. Cap memory, cap tracked keys, and define a fallback rate.
• Redact sensitive attributes before data reaches any buffer.

After you launch

• Monitor the sampler with the signals from the monitoring chapter.
• Review key lists regularly. New services and routes appear all the time, and old rules go stale.
• Test with game days. Simulate a traffic spike and a control-channel outage and confirm the system behaves as designed.
• Teach your team that stored traces are samples, and show them how weighted counts work.

A vocabulary cheat sheet

Sampling conversations go wrong quickly when two people use the same word for different things. Agree on these terms as a team and your design discussions will be shorter and calmer.

Team habits that keep sampling healthy

• Review the dropped data, not just the kept data. Occasionally inspect a sample of what the sampler discarded and confirm it really was boring.
• Write down the budget and the reasons for it. When finance asks why the tracing bill changed, the answer should be in a document, not in someone’s memory.
• Keep a dashboard for the sampler. The same people who watch service health should be able to see sampler health at a glance.
• Include sampling in incident reviews. After every serious incident, ask whether the sampler kept the traces that would have helped, and tune the rules if it did not.
• Teach new engineers early. A short onboarding note on “traces are samples, metrics are complete” prevents a whole class of confusion.

How the ideas show up in practice, from open-source tools to typical company situations.

Tools and ecosystems

Questions to ask before choosing a product

If you decide to buy rather than build, a short list of questions will quickly separate serious products from marketing language.

• Does it keep all errors and slow traces? Ask how it defines “slow” and whether the definition adapts per operation.
• Does it record the sampling probability? If not, ask how it keeps counts honest.
• How does it share the budget between keys? Look for floors, fairness rules, and caps on key count.
• What happens during overload or when its control service is unreachable? You want a calm, documented fallback.
• Where does it run and what does it need in memory? Tail sampling at scale needs real capacity.
• Can you see why a trace was kept or dropped? Transparency is essential for trust and for debugging.
• Does it work with standard OpenTelemetry data and propagation? Lock-in is a hidden cost that grows over time.

Typical situations (illustrative)

Short, direct answers to the questions people ask most often.

Not as a single switch. OpenTelemetry supplies samplers, probability fields in the trace state, a probabilistic sampler processor, a tail sampling processor, a trace-aware load balancing exporter, and channels for remote configuration. Fully automatic per-key rate adjustment is usually provided by a backend, a sampling proxy, or custom logic built on these parts.

Yes. A central service can watch volume and publish new rates that SDKs fetch regularly. The decision still happens at the start of the trace, so it cannot use errors or latency, but it can adapt to traffic levels and to different keys such as the operation name.

Yes, and that is where adaptive designs are most powerful. The gateway can keep all errors and slow traces and adjust the rate for everything else to meet a budget. The rate limiting and composite policies of the tail sampling processor give simple budget control, and custom or external controllers can add per-key adjustment.

Because the decision is based on the complete trace. If spans of one trace are split across instances, each instance sees only a part, and rules about total duration, errors anywhere in the trace, or span counts give wrong answers.

Only if you compute metrics from the sampled traces without correction. Generate metrics from all data before sampling, or weight each kept trace by one divided by its keep probability. Either approach keeps totals honest.

There is no universal number. Start from your budget and your questions. Many teams keep all errors, a strong share of critical flows, and a small percentage of routine traffic. Adaptive sampling is useful precisely because the right percentage changes with load.

SDKs and Collectors should keep using their last known strategy for a while and then fall back to a safe default rate that you chose in advance. Avoid fallbacks that drop everything or keep everything, because both create unpleasant surprises.

Trace sampling affects traces. Logs and metrics follow their own rules. A helpful habit is to link logs to trace identifiers and to record the sampled flag, so you can tell whether a matching trace should exist in your backend.

No. Tail sampling is smarter but costs more in network, memory, and operations. Head sampling is cheap and simple. Many teams combine them, and some stay with head sampling because their questions do not require outcome-based selection.

Check that output volume stays near the target, that important keys always have traces, that late and orphan spans are rare, that memory stays inside limits, and that weighted counts from sampled data agree with metrics computed from full data. If those hold, the sampler is doing its job.

Head-style samplers inside the SDK add a very small calculation per trace, usually unnoticeable. Tail sampling does not slow the request itself, but it does delay when a trace becomes visible in your backend, because the gateway waits before deciding.

Usually not. Metrics are small and valuable, so most teams keep them complete. Logs can be filtered by their own rules, and a useful habit is to keep logs that belong to kept traces so that the two tell a matching story.

Yes, and you should plan for it. A common approach is a special marker, such as a debug flag, that a trusted tool adds to a request and that a high-priority rule recognises. Make sure outsiders cannot set the marker on public traffic and that the override expires automatically.

Move your headline metrics so that they are computed before sampling, switch to parent-based head sampling, and add a rule that keeps errors. These three steps deliver most of the value, and each can be rolled back on its own.

You do not need to build everything at once. This roadmap moves from safe and simple to advanced, so each stage pays for itself before the next begins.

Six stages from fixed to adaptive

How mature is your setup?

How to decide you are ready for the next stage

Move forward only when the current stage is boring. If the sampler’s own metrics are steady, if engineers trust the traces they find, and if the monthly bill matches the forecast, you are ready. If any of those three is shaky, fix it first. A calm foundation makes every later improvement easier to measure and easier to roll back.

To make everything concrete, let us follow an imaginary adaptive sampler through one full day at an online shop. The numbers are illustrative, not measurements.

The setup

The shop runs a two-tier Collector design with a tail-sampling gateway. The team has set a budget of about 150 kept traces per second. Errors and very slow traces are always kept and do not count against the adaptive share. Every key has a floor, so no operation ever falls to zero, and every kept trace carries its probability.

The day in numbers

Dynamic, or adaptive, sampling is the practice of letting your tracing pipeline watch itself and adjust how many traces it keeps. It replaces a rule that follows a number with a controller that follows a goal. In OpenTelemetry you assemble it from SDK samplers, consistent probability fields, Collector processors, trace-aware routing, and remote configuration channels, or you adopt a backend or proxy that does the adjustment for you. The reward is a predictable bill and a store of traces that still answers your hardest questions. The price is more moving parts and a need to treat sampled data with statistical care. Start simple, protect errors first, record the probability on every kept trace, monitor the sampler like any other critical system, and grow the design only when a real question demands it.