Before we learn how to throw data away, we need to understand why anyone would want to. It all starts with the sheer size of modern traffic.

When a program runs on your laptop, you can look at every detail of what it does. When a program runs in production, serving real customers on many machines, every detail it produces has to be collected, sent over a network, stored on disks and searched later. That costs money, time and computing power.

The most detailed kind of information a system can produce is a trace, which is a record of the full journey of one request through all the services it touched. Traces are wonderful for finding out why something was slow or broken. But they are also big. One request can produce dozens of small records, called spans, and a busy system handles thousands of requests every second.

A quick calculation

Let us do the arithmetic for a modest shopping website. Suppose it handles 10,000 requests per second, and each request produces about 20 spans as it travels through the services. That is 200,000 spans every second. Multiply by the 86,400 seconds in a day and you get about 17.3 billion spans per day. If each span takes even one kilobyte to send and store, that is more than 17 terabytes of data every day, just for traces.

Now imagine that you keep only one request out of every hundred. The numbers fall to about 2,000 spans per second and about 173 million spans per day, which is a much more comfortable amount. You still see a statistically fair picture of what your system is doing, but the cost has dropped by a factor of one hundred.

The catch

Sampling saves a great deal of money, but it comes with a price: some requests will not be recorded at all. The big question of sampling is therefore not only “how much should we keep?” but also “when do we decide which ones to keep?” That second question is exactly what separates head-based sampling from its cousin, tail-based sampling, and it is the subject of this guide.

A few small words appear everywhere in tracing. Learn them once and the rest of this guide becomes easy.

Span

A span is a record of one piece of work, such as “the payment service charged the card” or “the database answered a query.” A span has a name, a start time, an end time and some extra details called attributes. Think of a span as one line in a diary that says what happened and how long it took.

Trace

A trace is a collection of spans that all belong to the same request. If a customer’s checkout touched six services, the trace holds the spans from all six, arranged like a family tree. The first span is the parent, the work it triggered creates child spans, and those can have children of their own.

Root span

The very first span of a trace, the one that has no parent, is called the root span. It is usually created by the first service that receives the request from the outside world, such as a gateway or a web server. The root span is the “head” of the trace. That is where the name “head-based” comes from.

Trace ID and span ID

Every trace has a unique trace ID, a long random number that is shared by all its spans. Every span has its own span ID. When one service calls another, it passes along the trace ID and the ID of the calling span, so that the second service can create a child span in the same trace.

Context and propagation

Context is a small bag of information that travels with a piece of work. In OpenTelemetry, it holds the identity of the current span. When the work crosses the network from one service to another, the context is written into the message headers by the sender and read out again by the receiver. This is called context propagation. It is the reason the second service can continue the same trace instead of starting a new one.

Now we can define sampling clearly and see the two big families of it.

Sampling is the practice of choosing which traces to keep and which to throw away, so that the cost of telemetry stays under control while the picture of the system stays useful. Sampling is not a failure or a shortcut. It is a standard, planned part of running tracing at scale.

Every sampling approach has to answer two questions. The first is who decides: which piece of software makes the call. The second is when the call is made: before the request starts, or after it is finished. The answer to the second question divides sampling into two families.

Where else the word “sampling” appears

The word is used for other things in monitoring too. Metrics can be sampled by collecting a reading every few seconds instead of continuously. Logs can be sampled by keeping only some of the repeated messages. In this guide, sampling always means trace sampling, the decision to keep or drop whole traces.

Here is the clear definition, followed by a walk through the idea piece by piece.

Head-based sampling is a sampling strategy in which the decision to keep or drop a trace is made at the very beginning of the trace, when the root span is created, using only the information that is available at that moment. Once the decision is made, it is recorded in the context and passed to every downstream service, which simply follows it.

In OpenTelemetry, the decision is made by a small component called a sampler. When your code or your framework starts a new span, the SDK asks the sampler: “Should this span be recorded and exported?” The sampler looks at what it knows and answers right away. The answer takes microseconds, and it costs almost nothing.

graph LR
  A["Request arrives at the first service"] --> B["SDK creates the root span"]
  B --> C{"Sampler: keep this trace?"}
  C -->|"Yes"| D["Span is recorded and exported, sampled flag set to on"]
  C -->|"No"| E["Span is not exported, sampled flag set to off"]
  D --> F["Context with the flag travels to the next service"]
  E --> F
  F --> G["Next service follows the same decision"]
    

Follow the figure from left to right. A request arrives. The SDK creates the root span. The sampler is asked for a decision. If the answer is yes, the span is recorded and later exported, and a small flag in the context is turned on. If the answer is no, the span is not exported, and the flag is turned off. Either way, the context travels along with the request, so the next service can read the flag and make the same choice.

The two key properties

• The decision is early. It happens before the request has been processed, so the sampler cannot know whether the request will be slow, will fail or will be boring. It must decide using only what is visible at the start, such as the trace ID, the name of the operation, and some attributes.
• The decision is shared. Because the flag travels with the context, all services in the same trace follow the root’s decision. A trace is therefore either kept whole or dropped whole, not kept in pieces.

Why it is called “head”

A trace has a head, which is its beginning, and a tail, which is its end. A head-based sampler decides at the head. A tail-based sampler waits for the tail. The names are simple pictures of when the decision is made.

Inside the SDK, a sampler is a tiny function with clear inputs and a clear output. Understanding them makes every later chapter easier.

What the sampler is told

When a new span is about to start, the SDK gives the sampler a small set of facts. These are the only things a head-based sampler can use, because nothing else exists yet.

• The parent context. Is there a parent span? If so, was the parent sampled? Was the parent in the same program or in a remote service?
• The trace ID. The long random number that identifies the trace. Many samplers use it to decide fairly and consistently.
• The span name. For example, “GET /checkout.”
• The span kind. Is the span a server handling a request, a client making a call, a producer sending a message, or an internal step?
• The starting attributes. Details known at the start, such as the HTTP method or the target address.
• Links. References to other spans that are related but are not the parent.

What the sampler answers

The sampler returns one of three decisions. The names come from the OpenTelemetry specification.

The sampler can also add extra attributes to the span and can change a note called tracestate, though most samplers do not need to.

“Recorded” and “sampled” are not the same

Beginners often mix up these two words. A span is recording when the SDK collects its data, such as attributes and events, in memory. A span is sampled when it will actually be sent to the backend. A “record only” span is useful for internal processing, such as a span processor that wants to see every span’s data even though it will not be exported. In most simple setups, you will only meet the other two decisions: drop, where nothing is collected, and record and sample, where everything is collected and exported.

A dropped span still travels

One important detail: even when a span is dropped, the context still travels to the next service, with the sampled flag turned off. A dropped span is a lightweight placeholder that costs almost nothing, but it keeps the trace ID and span ID flowing. This is what allows downstream services to see the “off” flag and also skip the trace, and it also allows logs from different services to share the same trace ID even when the trace itself is not stored.

OpenTelemetry ships with a small set of ready-made samplers. Most real systems use just a combination of these.

How they are used together

The first three are simple. The fourth, ParentBased, is the glue that makes head-based sampling work across many services. A typical production setup is “ParentBased wrapped around TraceIdRatioBased.” In words: when a request arrives with a decision already attached, obey it. When a request arrives with no decision, which means this service is the root, make a fresh random decision with a fixed ratio.

The ratio sampler is the workhorse of head-based sampling. Its secret is that it uses the trace ID itself as a source of fairness.

A trace ID is a long random number. If the IDs are truly random, then the digits of the number are spread evenly. That means you can look at a piece of the number and use it like a lottery ticket. The ratio sampler takes a portion of the trace ID, treats it as a number between zero and one, and compares it with the configured ratio. If the number is smaller than the ratio, the trace is kept. If not, it is dropped.

Why determinism is helpful

Because the answer depends only on the trace ID, the ratio sampler does not need to remember anything. It keeps no counters and no memory of earlier traces. It does not need to contact any other component. It is a stateless, instant calculation, which is why it is so cheap to run on every request.

Determinism also helps in situations where the sampled flag is lost. Imagine a message that passes through a system that drops headers. The next service starts a brand new context for the same trace ID. If every service uses the same ratio, they still make consistent decisions for the same ID. That is a safety net, although you should not rely on it. The proper way to keep decisions consistent is to follow the parent, as described in the next chapter.

A note on newer approaches

The OpenTelemetry community has been refining how probability sampling works so that services using different ratios can still make consistent choices, and so that the actual sampling rate can be recorded in the trace and used later to correct counts. This work is often called consistent probability sampling. It builds on the same idea of using the trace ID, adds a small note in the trace state, and is rolling out gradually. Check the documentation for your SDK version to see what is currently supported.

ParentBased is the single most important piece of head-based sampling across services. It turns a local decision into a system-wide decision.

A request can enter a service in several different situations. The service might be the very first one, with no parent at all. Or it might receive the request from a remote service whose decision was “sampled.” Or from a remote service whose decision was “not sampled.” The same three situations can occur for a parent that lives inside the same program. ParentBased lets you set a separate behavior for each situation.

Almost nobody changes the four “follow the parent” defaults. The only slot you normally set is the root sampler. That is the one that decides what happens when a new trace begins in this service.

Why following the parent is so valuable

Imagine that three services each used a plain 10 percent ratio sampler with no ParentBased wrapper, and imagine that their trace ID checks happened to differ, for example because they used slightly different ratios. Service A might keep a trace, while service B drops it, and service C keeps it again. The result is a trace with a hole in the middle. The spans of A and C exist, but they cannot be connected, because B’s span is missing.

With ParentBased, the problem disappears. Service A, the root, decides. B and C simply obey. The trace is either whole or entirely absent.

The remote and the local parent

Why does the specification distinguish between “remote” and “local” parents? Because sometimes you might want different behavior at the boundary of your system. For example, you might want to ignore the sampling decision of an untrusted outside caller (a remote parent) but obey the decisions made by spans inside your own program (local parents). Having separate slots allows exactly this kind of rule.

Let us follow one sampled request and one unsampled request through a small system, and watch how the decision travels.

sequenceDiagram
  participant U as User
  participant A as Gateway root, ratio 10 percent
  participant B as Orders Service, ParentBased
  participant C as Payments Service, ParentBased
  U->>A: Request one
  Note over A: Trace ID random check says keep, flag on
  A->>B: Call with context, sampled flag on
  B->>C: Call with context, sampled flag on
  Note over A,C: All three spans recorded and exported
  U->>A: Request two
  Note over A: Trace ID random check says drop, flag off
  A->>B: Call with context, sampled flag off
  B->>C: Call with context, sampled flag off
  Note over A,C: No spans exported, IDs still travel
    

In the first request, the gateway is the root. Its sampler looks at the new trace ID and decides to keep the trace, so the flag is on. The gateway calls the orders service and passes the context. The orders service has a ParentBased sampler. It sees a remote parent with the sampled flag on, so it also samples. It calls the payments service, which does the same. The result is a complete trace of three spans.

In the second request, the gateway’s random check says drop, so the flag is off. Both downstream services see the “off” flag and also drop. Nothing is exported, but the trace ID and span IDs still travel along, which is helpful for logs.

The lifecycle of one span

• Start. The SDK asks the sampler for a decision. If the decision is drop, it creates a lightweight non-recording span that does almost nothing.
• During the work. Recording spans collect attributes and events in memory. Non-recording spans ignore all such calls, which is why dropping saves processing time.
• End. A recording and sampled span is handed to the span processor, which batches it and gives it to the exporter. A dropped span simply disappears.
• Export. The exporter sends the batch to a collector or backend, which stores it and makes it searchable.

Notice where the savings come from. A dropped span avoids the work of storing attributes, building batches, serializing data, sending it across the network and storing it in the backend. That is a major reason head-based sampling is cheap: the savings begin immediately, at the source.

The good news is that you rarely need to write code. OpenTelemetry defines two standard settings that control sampling in almost every SDK and agent.

The first setting is OTEL_TRACES_SAMPLER. It chooses the sampler by name. The second is OTEL_TRACES_SAMPLER_ARG. It carries the argument that some samplers need, such as the ratio.

For most production systems, the best choice is parentbased_traceidratio with an argument such as 0.05, meaning five percent. Put this same setting on every service. The services that are roots will apply the ratio, and the others will follow their parents.

Where to set it

• In deployment files. Put the two settings in your container or orchestration configuration, next to your other environment variables. That way the rate is visible, reviewable and easy to change per environment.
• In an agent configuration. Auto-instrumentation agents read the same variables, so you can control sampling without changing any code.
• In code. If you need a custom sampler, you configure it when you build the SDK’s tracer provider. The environment variable cannot describe custom logic.

Different rates in different environments

It is common to use “keep everything” in development and testing, a small ratio in production and a larger ratio in a staging environment where you want enough traces to validate changes. Because the setting is just text in a configuration file, switching between these is simple.

The question every team asks is “what percentage should we keep?” There is no magic number, but there is a sensible way to think about it.

Choosing a rate is a balance between three forces: cost, detail and the chance of seeing rare events. Let us look at each force with numbers.

Force one: cost

Cost grows in direct proportion to the rate. Keeping 10 percent costs ten times as much as keeping 1 percent. If your tracing budget is fixed, the budget divided by the per-trace cost tells you the maximum affordable rate. Remember that the cost includes not only storage but also network use, processing in the collector and the time engineers spend searching through more data.

Force two: detail

For common, frequent operations, a small rate still gives plenty of traces. Suppose the checkout endpoint handles 100 requests per second. At 1 percent, you keep one trace per second, which is 3,600 traces per hour. That is more than enough to understand typical behavior.

Force three: rare events

Rare events are the hard part. Suppose errors happen in one request out of a thousand, and your system handles 10,000 requests per second. There are 10 errors every second. With a 1 percent head-based rate, you keep about 0.1 error traces per second, which is about 6 per minute. That is probably fine. But if the system handles only 10 requests per second, there is one error every 100 seconds, and a 1 percent rate keeps one error trace every 10,000 seconds, which is about once in nearly three hours. For low-traffic or very rare events, head-based sampling at a small rate may effectively never catch what you are looking for.

A practical way to decide

• Start from the traffic of your busiest important endpoint. Decide how many traces per second you want to see from it, for example one to five.
• Compute the ratio. Divide the number of traces you want per second by the number of requests per second.
• Raise the rate for low-traffic services. A rate that works for a busy gateway is too low for a service that handles ten requests a minute.
• Plan for rare events separately. Use rule-based sampling or tail-based sampling for errors and slow requests rather than raising the global rate to unaffordable levels.
• Review after launch. Check real costs and real usefulness after a few weeks, and adjust.

A flat percentage treats every request the same. Real systems often want different rules for different kinds of traffic. Custom samplers make that possible, within the limits of what is known at the start.

Because a sampler receives the span name, the span kind and the starting attributes, you can write a sampler that makes different choices for different requests. This is still head-based sampling, because the decision is made at the start, but it is smarter than a single percentage.

Examples of useful rules

The big limit: only the start is visible

A head sampler can only use facts that exist at the moment the span starts. It does not know:

• Whether the request will end in an error.
• How long the request will take.
• What the response code will be.
• What happens in downstream services.

So a rule such as “keep every trace that has an error” cannot be done at the head, because the error has not happened yet. That kind of rule belongs to tail-based sampling, which is discussed in the next chapters. A head sampler can only guess in advance, using the information it has: which route, which customer, which method.

Writing a custom sampler safely

• Keep it fast. The sampler runs on every new span, so it must finish in microseconds. Never call the network or read from a database.
• Keep it safe. Never throw an error. If the rules cannot decide, fall back to a default decision.
• Always wrap it in ParentBased. Your custom logic should be the root sampler. Let ParentBased make sure that the rest of the system follows the decision.
• Keep it deterministic when possible. Where it makes sense, base the decision on the trace ID so that results are fair and repeatable.
• Document the rules. A custom sampler is hidden logic. Write down what it does so that others can understand why traces are missing.

To fully understand head-based sampling, it helps to compare it with its sibling. Each has strengths that the other lacks.

graph TD
  A["Request starts"] --> B["Head-based: decide now using the trace ID and start details"]
  B --> C["Keep or drop immediately at the source"]
  A --> D["Tail-based: record spans and send them to a collector"]
  D --> E["Collector holds all spans of the trace in memory"]
  E --> F["Trace finishes, collector looks at errors, duration and attributes"]
  F --> G["Keep or drop after seeing the whole story"]
    

Look at the two paths in the figure. On the first path, the decision is quick and happens at the source, so almost no data is produced for dropped traces. On the second path, every span is produced and sent to a collector, which holds the spans of each trace in memory until the trace seems complete, and only then decides. The second path is more powerful, and also more demanding.

The honest summary

Head-based sampling is simple, cheap and robust, but it is blind to outcomes. Tail-based sampling is smart and can keep the interesting traces, but it needs infrastructure, memory and care. Neither is better in every case. They answer different needs.

Why tail-based sampling needs all spans in one place

To judge a whole trace, the tail sampler must see all of its spans. But the spans come from many services, and they are sent to the collector at different times, possibly through different collector instances. If the spans of one trace are scattered across several collector machines, none of them has the full picture. So tail-based setups usually add a routing layer that sends all spans with the same trace ID to the same collector instance. They also wait for a fixed period for late spans before deciding. These extras are the price of the extra power.

The two approaches are not rivals. Many large systems use both, so that each covers the other’s weakness.

A common design has two stages. First, a head-based sampler in the SDKs removes the bulk of ordinary traffic cheaply. Second, a tail-based sampler in the collector looks at the remaining traces and keeps the interesting ones. The head stage reduces the load on the tail stage, and the tail stage finds the stories that the head stage would have missed.

Collector-side probabilistic sampling

The OpenTelemetry Collector also offers a probabilistic sampler that makes a decision using the trace ID, much like the ratio sampler in an SDK. It is applied after spans have already been produced and sent, so it does not save the cost of producing and transmitting them. But it can reduce the cost of storing them, and it is a convenient way to apply one rate to traces coming from many services that are hard to change. Because it uses the same trace ID idea, it keeps whole traces together as long as the ratios are consistent.

Here are the situations where head-based sampling is the natural choice. The scenarios are illustrative examples meant to build intuition, not descriptions of any particular company’s internal systems.

Sampling does not only change your trace storage. It changes how you should read everything that is built from traces, and how traces connect to logs.

Metrics built from traces can lie

Some tools compute numbers from spans, such as the number of requests per endpoint, the error rate or the average duration. If only one in a hundred traces is kept, then the number of spans is also one hundredth of the true number. A chart that says “12 requests per second” may really mean “1,200 requests per second, seen through a 1 percent window.” If you forget this, dashboards will appear to show a collapse in traffic every time somebody lowers the sampling rate.

There are two main remedies. The first is to compute such metrics from separate metric instruments that count every request, not from sampled spans. A counter that increments for every request is unaffected by trace sampling. The second is to record the sampling rate along with each kept trace, and to multiply counts by the inverse of the rate when estimating totals. The second approach is what the newer consistent probability sampling work aims to make easy.

Alerts should not depend on sampled traces

An alert that says “page me if there are more than five error traces per minute” will behave differently at different sampling rates. Worse, at a low rate, a real incident might produce zero sampled error traces for a few minutes. Alerts should be based on metrics that count every event, such as error counters and latency histograms, which are not sampled. Traces are for investigating, not for detecting.

Logs and traces

Many systems add the trace ID and span ID to every log line, so that an engineer can jump from a log message to the trace. With sampling, many of those trace IDs point to traces that were dropped. The link leads nowhere. This is normal, but it surprises people. Remember that the trace ID is still present even for dropped traces, because the context travels even when the span is not recorded. You can still use the trace ID to search logs across services, which keeps the logs useful on their own.

Some log tools can also be told to keep only the logs that belong to sampled traces. That reduces log volume, but it also means that logs for dropped traces are lost, so choose carefully.

Head-based sampling is simple and powerful, but it is not magic. An honest look at both sides helps you use it wisely.

The central trade-off: cost and simplicity versus insight

Every choice in sampling moves a slider between two ends. On one end, you pay little, run little infrastructure and see a fair but blind sample. On the other end, you pay more, run more machinery and see the stories that matter most. Head-based sampling sits at the cheap and simple end. For many systems, that is exactly the right place to be. For systems where every failure matters, it is a good first layer but not the whole answer.

When head-based sampling is a good fit

• You have high traffic, and the typical behavior of common requests matters most.
• You need a low-cost, low-effort setup that works across many teams.
• You can detect failures by other means, such as unsampled metrics and alerts.
• Your budget is fixed and must be predictable.

When it is not enough on its own

• You must be able to inspect every failed or slow request.
• Traffic is low and failures are rare, so a small rate would hide them.
• Compliance or audit rules require a record of every transaction, which should be handled by dedicated audit logs and not by traces.

Sampling is not a single place in your system. Data can be thinned out at several points along its journey, and each point has different powers and different costs.

graph LR
  A["Application with SDK sampler"] -->|"spans that were kept"| B["Agent or Collector close to the application"]
  B -->|"optional probabilistic or tail sampler"| C["Central Collector"]
  C -->|"optional tail sampler"| D["Tracing Backend"]
  D --> E["Retention rules and storage limits"]
    

Read the figure from left to right. The first and earliest place is the sampler inside the SDK. This is where head-based sampling lives. Everything dropped here costs almost nothing, because it never becomes a real span. The next places are collectors, which receive spans and can apply their own rules, such as a probabilistic sampler or a tail sampler. Finally, the backend itself may apply retention rules, discarding older data or limiting how much it will accept.

Why the earliest cut saves the most

Imagine a factory that makes bread. If you decide not to bake a loaf at all, you save the flour, the oven time, the packaging and the delivery. If you decide to throw the loaf away at the shop, you have already paid for everything before that. A head-based sampler decides before baking. Later stages decide after much of the cost has been paid, although they may still save storage and some processing.

The danger of stacking samplers carelessly

When several stages sample independently, their rates multiply. If the SDK keeps 10 percent and a collector then keeps 10 percent of those, you end up with only 1 percent. This is not always intended. Teams sometimes find that their effective rate is far below what they configured, because somebody added a second sampler in a collector without telling the application teams. Keep a written map of every place where sampling happens, and calculate the combined rate.

Consistency between stages

Stages that make decisions by trace ID can stay consistent with each other if they use the same method and compatible rates. Whole traces are then kept or dropped together. If a collector sampler uses a different method, for example random choice per span instead of trace ID, it will cut holes into traces that the head stage kept whole. Always choose collector samplers that decide per trace, not per span, unless you have a very specific reason.

Web calls are the easy case. Many real systems also pass work through queues, scheduled jobs and short-lived functions. Sampling works there too, but a few details need care.

Message queues

When a producer puts a message on a queue, it writes the trace context, including the sampled flag, into the message’s metadata. A consumer that reads the message later makes its decision using a ParentBased sampler, which follows the flag. This keeps the whole chain, from the producer through the queue to the consumer, either fully recorded or fully dropped.

There are two details worth knowing. First, the consumer may run long after the producer, so the trace can be spread over a long time. Second, a consumer that processes a batch of messages from many different traces handles several decisions at once. A good consumer reads the context of each message separately, and links its own work to each producer rather than pretending that all messages belong to one trace.

Scheduled and batch jobs

A job that starts on a timer has no incoming request, so it has no parent. It is a root. The sampler for its root span decides whether to keep this run. For jobs that run once an hour or once a day, a low ratio such as 1 percent would almost never keep a single trace. For such rare, important runs, it is usually better to sample at 100 percent, because the volume is tiny and each run matters.

Serverless functions

Functions that run on demand in a serverless platform start quickly and stop quickly. Two points matter here. The sampler must be cheap, because it runs on every start, and a ratio sampler fits well. And the exporter must send its data before the function is frozen or stopped, otherwise sampled spans can be lost. Many serverless tools provide a way to flush data at the end of each call. Because a function is often called by another service, a ParentBased sampler lets it follow the caller’s decision.

Long-running requests and streaming

Some requests last for minutes or hours, such as file uploads, video streams or long polling connections. Their decision is made at the start, like any other, but the spans can stay in memory a long time. Consider sampling them at a lower rate or breaking the work into several shorter traces, so that memory use stays under control.

These mistakes appear again and again in real systems. Each one looks harmless, and each one can silently break traces or inflate costs.

These habits keep head-based sampling predictable, cheap and trustworthy. Think of them as a checklist to run whenever you touch sampling settings.

A short pre-launch checklist

• Is every service configured with ParentBased, and do all root services share the same intended rate?
• Do all services propagate context correctly, including through queues and proxies?
• Do dashboards and alerts rely on unsampled metrics rather than on sampled spans?
• Is the public edge configured to handle untrusted sampling flags?
• Have noisy routes such as health checks been handled?
• Do low-traffic services have a rate that produces a useful number of traces?
• Is there a plan for catching rare failures, for example error metrics, logs or a tail-sampling stage?

Sampling looks like a purely technical dial, but it affects what data you store, who can influence your system, and how much you pay.

Security

• Forced sampling. The sampled flag travels in a header that the caller controls. If your public edge trusts it blindly, an attacker can set it on every request and force your system to record and export everything. This is a form of resource exhaustion that hits your bill and your storage. Make a fresh decision at the public edge.
• Trusted debug headers. A special header that forces sampling is a very handy debugging tool, but it must only be honored for authenticated, trusted callers. Otherwise it becomes the same attack, with an invitation.
• Sampler logic and untrusted input. A custom sampler that uses request attributes must not let outsiders raise their own sampling rate or crash the sampler with strange input.
• Telemetry transport. The spans that you keep must travel and be stored securely, with encryption in transit and access control in the backend.

Privacy

Sampling reduces the amount of data stored, but it does not remove sensitive data from the traces that are kept. A kept trace still carries every attribute that your instrumentation added. If a span contains an email address, a card number or a patient name, that data is stored whenever the trace is sampled. Sampling is a cost tool, not a privacy tool. Protect sensitive data by not putting it in attributes in the first place, and by using processors in the collector to remove or mask it.

One subtle privacy point is that a lower sampling rate can make a rarely used feature appear almost nowhere in the data. That is fine for privacy but might hide security-relevant activity. Security monitoring should not rely on sampled traces.

Cost

Cost is where sampling shines. A rough way to think about total tracing cost is:

• Requests per second, multiplied by
• spans per request, multiplied by
• the sampling rate, multiplied by
• the average size of a span, multiplied by
• the retention period in the backend.

Of these five factors, the sampling rate is the easiest one to change quickly. Halving the rate halves the number of stored spans. A change in a configuration file takes effect after the next restart or deployment, which is much easier than shortening retention or removing instrumentation.

A wrong sampling setup rarely causes an error message. It causes missing or lopsided data, which you notice only when you need it. A few deliberate checks prevent that.

Check the ratio statistically

Generate a large number of random trace IDs, for example one hundred thousand, and run them through your configured sampler. Count how many are kept. The share should be very close to the configured ratio. With a ratio of 10 percent and one hundred thousand trials, you should see about ten thousand kept, with a small natural variation. If you see a very different share, the trace IDs are not random enough, or the configuration is not what you think.

Check that decisions are consistent across services

Create two samplers configured like your two services and feed them the same trace IDs. Confirm that, with ParentBased, the second always follows the first’s decision when it is given the first’s context. Also test the case with no parent, to confirm that the root sampler is used.

Check that flags survive the journey

In a staging environment, send requests through every kind of path in your system: web to web, web to queue to worker, and through each proxy or gateway. Confirm that the sampled flag arrives unchanged at the last hop. If it is lost or reset at some hop, the downstream services will start making their own decisions and traces will break.

Check the edge policy

Send a request from outside with the sampled flag set on, and another with it set off, and confirm that your edge handles both as your policy says. If the policy is “ignore incoming flags,” neither should change what the edge decides.

Check the effect on the backend

After deployment, compare the real share of traces that arrives in the backend with the intended rate. Measure the number of root spans per minute against the known request rate from an unsampled metric. The ratio of the two should match your configured rate.

When traces look strange, sampling is often the first suspect. Here is a calm way to find out whether it is really the cause.

Signs of a sampling problem

• Traces that start in the middle of a request, with no root span.
• Traces that stop halfway, with some services missing from the middle.
• The number of traces suddenly changes after a deployment, without a change in traffic.
• Engineers cannot find the trace for a known failing request.
• Costs rise or fall in a way that does not match traffic.

A troubleshooting walk-through

Measuring sampling health over time

Treat sampling as something to observe, not just configure. Track the number of root spans created, the number sampled, and the number exported, and compare them with the true request rate from an unsampled counter. The ratio of sampled to created should match the configured rate. A sudden change signals a configuration problem. A growing gap between sampled and exported signals trouble in the export path.

Short, direct answers to the questions beginners ask most often.

You started with a museum handing feedback cards at the entrance and ended with a full picture of how head-based sampling works in OpenTelemetry. Here is everything that matters in one place.

Head-based sampling makes one quick decision at the start of every trace, using only what is known at that moment, and then passes the decision to every other service through the context. It is cheap, simple and predictable, and it keeps traces whole. Its weakness is that it is blind to what will happen later, so it cannot promise to keep errors or slow requests. Used with ParentBased everywhere, a sensible rate, a careful public edge, and unsampled metrics for counting and alerting, it is the foundation of tracing at scale. When you need more insight, add a tail-based stage on top.