Before we meet the rate-limiting sampler, we need to understand the problem it was invented to solve.

Observability produces a lot of data

Modern applications are built from many small services that talk to each other. When a customer clicks “Buy”, that single click can touch a dozen services, several databases, and a couple of external payment providers. To understand what happened, engineers record a trace, which is a story of the whole journey of that one request. A busy system can handle tens of thousands of requests every second, and each request can produce dozens of small records called spans.

Recording and storing every one of those spans is expensive. It costs network bandwidth to ship them, CPU time to build them, memory to buffer them, and money to store and search them in a tracing backend. At large scale the bill for tracing everything can rival the bill for running the application itself. That is the first reason sampling exists: cost.

Data you cannot use is also a problem

The second reason is human attention. If a team stores a billion nearly identical healthy traces, the few interesting ones are buried like needles in a haystack. Searching becomes slower, dashboards become noisier, and engineers spend more time filtering than learning. A well-chosen sample keeps the picture clear while discarding repetition.

The third reason is overhead on the application itself. Creating a full trace with all its attributes, events and links takes work inside the running program. When a service is already under heavy load, extra instrumentation work can make a bad situation worse. Sampling lets the application skip that work for requests it has decided not to record.

The hidden price of keeping everything

Teams often discover the cost of full tracing only after the first large invoice arrives. The expense is not one single item but a chain of small ones. Each span must be built in memory, serialised into a transport format, pushed through a network connection, received by a collector, queued, written to storage, indexed so it can be searched, and finally kept for weeks or months. Every link in that chain has its own price, and every link grows in step with the number of spans.

There is also a quieter cost in reliability. Telemetry pipelines are real software with real limits. If a sudden flood of spans overwhelms a collector or a backend, the pipeline starts to lag, drop data at random, or even slow down the applications that feed it. In the worst case, the tool you rely on to understand an outage becomes one more thing that is broken during the outage. Deliberate sampling is therefore not only a money-saving trick but also a way of keeping the observability system healthy under pressure.

Sampling is the discipline of deciding in advance, calmly, what share of the story is worth telling. The remaining chapters of this guide focus on one particular way of making that decision: giving each service a steady allowance of traces per second.

Three questions every sampling plan must answer

A rate-limiting sampler answers the first question beautifully, gives only a partial answer to the second, and does nothing for the third. Keeping these three questions in view prevents you from expecting more from any single sampler than it can give, and helps you assemble a plan in which each part has a clear job.

A few building blocks must be clear before sampling makes sense.

Why the root span matters most

In most systems the decision is made once, at the front door, and then everyone else follows it. If the first service says “record this trace”, every later service sees the sampled flag and records its spans as well. If the first service says “do not record”, later services skip the work. This keeps traces whole instead of leaving them with missing pieces, which would make them confusing to read.

Attributes, events and links: what a span carries

A span is more than a name and a stopwatch. It carries attributes, which are labels such as the route requested or the database table used. It can carry events, which are small time-stamped notes such as “cache missed” or “retry started”. It can carry links, which point to other traces that are related but not strictly parents, for example the message that triggered a background job. All of this makes spans valuable, and also makes them heavy. The more detail a span carries, the more it costs to keep, which is one more reason the number of recorded traces needs a firm limit.

Span kinds and their role in decisions

Spans are labelled by kind. A server span represents the handling of an incoming request. A client span represents an outgoing call to another service or a database. Internal spans represent work that stays inside one process. Producer and consumer spans describe sending and receiving messages. Sampling decisions are most often made on server spans that have no parent, since these mark the start of a trace entering your system. Understanding the kinds helps you see why only a few places in a large system actually consume credits.

Sampling can occur at two very different moments, and a rate-limiting sampler lives in one of them.

Head-based sampling

Head-based sampling decides at the beginning of a trace, when the root span is created. At that moment the system knows very little: the name of the operation, some starting attributes, and the trace identifier. It does not yet know whether the request will succeed, fail, or run slowly. The great advantage is cheapness, because a dropped trace costs almost nothing afterwards.

Tail-based sampling

Tail-based sampling waits until the trace is finished, or nearly finished, and then decides. By then it knows everything: whether there was an error, how long it took, and which services were involved. This allows clever rules such as “keep every failed trace and every slow trace”. The price is that all spans must be collected and held in memory until the decision is made, which needs more infrastructure.

flowchart LR
  A[Request arrives] --> B[Root span created]
  B --> C{Sampler decides}
  C -->|Record| D[Spans created and exported]
  C -->|Drop| E[Minimal work, nothing exported]
  D --> F[Collector]
  F --> G{Optional tail sampling}
  G -->|Keep| H[Tracing backend]
  G -->|Discard| I[Dropped]
        

A rate-limiting sampler is most commonly a head-based sampler that runs inside the application SDK. As we will see later, the same idea also exists as a policy in a collector doing tail-based sampling. The two versions share a goal but behave differently, and mixing them up is a frequent source of confusion.

Why not simply use tail-based sampling everywhere?

Tail-based sampling sounds like the obvious winner, because it knows the whole story before it decides. In practice it carries real burdens. Every application must still create and export every span, so the savings inside the application disappear. Collectors must hold spans in memory while they wait for traces to finish, and the amount of memory grows with traffic and with how long you wait. Spans of one trace must be routed to the same collector, which requires extra infrastructure and careful operation. If a collector crashes, the spans it was holding are lost.

Head-based sampling, by contrast, is almost free to run. This is why many organisations use both: a head-based stage near the application to cut the volume cheaply, and a tail-based stage in the collector to rescue the interesting traces from what remains. The rate-limiting sampler is a natural fit for the first stage, because its cost is tiny and its output is bounded.

A useful mental model: two filters in a row

Picture two sieves stacked one above the other. The upper sieve has large holes and removes most of the sand quickly and cheaply; it does not care what the grains look like. The lower sieve is finer and slower, and it keeps only the grains of a certain shape. Pouring everything through the fine sieve would clog it, but pouring the output of the coarse sieve through the fine one works well. A head-based rate limiter is the coarse sieve, and tail-based policies are the fine one.

OpenTelemetry defines a clear contract that every sampler follows, which is why you can swap one for another.

A sampler in OpenTelemetry is a small component attached to the tracer provider. Whenever the application wants to start a new span, the SDK asks the sampler a question: “Should this span be recorded?” The sampler is given the context of the parent, the trace identifier, the span name, the span kind, the starting attributes, and any links. From these it must produce an answer quickly, because this question is asked on the hot path of every request.

The three possible answers

Besides the decision, a sampler can return extra attributes to attach to the span and can supply an updated trace state. The trace state is a small, ordered list of vendor-specific key and value pairs that travels along with the trace context. Advanced sampling schemes use it to record information about how the decision was made.

What information is actually available at decision time

It helps to be precise about what the sampler can see, because that defines what it can sensibly decide. It sees the identifier of the trace, which is a long random number. It sees the name chosen for the span, such as the name of a route or an operation. It sees the kind of span, which says whether this is a server handling an incoming call, a client making an outgoing call, or an internal step. It sees the attributes that were supplied when the span was created, for example the HTTP method or the target address. It also sees any links to other traces and the context of the parent, if there is one.

What it cannot see is anything that happens later: the response code, the duration, the exceptions raised, or the results of downstream calls. Those facts do not exist yet. This explains why samplers that rely only on early information are good at controlling volume and poor at choosing interesting cases. A rate-limiting sampler goes one step further in simplicity, because it often ignores most of this information and looks only at whether a credit is available.

Why the decision must be fast

The question is asked for every root span, and in a busy service that means thousands of times per second. If a sampler took even a millisecond to think, it would add noticeable delay to every request. Good samplers therefore avoid network calls, heavy locking and slow computation. A simple arithmetic check on a counter, as used by a rate limiter, fits this requirement very well.

Knowing the standard menu of samplers makes it clear why a rate-limiting option is valuable.

Where does the rate-limiting sampler appear?

Here is something that surprises many beginners. The core OpenTelemetry specification lists the four standard samplers above, and a rate-limiting sampler is not one of them. Rate limiting is offered through other parts of the ecosystem. The best-known example is the Jaeger remote sampler family, available as extensions or contributed packages for several languages, which includes a rate-limiting strategy inherited from the Jaeger tracing project.

Rate limiting also appears in the OpenTelemetry Collector, where the tail sampling processor offers a policy that limits the number of spans passing per second. Other vendors and cloud providers ship samplers with reservoir-style limits as well, and teams are free to write their own sampler because the interface is open. Exact names and availability differ by language and version, so always confirm in the documentation of your SDK.

The remote sampler idea

The Jaeger-style remote sampler deserves a closer look because it is the most widely known home of rate limiting in this ecosystem. Instead of hard-coding the sampling rule inside the application, the application asks a central service, at regular intervals, which strategy it should use. The service replies with a strategy for that particular service name. The strategy can be a fixed yes-or-no, a percentage, or a rate limit expressed as a maximum number of traces per second. The application then applies the strategy it received.

The benefit is operational. Operators can change sampling behaviour for a whole fleet from one place, without rebuilding or redeploying any application. They can lower the rate when a backend is struggling and raise it when they need more evidence. The application keeps working with its last known strategy if the central service becomes unreachable, and falls back to a safe default at startup before the first reply arrives. This makes remote sampling a pleasant match for rate limiting, since the rate is exactly the number operators most want to adjust.

Reservoir-style variants

Some systems extend the simple bucket with a reservoir. A reservoir guarantees that a minimum number of traces per second are always kept, and beyond that minimum a percentage applies. This combination gives low-traffic operations a floor of visibility while preventing high-traffic operations from flooding the system. It is a close cousin of rate limiting, and you may meet it under cloud-provider tracing products. When you read about such a sampler, check whether its numbers are guaranteed minimums, ceilings, or both.

Now we can state the definition clearly.

A rate-limiting sampler is a sampler that lets through at most a fixed number of new traces per second and drops the rest. You configure a single number, for example “ten traces per second”, and the sampler enforces it. When traffic is light, nearly everything is recorded. When traffic surges, the sampler holds the output steady at the configured ceiling instead of letting it grow with the load.

Why a fixed rate is different from a percentage

Suppose you keep one percent of traces. At one thousand requests per second you record ten traces per second, which feels comfortable. When a marketing campaign pushes traffic to one hundred thousand requests per second, you suddenly record one thousand traces per second, a hundred times more. The cost, the network load and the storage all explode precisely when your system is already stressed.

A rate-limiting sampler solves this by making the output independent of the input. Ten per second stays ten per second, whether the system receives one hundred requests or one million. That predictability is the entire reason people choose it.

What the sampler promises and what it does not

It is worth stating the promise of a rate-limiting sampler very exactly. It promises that, over time, the number of new traces it admits will not exceed the configured rate, apart from a small allowed burst. It does not promise that the admitted traces are typical, that they are the most useful ones, or that any particular request will be kept. It also does not promise that the stored traces form a fair mirror of your traffic. Those are separate goals, and they are served by other tools.

Understanding this narrow promise protects you from disappointment. Engineers sometimes adopt a rate limiter expecting it to “keep the important stuff” and are upset to find that a rare failure was dropped. The sampler never claimed to know what was important. It only claimed to keep the flow within budget, and it does that job reliably.

Who benefits from this design

A short thought experiment

Picture two identical services. Both normally receive one hundred requests per second, and both are configured to record about ten traces per second. One uses a ten percent sampler, the other a limit of ten traces per second. On an ordinary day they behave identically and nobody can tell them apart.

Now imagine traffic leaps to one thousand requests per second for an hour. The percentage sampler now records one hundred traces per second, ten times its usual load, and its network usage, backend writes and storage all jump accordingly. The rate-limited service still records ten traces per second, and its costs do not move. Next imagine traffic falls to ten requests per second overnight. The percentage sampler records a single trace per second, while the rate limiter records all ten requests. Same configuration targets, very different behaviour under change. This simple picture explains almost every reason to choose one over the other.

Put simply, under steady traffic the two samplers look the same, while under changing traffic they behave as opposites.

Words you will meet when reading about rate limiting

These words appear throughout documentation, dashboards and architecture discussions. Knowing them makes it easier to follow conversations with platform teams, who often describe sampling problems in terms of bursts, headroom and retention rather than in terms of individual traces. A shared vocabulary lets everyone agree on what the limit protects and what it leaves unprotected.

Under the hood, most rate-limiting samplers behave like a bucket that slowly fills with credits.

Imagine a small piggy bank that receives a fixed number of coins every second, for example ten coins. Each time a new trace wants to be recorded, it must spend one coin. If a coin is available, the trace is recorded and the coin disappears. If the piggy bank is empty, the trace is dropped. The piggy bank also has a maximum size, so coins cannot pile up forever.

This design is commonly described as a leaky-bucket or token-bucket approach. The sampler does not need a background thread to count seconds. Instead, when a decision is requested, it looks at how much time has passed since the last request and adds the matching number of credits, up to the limit. This makes the check very fast and cheap, which matters because it runs for every request.

flowchart TD
  A[New root span requested] --> B[Add credits for elapsed time]
  B --> C{Credits at least 1?}
  C -->|Yes| D[Spend 1 credit]
  D --> E[Record and sample]
  C -->|No| F[Drop]
        

The two settings that shape behaviour

The first setting is the refill rate, which is the steady number of traces allowed per second. The second is the bucket capacity, which decides how large a short burst can be. If the capacity equals one second’s worth of credits, a quiet system can briefly absorb a burst of about that size before throttling begins. A tiny capacity makes the output very smooth, while a larger capacity tolerates bursts at the cost of occasional spikes.

Time, clocks and fairness

Because the sampler measures elapsed time, it depends on a clock. Implementations normally use a monotonic clock, which only moves forward and is not disturbed when the machine adjusts its wall-clock time. This matters: if the sampler used an ordinary clock and the system time jumped backwards, the credit calculation could behave strangely. Reliable implementations avoid this by measuring durations rather than reading calendar time.

Another detail is thread safety. A busy service handles many requests at the same moment on many threads, and all of them ask the same sampler for a decision. The sampler must update its credit balance safely, so that two requests never spend the same credit. Implementations use atomic operations or short locks for this. When both are cheap, the sampler adds almost no delay, but it does mean the sampler is one shared point of coordination inside the process.

What happens when the bucket is empty for a long time

During a long surge the bucket stays almost empty. Every second, a handful of new credits arrive, and the first few requests of that second consume them immediately. Everything else is dropped. The result is a steady trickle of sampled traces, each one admitted because it happened to arrive just after a refill. Nothing is queued or delayed; dropped requests are simply not recorded. The application itself keeps serving users at full speed, which is the whole point.

Following one request from entry to storage shows how the sampler cooperates with the rest of the pipeline.

sequenceDiagram
  participant U as User
  participant A as Service A (entry)
  participant S as Rate-limiting sampler
  participant B as Service B
  participant C as Collector
  participant T as Tracing backend
  U->>A: Request
  A->>S: Should this root span be sampled?
  S-->>A: Yes (credit available)
  A->>B: Call with sampled flag set
  B->>B: Parent says sampled, so record
  A->>C: Export spans
  B->>C: Export spans
  C->>T: Store complete trace
        

Notice that only the entry service pays a credit. This is crucial for correct accounting: the configured rate counts traces, not spans. A trace with fifty spans still costs a single credit, because the decision is made once for the whole trace. Planning your storage budget therefore requires multiplying the trace rate by the typical number of spans per trace.

What travels with the request

The sampled flag is carried in standard request headers defined by the trace context standard, so that services written in different languages and running on different platforms can understand each other. The header holds the trace identifier, the identifier of the calling span, and a small set of flags, one of which is the sampled bit. Each service reads these values when a request arrives, creates its own spans as children, and writes updated values into the headers of any outgoing calls.

A rate limiter spends credits only at the front of this chain, but the flag it sets influences every hop. This is why a broken link in propagation is so damaging. If one service forgets to forward the headers, the next service believes it is the start of a new trace, spends a credit of its own, and creates a separate trace. You then see two partial traces where there should be one, and your credit accounting is quietly wrong.

Asynchronous work and the decision

Work that continues after the original request has finished, such as a background task triggered by a message, raises an interesting question. Should it follow the original decision, or should it decide for itself? Many teams let the background task link back to the original trace rather than continue it, and give the new task its own decision. In that case the rate limiter treats it as a fresh root and spends a credit. Either choice can be reasonable, but you should make it on purpose and remember that it changes how many credits are consumed.

A rate limiter on its own is rarely enough in a distributed system.

If every service ran its own independent rate-limiting sampler, the results would be chaos. Service A might keep a trace while Service B drops the same trace because its own bucket happened to be empty. You would store broken traces with missing branches, which are confusing and wasteful. The cure is to wrap the rate limiter in a parent-based sampler.

The parent-based wrapper works like a polite rule: “If someone before me already decided, I follow them. If I am the very first, I ask the rate limiter.” In practice the rate limiter is used only for root spans, and every other span inherits the answer from its parent. The wrapper also lets you choose different behaviour for several situations, such as a remote parent that was sampled, a remote parent that was not sampled, a local parent that was sampled, and a local parent that was not.

A subtle point about entry points

Not every service that receives a request is a true entry point. A gateway may receive a call from a browser with no trace context and start a trace. An internal service receiving a call from that gateway should find a context and follow it. But sometimes a caller outside your control, such as a partner system or a browser tracing library, sends its own context. In that case your service sees a remote parent and, under the parent-based rule, follows the caller’s flag. Your rate limiter is bypassed, and the external caller has effectively decided how many traces you record.

This is usually harmless, but it can be abused or misconfigured. A badly behaved client that marks every request as sampled could force your services to record everything, defeating the budget. Teams that accept context from untrusted sources therefore often ignore incoming sampling flags at the edge, and apply their own rate limiter to every request that enters from outside. Internal hops then continue to follow the parent as normal.

The two most popular head samplers solve different problems, and choosing between them is a core design decision.

The statistics trap

One of the most important differences is how you count things afterwards. With a fixed percentage you can estimate real traffic by dividing the number of stored traces by that percentage. With a rate limiter, the share of requests that survive changes constantly: during quiet moments it may be close to one hundred percent, and during spikes it may fall below one percent. Counting stored traces and treating them as a proportional picture of real traffic will therefore mislead you.

Some teams combine both ideas, applying a percentage first to thin out the flow and a rate limit afterwards as a safety ceiling. Others use the rate limiter alone for simplicity. The right answer depends on whether you value predictable cost more than predictable statistics.

How the two behave during a day

Follow a single service through twenty-four hours. In the quiet early hours, traffic is low. A percentage sampler keeps a tiny fraction of that small stream, so engineers looking at overnight problems may find almost nothing. A rate limiter keeps nearly everything, giving rich detail precisely when there is little to look at. In the busy afternoon, the picture reverses. The percentage sampler produces a large and costly flow, while the rate limiter holds steady and gives a thin slice.

This means the two samplers give you visibility in different places. A percentage sampler gives you proportionally more evidence when you are busiest, which can help when you want to study heavy-load behaviour. A rate limiter gives you proportionally more evidence when you are quiet, which can help when you want to study rare events. Neither is better in general; each shapes the evidence differently.

Consistency across services

A percentage sampler based on the trace identifier has a pleasant property: any two services applying the same percentage to the same trace identifier reach the same answer without talking to each other. A rate limiter has no such property, because its answer depends on a local balance of credits. That is why the parent-based wrapper is not optional with a rate limiter. It supplies, through propagated context, the agreement that a trace-identifier rule would otherwise provide for free.

The same budget idea can also be applied after traces are complete.

The OpenTelemetry Collector can run a tail sampling processor that gathers spans, waits for traces to finish, and then applies a set of policies. One of the available policies limits the number of spans per second that may pass through. Because this policy runs in the collector rather than in each application, it controls the total flow reaching your backend, no matter how many application instances are producing data.

This is a different kind of limit from the SDK sampler. The head-based limiter counts traces and acts before spans exist, so it saves work in the application. The collector policy counts spans and acts after the work is done, so it saves storage and backend load but not application overhead. The unit also differs, which matters when you do your capacity arithmetic: traces per second in one case, spans per second in the other.

flowchart LR
  subgraph Apps
    A1[Service instance 1]
    A2[Service instance 2]
    A3[Service instance 3]
  end
  A1 --> L[Load-balancing collector layer]
  A2 --> L
  A3 --> L
  L --> TS[Tail sampling collectors]
  TS --> P{Policies: errors, latency, rate limit}
  P --> B[Tracing backend]
        

Why routing matters

Tail sampling only works if all spans of one trace reach the same collector instance. Otherwise each collector sees only fragments and cannot judge the whole trace. For that reason, larger setups place a first layer of collectors that routes spans by trace identifier to a second layer that performs the actual sampling. This extra machinery is the cost of the extra intelligence.

Memory, waiting time and the cost of patience

A tail sampling processor must decide how long to wait before judging a trace. Wait too briefly, and slow traces are judged before their late spans arrive, so they look incomplete or fast. Wait too long, and memory fills with traces that are still waiting. The waiting window is therefore a trade-off between accuracy and resource use, and it needs tuning to match how long your slowest legitimate requests take to finish.

Rate-limiting policies add one more consideration. Because they count spans per second, a trace with many spans uses up more of the allowance than a trace with few. Two systems with the same limit can therefore keep very different numbers of traces if their traces differ in size. When you convert between traces and spans, use the average span count from real measurements rather than a guess.

Picking the number is the hardest part, and it deserves real arithmetic rather than a guess.

Start from the budget, not the traffic

Begin with how much trace data your backend can store and how long you want to keep it. Estimate the average number of spans in a trace and the average size of a span, including its attributes. Multiplying these gives the typical size of one trace. Dividing your daily storage budget by that size tells you roughly how many traces per day you can afford, and dividing again by the seconds in a day gives a rate per second for the entire system.

Remember that limits are per process

An SDK sampler lives inside each application process and keeps its own credits. If you configure ten traces per second and run twenty instances, the whole fleet may produce up to two hundred traces per second. Forgetting this multiplication is the single most common reason teams are surprised by their tracing bill. Divide your system-wide target by the number of instances, and revisit the number whenever you scale out or in.

Autoscaling makes this trickier, because the number of instances changes with load. A fixed per-instance rate means the total grows exactly when you add instances during a spike, partially defeating the goal of a flat total. This is one reason some organisations prefer a collector-level limit for the total, and use the per-instance limit only as a coarse safety valve.

A worked example in words

Suppose a company keeps traces for fourteen days and can afford to store a certain number of gigabytes in that period. After measuring, the team finds that an average trace contains about thirty spans and that each span averages roughly one kilobyte once its attributes are included. A typical trace is therefore around thirty kilobytes. If the team can afford about two hundred gigabytes for the retention window, they can store roughly seven million traces over fourteen days. That is about five traces every second across the whole system.

Now suppose the entry services run on ten instances. Dividing the target of five traces per second by ten gives half a trace per second per instance. The team configures that fractional rate, which a credit-bucket sampler handles easily, and keeps a small headroom in case the number of instances grows. They then watch the real storage consumption for a week and adjust. The figures here are illustrative only; your own measurements will differ, and the method is what matters.

Revisit the number regularly

Traffic patterns, trace shapes and prices all change. A rate that was perfect in spring may be too generous or too stingy by autumn. Add the sampler rate to the list of settings that are reviewed on a schedule, for example every quarter or whenever a major release changes how many spans each request creates.

Every sampler is a compromise, and the rate limiter is no exception.

The bias problem in detail

A bucket refills over time, and requests arriving at the beginning of a busy burst find credits available while those arriving later find none. This means the sampled traces are not an even cross-section of the traffic; they favour whichever requests happened to arrive when credits existed. If a particular kind of request tends to arrive in bursts, it may be over-represented, while steady background requests may be under-represented.

Careful teams reduce this bias by using separate limits for different kinds of operations, by keeping the bucket small so that bursts do not dominate, or by moving the important decisions into a tail-based stage where errors and slow traces can be selected on purpose.

A fair summary of the trade-off is that a rate limiter protects your wallet, not your curiosity.

Reading the trade-offs as a decision

The disadvantages above are not reasons to avoid rate limiting. They are reasons to be clear about what you are buying. A rate limiter buys cost certainty and protection from surges, and it pays for them with less control over which traces survive. If your main fear is a large and unpredictable bill, or a telemetry pipeline that collapses under load, this is a good trade. If your main need is to study every failure in detail, you should add other mechanisms alongside it.

Per-operation limits as a compromise

Some implementations allow a separate rate for each named operation, plus a default for everything else. This softens the fairness problem, because a rarely used operation gets its own small bucket that busy operations cannot drain. The cost is extra configuration: someone must decide which operations deserve their own allowance, and the list must be kept up to date as the application changes. Even so, this arrangement is often the most practical middle path for a mature service.

Sampling is also a lever for safety, but only if you understand its limits.

Less data means less exposure

Traces can contain sensitive details, such as user identifiers, internal addresses or query fragments. Recording fewer traces naturally reduces the amount of such information stored. A rate limit can therefore be seen as one small layer of defence in depth, although it is never a substitute for properly scrubbing sensitive attributes before they leave the application.

Denial-of-wallet attacks

Attackers or faulty clients can flood a service with requests. With an always-on sampler, that flood is faithfully recorded, shipped and stored, which can overwhelm your telemetry pipeline or run up your bill. A rate limiter turns such a flood into a bounded, boring stream. In this sense it acts like a circuit breaker for your observability budget.

Blind spots can hide attacks

The same property has a downside. If an attack is mostly invisible to a limiter that is saturated by normal traffic, interesting malicious requests may be dropped. Security-relevant signals should therefore never depend solely on sampled traces. Use dedicated logs, metrics and audit trails that record every relevant event, and treat sampled traces as a debugging aid rather than a security record.

Protecting sensitive attributes before sampling matters

A common mistake is to think that dropped traces cannot leak anything, so there is no need to be careful about attributes. In reality, sampling decisions and data scrubbing are independent. Even sampled traces at a modest rate can carry personal data, tokens or secrets if the instrumentation records them. Remove or mask sensitive values at the source, using processors that run before data leaves the application or in a collector before it reaches storage. Think of the rate limiter as reducing how often a mistake is repeated, not as preventing the mistake.

Retention and access

The rate of collection is only one part of a data policy. How long traces are kept and who may read them matter just as much. A lower rate means less data to protect, which simplifies reviews and audits, but access controls on the backend still need to match the sensitivity of what is stored. Align your sampling rate, retention period and access rules as a single, documented decision.

A sampler that silently drops almost everything can be worse than no tracing at all, so watch it.

The most valuable habit is to keep a metric that counts every request before any sampling happens. Many teams generate request counts, error counts and latency histograms from the instrumentation itself, independent of whether a trace is stored. With that full-population view in hand, sampled traces become illustrative examples rather than the source of truth, and the limitations of rate limiting stop being a danger.

Alerts worth having

Sanity checks when something looks wrong

When engineers cannot find a trace they expect, the first question should be whether the request was ever sampled. Look at the sampled flag on any neighbouring traces, compare kept and dropped counts for the relevant time, and confirm that the missing request passed through a service that actually creates root spans. Many puzzling reports turn out to be nothing more than a dropped trace on a busy afternoon.

Teams that investigate often benefit from a small, deliberate escape hatch: a way to force sampling for a specific request, such as a debug header honoured at the entry point. This lets an engineer reproduce a problem and be sure the trace will be recorded regardless of the current state of the bucket. Such an override must be protected, since it bypasses the budget, but used carefully it removes much of the frustration of working with sampled data.

A few arrangements appear again and again in healthy systems, and a few mistakes appear just as often.

Helpful patterns

Harmful anti-patterns

A short checklist in plain language, drawn from everything above.

A rollout approach that reduces risk

Introducing a rate limiter into a running system is best done gently. Begin in a test or staging environment with a generous rate, and confirm that traces arrive whole and that the kept and dropped counts look sensible. Then move to a small slice of production, such as a single service or a single region, and compare the volume reaching the backend with your calculations. Only after this should you apply the setting widely.

Keep the old behaviour available for a while so you can switch back quickly if something looks wrong. Tell the engineers who use traces that the data is now sampled, and explain what that means: they will not find every request, and they should not use traces to count things. Communication prevents a surprising amount of confusion, because the most frequent complaint after adopting sampling is simply “I cannot find my trace”.

Some typical situations show how the choice plays out in practice.

graph LR
  A[Low steady traffic] --> B[Nearly all traces kept]
  C[Sudden traffic surge] --> D[Output capped at the ceiling]
  D --> E[Backend cost stays predictable]
  B --> E
        

Lessons that repeat across scenarios

Across these examples, the same lessons appear. Successful teams decide the budget first and the rate second. They keep metrics for every request. They make sure only entry points decide. They give themselves a way to raise visibility temporarily during an incident, and they review their settings from time to time. Unsuccessful teams usually skip one of these steps, and the symptoms are broken traces, misleading charts or unexpected bills.

Short, plain answers to the questions people ask most often.

Not as one of the four standard samplers. Rate limiting is offered through extensions such as the Jaeger remote sampler packages, through collector policies, and through vendor or custom samplers. Check the documentation for your language and version to see what is available.

An SDK-level limiter counts traces, because it decides once per root span. A collector policy typically counts spans. Always check the unit before doing capacity maths.

The application skips most of the work of recording it, the sampled flag is left unset, and downstream services that follow the parent also skip recording. Nothing is exported for that trace.

No. A head-based limiter decides before the outcome is known. To keep errors reliably, add tail-based policies in a collector, and also record error counts as metrics.

With an SDK sampler, no. Each process keeps its own credits, so the fleet total is roughly the per-instance rate multiplied by the number of instances. A collector-level policy can enforce a shared total instead.

Choose rate limiting when predictable cost matters most, and a percentage when proportional statistics matter most. Many teams use both, with a percentage for thinning and a rate limit as a hard cap.

Each stored trace is still accurate, but the collection of stored traces is a partial picture. Use metrics for counts and rates, and traces for understanding individual requests.

A common cause is services making independent decisions instead of following the parent. Wrap the sampler in a parent-based sampler and make sure context propagation headers are passed between every service.

Some implementations, notably remote samplers that fetch their strategy from a central service, can update the rate while running. Others read their settings at startup. Your SDK documentation tells you which applies.

Not necessarily. A larger bucket tolerates bursts but allows short spikes above the average rate, while a small bucket gives smoother output but drops more of a sudden burst.

Very little. The application still evaluates the sampler and creates a lightweight placeholder so that surrounding code works, but it skips building attributes, events and exports for that trace. The cost of a dropped trace is a tiny fraction of the cost of a recorded one.

Not noticeably. The check is a small piece of arithmetic on a shared counter. In fact, by reducing the number of spans recorded and exported during busy periods, it usually lowers the overall overhead of tracing compared with recording everything.

Exclude them before they reach the limiter. They are frequent and uninteresting, and if they spend credits, real user requests will be dropped in their place.

Prepare a higher rate or a forced-sampling mechanism in advance, keep error counts in metrics, and use tail-based policies in a collector to retain failed and slow traces. Raise the rate temporarily during the incident and return it to normal afterwards.

The idea of capping output per second applies to any telemetry, but the tools differ. Metrics are usually small and aggregated, so they rarely need it. Logs sometimes use their own rate limits. The sampler described in this guide applies to traces only.

Usually not much. The sampler is configured when the tracing system is set up, and the instrumentation that creates spans stays the same. The change is in configuration, not in the business logic of the application.

A rate-limiting sampler answers one question with great clarity: “How many traces can we afford each second?” It sets a ceiling, spends one credit per new trace, and drops whatever exceeds the budget. That makes costs predictable and protects both your application and your backend during traffic surges, which is exactly when a percentage-based approach would grow most. The price is paid in representativeness, because the sampler decides early, cannot favour failures, and makes statistics harder to compute.

Used well, it is one layer in a larger design: a parent-based wrapper to keep traces whole, full-population metrics for accurate numbers, and collector policies to preserve the traces that matter most. Used carelessly, it produces broken traces, surprising invoices and misleading dashboards. Understand its limits, calculate its rate from your budget, and keep watching how it behaves.