Three ordinary words hide one big idea. Let us take them apart one at a time.

Sampling: keeping a part instead of the whole

Sampling means choosing a small portion of something to represent the whole. A cook tastes one spoon of soup to judge the whole pot. A pollster asks a few thousand people instead of a whole country. In tracing, sampling means saving only some traces and discarding the rest.

The trade is simple. You give up the ability to look at every single request, and in return you pay much less for sending, storing, and searching data. A good sample still tells you the truth about your system. A bad sample hides problems or tells lies.

Cost-aware: knowing the price of every choice

Many teams pick a sampling rate by feeling. They say “let us keep ten percent” because ten sounds like a nice round number. Nobody checks what that costs, and nobody checks whether the right traces survived. Cost-aware design flips this. You begin with a budget, an amount of money or data you are willing to spend each day. Then you decide how to spend that budget so that every dollar buys the most useful traces.

Strategy design: a plan, not a single setting

A strategy is a full plan that answers several questions together. Which traces are worth always keeping? Which are noise that should be dropped immediately? How much of the ordinary, healthy traffic should we keep as a baseline? What happens when traffic doubles, or when a big outage causes a flood of errors? Who can change the plan, and how do we know the plan is working? Design means answering these questions on purpose, writing the answers down, and checking them with real numbers.

Before we choose which traces to keep, we need to know exactly what a trace is and how it travels.

Trace and span

A trace is the full story of one request. When a shopper presses the “Pay” button, that single press may touch a web server, a cart service, a payment service, a fraud checker, a database, and a message queue. The trace records the whole journey from start to finish.

A trace is made of spans. A span is one step in the journey, such as “the payment service called the bank” or “the database ran a query”. Each span has a start time, an end time, a name, and some labels called attributes, for example the name of the service or whether the step succeeded. Spans are linked in a tree: a parent span can have many child spans, and together they form the trace.

Context propagation: carrying the parcel number

How does the cart service tell the payment service that they belong to the same trace? It passes a small piece of information along with every call. This is called context propagation. The most common format is the W3C Trace Context standard, which uses an HTTP header named traceparent. That header carries four things: a version number, the trace ID, the ID of the calling span (the parent), and a small set of flags.

One flag is especially important for this guide: the sampled flag. It says “the first service decided to keep this trace, so everyone downstream should keep it too”. If every service respected this flag, then either the whole trace is kept or the whole trace is dropped. If services ignore it and each flip their own coin, you end up with traces that have holes in them. We will come back to this many times.

The pieces of OpenTelemetry

Sampling can happen in two places: inside the SDK in your application, and inside the Collector. That single fact shapes the whole design. Deciding early (in the SDK) is cheap but blind. Deciding late (in the Collector) is smart but needs more machines. The next chapters explain why.

You cannot control a cost you cannot see. Let us look at every place a trace costs something.

Many people think the only cost of tracing is the monthly invoice from a vendor. In truth, a trace costs money at every stop on its journey. Cost-aware design starts by listing those stops.

A worked example with simple numbers

Let us build a small, made-up platform and calculate its trace volume. These numbers are for teaching only. They are not anyone’s real price list.

• The platform handles 5,000 requests per second on average.
• Each request becomes one trace with about 40 spans.
• Each span averages about 500 bytes, so one trace is about 20 kilobytes.

In one day there are 86,400 seconds. That gives 432 million traces, or about 17.3 billion spans. At 20 kilobytes per trace, the total is about 8,640 gigabytes (8.64 terabytes) per day if you keep everything. Now suppose a backend charges an illustrative ten cents per gigabyte ingested.

Look at the last two rows. The smart mix costs about three times more than a plain one-percent sample, yet it is still more than thirty times cheaper than keeping everything. And it keeps the traces that actually explain failures. This is the heart of cost-aware design: you do not simply chase the lowest bill. You chase the best value per byte.

The ideas in modern OpenTelemetry did not appear overnight. Each one answered a pain someone felt before.

Notice the direction of travel. The field moved from “keep a random slice” to “keep a slice that is chosen with a purpose, controlled from the center, and measurable afterwards”. Cost-aware sampling is where that road leads.

A handful of ideas explain almost everything about how sampling behaves. Learn these and the rest feels easy.

Sampling probability and sampling rate

The probability is the chance that a given trace is kept. A probability of 0.01 means one trace in a hundred. People also say “a rate of one percent” or “one in a hundred”. They all mean the same thing. Some systems use the word rate for traces per second instead, so always check which one a tool means.

Adjusted count: giving each kept trace a weight

If you keep one trace in a hundred, then each kept trace stands in for about a hundred. That number is the adjusted count, simply one divided by the probability. When you later count “how many checkout requests happened”, you add up the adjusted counts instead of the raw number of stored traces. Without this step, your charts would shrink every time you lowered the sampling rate, and nobody could trust them.

Determinism and consistency

A sampler is deterministic when the same trace ID always gets the same answer. This is usually done by turning the trace ID into a number and keeping the trace if the number falls below a threshold. Because every service sees the same trace ID, they all reach the same decision without talking to each other. A sampler is consistent when a trace kept at a low rate is also kept at every higher rate. This lets different parts of a system use different rates and still agree about which traces form a complete set.

Parent-based decisions

A parent-based sampler looks at the incoming request. If a parent span already decided to keep the trace, it keeps its part too. If the parent decided to drop, it drops. Only when there is no parent does it make its own decision. This is what keeps traces whole across many services.

Recording versus sampling

These two words are easy to mix up. A span that is recorded exists in memory and can collect attributes. A span that is sampled is also marked to be exported. A span can be recorded but not sampled, which is useful when something local, such as a metric, wants to look at it even though the trace will not be saved.

Stratification: separate pots for separate kinds of traffic

If you sample everything at one percent, a rarely used but important endpoint with ten calls a day might never appear. Stratified sampling gives each kind of traffic its own rule, for example “always keep refund requests, keep 0.1 percent of health checks, keep 5 percent of search”. Cost-aware design relies on this heavily.

This is the biggest fork in the road. The difference is simply when the decision is made.

flowchart LR
  subgraph H["Head sampling: decide at the start"]
    A1["Request arrives"] --> A2["SDK sampler decides"]
    A2 -->|"keep"| A3["Record all spans"]
    A2 -->|"drop"| A4["Create no data"]
    A3 --> A5["Export to backend"]
  end
  subgraph T["Tail sampling: decide at the end"]
    B1["Request arrives"] --> B2["Record all spans"]
    B2 --> B3["Collector buffers the whole trace"]
    B3 --> B4{"Do policies match"}
    B4 -->|"yes"| B5["Export to backend"]
    B4 -->|"no"| B6["Discard the trace"]
  end
    

In Figure 1, the top path shows head sampling. The decision happens the instant a request begins. If the answer is “drop”, the application barely does any work, because no spans need to be built or sent. The bottom path shows tail sampling. Every span is created and sent to a Collector, which holds the pieces of a trace in memory until the trace looks finished. Only then do the policies decide whether to keep it.

The samplers that ship with OpenTelemetry SDKs

Every OpenTelemetry SDK includes a small family of built-in samplers. They are chosen through code or through simple settings such as OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG.

• Always on keeps every trace. Good for development, expensive in production.
• Always off keeps none. Useful to switch tracing off for one service.
• Trace ID ratio keeps a fixed fraction, decided by the trace ID so that all services agree.
• Parent-based wraps another sampler and obeys the parent’s decision when one exists. The popular production choice is parent-based wrapped around a trace ID ratio sampler.

Side by side

Tail sampling feels like magic from the outside. Inside, it is a patient bookkeeper with a very big desk.

The step-by-step life of a trace inside the sampler

The tail sampling processor in the OpenTelemetry Collector works roughly like this. Spans arrive one at a time and from many services. The processor looks at the trace ID on each span and files the span into a bucket for that trace, held in memory. When the first span of a new trace shows up, a timer starts. The processor then waits for a configured period, called the decision wait, which defaults to thirty seconds. When the wait is over, it runs the trace through its policies. If the policies say “keep”, all the spans of that trace are passed on to the next step. If not, they are thrown away and the memory is freed.

The policies you can combine

One rule of thumb is worth memorizing: by default, a trace is kept if any policy says “keep”. The policies behave like a list of reasons to save a trace. If you want “keep only when both A and B are true”, you use an explicit “and” policy. Newer versions add extra policies and options, so always read the release notes for the version you run.

Memory: the number that surprises everyone

Tail sampling holds traces in memory, so memory sets the limit. Let us use the platform from Chapter 3, which creates about 5,000 new traces every second, each about 20 kilobytes on the wire. With a thirty-second wait, about 150,000 traces are waiting at any moment. That is roughly 3 gigabytes of data as it travels on the network. Objects held in memory are usually bigger than the same data on the wire, so a careful designer plans for two to three times that amount, which means something like 6 to 9 gigabytes spread across several machines.

The processor also has a cap on the number of traces it holds, which defaults to 50,000. If more traces arrive than the cap allows, the oldest ones are pushed out before their time is up. These are decided too early, usually with too little information. A well-run system watches for this and sizes the cap and the machines to match real traffic, as Chapter 15 explains.

Late spans and the decision cache

Some spans arrive after the decision has been made, perhaps because a service was slow or a message queue delayed them. Without help, these late spans would look like the start of a brand new trace and be judged all alone. To prevent this, the processor can remember recent decisions in a decision cache. A late span for a trace that was already kept is kept too, and a late span for a dropped trace is dropped. This keeps results tidy without keeping everything in memory forever.

One trace, one machine

There is a rule that cannot be bent: all spans of one trace must reach the same tail sampling Collector. If half the spans go to machine A and half to machine B, neither can see the whole story. The usual solution is a first tier of Collectors that use the load balancing exporter. This exporter looks at the trace ID of every span and uses a hashing method to choose the destination machine, so the same trace ID always lands on the same machine. It can discover its destinations from a fixed list, a DNS name, or the Kubernetes service list.

Now we put all the pieces together into one picture that works for millions of requests per minute.

flowchart TD
  APP["Applications with OpenTelemetry SDK"] -->|"OTLP"| AGT["Agent Collector on each node"]
  AGT --> GW["Gateway Collector tier"]
  GW --> SM["Span metrics connector"]
  SM --> MET["Metrics backend"]
  GW --> LB["Load balancing exporter using trace ID"]
  LB --> TS1["Tail sampling Collector 1"]
  LB --> TS2["Tail sampling Collector 2"]
  LB --> TS3["Tail sampling Collector 3"]
  TS1 --> HOT["Hot trace backend"]
  TS2 --> HOT
  TS3 --> HOT
  GW -.->|"optional raw copy"| COLD["Cold archive in object storage"]
  CTRL["Budget controller and shared config"] -.-> TS1
  CTRL -.-> TS2
  CTRL -.-> TS3
    

Read Figure 2 from the top. Applications send spans to a small agent Collector running next to them. The agent forwards to a gateway tier. At the gateway, two things happen in parallel. First, a span metrics connector turns spans into counts and timings, so your dashboards see all traffic. Second, spans flow into the load balancing exporter, which routes each trace to one of the tail sampling Collectors. Those Collectors apply the policies and send the survivors to the hot trace backend. An optional side path stores a compressed raw copy in cheap object storage. A budget controller adjusts shared settings.

Why two Collector tiers instead of one

A single tier cannot be both a friendly, easy-to-scale receiver and a stateful sampler that needs every span of a trace in one place. Splitting them gives each its own job. The gateway can add and remove machines freely because it holds nothing important in memory for long. The sampling tier is stateful, so it is scaled with more care. This is the same reason big shops separate a “front desk” from a “records room”.

Why take metrics before sampling

If you computed request counts and error rates only from the traces you kept, your charts would be wrong, because you deliberately kept an unfair share of errors. By creating metrics from every span before sampling, your dashboards stay true to the real traffic. Chapter 12 covers this in depth.

A sampling strategy is a set of answers to the question “which traces are worth their price?”

Sort traces into value tiers

The design recipe

The decision flow

flowchart TD
  S["Trace complete or wait time over"] --> N{"Is it noise such as a health check"}
  N -->|"yes"| D1["Drop"]
  N -->|"no"| E{"Does it contain an error"}
  E -->|"yes"| Q1{"Error budget left"}
  Q1 -->|"yes"| K1["Keep"]
  Q1 -->|"no"| R1["Keep a small share only"]
  E -->|"no"| L{"Slower than the latency limit"}
  L -->|"yes"| K2["Keep within the slow budget"]
  L -->|"no"| V{"Business critical route"}
  V -->|"yes"| K3["Keep at a higher percentage"]
  V -->|"no"| B["Keep at the baseline percentage"]
    

Figure 3 shows the order of questions. Noise is thrown away first because it has no value at any price. Errors come next because they are the most valuable, but even they have a budget, since a bad outage can create a flood of failures. Slow requests follow. Business-critical routes get a higher keep rate than ordinary traffic. Everything else lands in the baseline.

A budget is only real if something enforces it. This chapter shows the tools that do the enforcing.

When the wishes are bigger than the budget

Return to our made-up platform. Suppose the daily budget is 150 gigabytes of kept traces. Our wishes, using one percent of errors, one percent of slow traces, a little for critical routes, and one percent baseline, add up to about 278 gigabytes. They do not fit. A budget forces choices, and good design makes those choices on purpose:

The total is 150 gigabytes. Notice that no bucket was removed. Each was trimmed in the way that hurts least. Repeated copies of the same error add little knowledge, so trimming them is cheap in information terms.

Reservoir plus percentage

AWS X-Ray popularized a clean pattern: a reservoir plus a fixed rate. In the default rule, one request per second is always traced, and a small percentage of the requests beyond that are also traced. The reservoir guarantees that even quiet endpoints produce some traces. The percentage keeps volume under control on busy ones. You can copy this shape with the OpenTelemetry Collector by combining a rate-limited policy with a probabilistic one.

Splitting one budget across many rules

The composite policy lets you set a maximum number of spans per second for the whole group and then hand out shares by percentage to sub-rules. For example, “forty percent for errors, thirty percent for slow traces, thirty percent for the rest”. When a rule does not use its whole share, the unused part can flow to the others. This is the Collector’s most direct form of a budget.

Error storms: the cost spike that arrives with the outage

Here is the most important trap in cost-aware design. If your rule says “keep every error”, then during a big outage the number of errors can jump twentyfold. Your bill spikes at exactly the moment everyone is already stressed. Worse, a flood of near-identical failures teaches you almost nothing new. The fix is to give the error bucket a ceiling and a smart way of spending it: keep the first few examples of each kind of failure, then thin out the repeats. The Collector’s built-in policies mainly cap total rates. Per-kind throttling is the strength of dynamic samplers such as those in Honeycomb’s Refinery, which adjust the keep rate for each group of similar events so that rare ones are kept and common ones are thinned.

Per-team budgets and showback

In a company with many teams, a single shared budget invites a tug of war. The loudest team wins, and quiet teams lose their traces. A fairer habit is to give each team a share of the budget, based on traffic and importance, and to show every team its own usage every week. This is called showback, because you show the cost without charging it. When teams can see that one chatty loop in their code costs more than their whole error budget, they usually fix it without being asked. Some companies go one step further to chargeback, where the cost is moved to the team’s own budget. Both depend on one simple thing: every span must carry a team label, so that usage can be added up per owner.

The feedback loop

flowchart LR
  M["Measure bytes kept per hour"] --> C{"Compare with the budget line"}
  C -->|"over budget"| DN["Lower the baseline percentage"]
  C -->|"under budget"| UP["Raise the baseline percentage"]
  C -->|"on target"| HOLD["Hold steady"]
  DN --> APPLY["Push new settings to Collectors"]
  UP --> APPLY
  HOLD --> M
  APPLY --> M
    

Two design rules keep such a loop safe. First, move only the baseline, never the must-keep rules, so that failures stay visible. Second, change settings in small steps and not too often, because tail sampling holds state and wild swings confuse the people reading charts. Settings can be pushed with a Collector management protocol such as OpAMP, or with plain configuration files delivered by your deployment system.

Follow a single checkout request from the shopper’s tap to long-term storage, and watch where each cost is saved.

What to shed first when the pipeline is crowded

Even a well-sized pipeline meets a day when traffic is far above plan. It helps to decide in advance, calmly, what gets sacrificed first, rather than letting the machines decide at random. A sensible order is this. First, shrink the baseline of healthy traffic toward zero, since it is the least valuable. Second, thin out repeated copies of the same error. Third, shorten the slow-trace bucket. Last of all, and only in an emergency, touch the rare, unique failures and the critical routes. Writing this ladder down turns a stressful night into a checklist, and it keeps the most precious traces the longest.

What happens when the pipeline is under pressure

Telemetry systems almost always choose availability over perfection. If the sampling tier is overloaded, it is better to shed some telemetry than to slow down the shopper’s checkout. That is why the SDK exporter has a size-limited queue and drops data when the queue is full, and why Collectors use a memory limiter that refuses new data when memory is nearly full. Dropped telemetry under pressure is not elegant, but it protects the real business. Your job is to make this rare, and to notice when it happens, as Chapter 15 describes.

Sampling saves money, but it can quietly make your charts lie. Here is how to stop that.

Why biased samples break naive counting

Our smart strategy keeps almost every error and only a thin slice of healthy traffic. If someone counts the stored traces and says “forty percent of requests fail”, they are badly wrong. The sample is deliberately slanted. Counting kept traces answers only the question “what do the kept traces look like?” and never the question “how is the whole system doing?”.

Three ways to keep counts true

The cost of metrics from spans

Span metrics are not free. Every different combination of labels, such as service, route, status, and customer tier, creates a separate time series, and many backends charge per series. If you add a label with thousands of different values, such as a user ID, the number of series explodes and the bill can be larger than the traces you saved. Choose labels with a small, fixed set of values, and keep high-detail questions for the traces themselves.

Logs that follow their traces

Logs can also be sampled, and a nice trick is to base the choice on the trace ID. If a trace is kept, its logs are kept, and if it is dropped, its logs are dropped. The probabilistic sampler in the Collector supports this for logs, so a kept trace always comes with its log lines and nothing is half-missing. Make sure the same hashing settings are used across tiers, otherwise the two will disagree.

Every strategy buys something and gives something up. Knowing the exchange rate is half of good design.

Comparing the common strategies

Trade-offs worth discussing out loud

• Delay versus quality. A longer decision wait gives better decisions for slow or branching traces, but costs more memory and delays when a trace becomes searchable.
• Early cuts versus smart cuts. Head sampling saves compute and network but cannot see errors. Tail sampling sees everything but pays for it first.
• Hot storage versus cold storage. Cheap object storage makes it affordable to keep much more raw data, which may let you sample less. But reading it later is slow, and the processing hops still cost money.
• Simplicity versus precision. Each extra rule is one more thing to test, explain, and keep working when people leave the team.

A sampling system looks harmless, but it touches every request and often every secret that rides inside one.

Do not let strangers control your bill

Remember the sampled flag from Chapter 2? It travels in a header that the caller sends. If your public gateway trusts that header, anyone on the internet can send “sampled” on every request and force you to keep one hundred percent of their traffic. Even a small script could multiply your telemetry cost. This is a cost attack, and it is easy to prevent: at the edge, ignore or overwrite the incoming trace headers from untrusted callers and start a new trace with your own decision. For partners you trust, use an allow list.

flowchart LR
  EXT["Outside clients"] -->|"may send trace headers"| EDGE["Edge gateway"]
  PART["Trusted partner"] -->|"allow listed"| EDGE
  EDGE -->|"untrusted headers removed, new trace started"| SVC["Internal services"]
  SVC --> COL["Collectors inside the trust boundary"]
    

The sampling tier sees everything

A tail sampling Collector holds full traces in memory, including every attribute. If spans contain email addresses, account numbers, or tokens, that machine is now a place where sensitive data sits for thirty seconds or more. Treat it like a database server: restrict who can log in, encrypt traffic between tiers with TLS, and keep these machines inside your private network.

Remove secrets before they spread

Best practice is to scrub sensitive attributes as early as possible, ideally in the application’s instrumentation and again at the agent or gateway. Collector processors can delete or hash chosen attributes. Doing this before sampling means that even the traces you keep, and the metrics built from them, do not carry secrets. It also makes your policies safer, because rules cannot accidentally depend on private values.

Compliance and retention

Traces are often treated as personal data under privacy laws when they contain user identifiers. Sampling reduces the amount stored, but does not remove the duty. Keep retention periods short for ordinary traces and longer only for a small set kept for investigations. Never use traces as your audit trail. An audit trail must be complete and tamper-evident, while sampled traces are neither.

A pipeline that watches everything else needs someone watching it. Otherwise it fails silently, and you discover the gap during the outage.

The OpenTelemetry Collector publishes its own metrics about how much data it received, sent, refused, and dropped. The exact metric names change slightly between versions, so treat the table below as the set of questions to ask, then find the matching metric in your version.

Alerts that matter

• Budget burn rate. Alert when today’s volume is on track to exceed the daily budget, early enough that someone can act, such as at the halfway point of the day.
• Error bucket exhausted. If the error cap runs out before the day ends, you are no longer seeing all kinds of failures. Page someone to review.
• Memory pressure. High memory use in the sampling tier means traces will start being dropped.
• Silence. A sudden drop in kept traces is as worrying as a spike. It may mean a broken route or a changed service name.

A canary trace for peace of mind

One of the best checks is a canary trace. A small scheduled job sends a synthetic request that is guaranteed to look like an error. Every few minutes you check that this trace arrives in the backend. If it does not, then something in the pipeline between the application and the store is broken, and you learn about it before real users do.

Patterns are solutions that many teams discovered the hard way. Anti-patterns are the traps they fell into.

Patterns that work

Anti-patterns to avoid

A short checklist of habits, followed by the slips that appear most often in real teams.

Best practices

• Start with the budget, then derive rates. Never pick a percentage first and discover its cost later.
• Standardize resource labels. Service name, environment, team, and version on every span make per-team budgets and targeted rules possible.
• Keep the policy list short and readable. A handful of clear rules beat dozens of overlapping ones.
• Set decision wait from data. Look at how long your longest ordinary traces really take, then add a safety margin.
• Size memory for peak, not average. Traffic spikes arrive when you least want the pipeline to drop data.
• Test changes on a copy of real traffic first. Compare predicted and actual volumes before promoting a rule.
• Roll out gradually. Change one tier or one team at a time, and watch the pipeline metrics.
• Explain the sampling to your users. Engineers who understand why a trace is missing do not waste hours hunting for it.

Common mistakes and quick fixes

The ideas in this guide were shaped by real systems. Here is what the public record shows, plus one clearly labeled made-up story.

The architecture is the same in any cloud. Here is how its parts map onto common AWS building blocks.

Where cloud cost sneaks in

Cloud providers usually charge for data moving between availability zones. In a tail sampling design, spans from every zone must travel to the machine that owns their trace, so a large share of the traffic crosses zones by necessity. The best defenses are to cut before the hop (drop noise, trim large attributes), compress the data, and avoid sending data that will be thrown away anyway. Check your own price list, because these charges differ by service and region.

Keep the stateful tier calm

• Avoid interruptible machines such as spot instances for the sampling tier. When a machine disappears, every trace in its memory disappears too.
• Drain before removing. Give machines time to finish decisions on the traces they hold before they shut down.
• Limit disruptions. Use settings that keep a minimum number of sampling pods available during upgrades and node replacements.
• Scale on memory and traces held, not only on CPU, since memory is the real limit.

Sampling looks like a simple yes-or-no choice. Behind that choice sit a few clever, old ideas from computer science.

Hash and threshold: choosing without talking

How can fifty services agree on which traces to keep without sending each other messages? They all read the same trace ID, turn part of it into a number, and compare that number with a cutoff. If the number is below the cutoff, the trace is kept. For a one percent rate, the cutoff is set so that one percent of all possible numbers fall below it. Because every service sees the same ID and uses the same cutoff, they reach the same answer. This is called deterministic sampling.

This trick only works fairly if trace IDs are truly random. Some older ID schemes put a timestamp at the front of the ID, as AWS X-Ray’s format does. A sampler that looked only at that front part would treat traces from different moments unequally. Samplers therefore read the random-looking part of the ID, and you should make sure that any custom ID generator you use produces enough randomness.

Token bucket: the coin jar for rate limits

A rate limit such as “keep at most 100 traces per second” is often built with a token bucket. Imagine a jar that receives a new coin every hundredth of a second, up to a maximum. Each trace you keep costs one coin. When the jar is empty, you must drop. A short burst can be handled with the saved-up coins, but the long-term average can never exceed the refill speed. This is how a reservoir plus a ceiling behaves in practice.

Reservoir sampling: a fair pick from an endless stream

What if you want exactly 100 traces per minute, chosen fairly, without knowing in advance how many will arrive? A classic method called reservoir sampling solves it. Keep the first 100 traces. For the next trace, number 101, flip a weighted coin: with a chance of 100 out of 101, throw it in and replace a randomly chosen trace already in the reservoir. For trace 102, the chance is 100 out of 102, and so on. At the end of the minute, every trace that arrived had exactly the same chance of being in the reservoir. The method needs only as much memory as the number of traces you want to keep.

Time buckets: expiring traces without scanning everything

A tail sampler must act on each trace once its waiting time ends. If it checked every trace in memory every second, it would waste a lot of effort. Instead it groups traces by the second in which they first appeared, in a ring of buckets as long as the decision wait. Every second the ring turns, and only the oldest bucket is examined. The work done is proportional to the number of traces that are due, not to the number held. This conceptual picture matches how the processor behaves, though the details differ between versions.

flowchart LR
  IN["Incoming spans"] --> MAP["Map keyed by trace ID"]
  MAP --> RING["Ring of one second buckets"]
  RING -->|"oldest bucket expires"| EVAL["Evaluate policies"]
  EVAL -->|"keep"| OUT["Forward spans"]
  EVAL -->|"drop"| FREE["Free the memory"]
  EVAL --> CACHE["Decision cache for late spans"]
    

Consistent hashing: seats around a round table

The load balancing exporter has to decide which sampling machine owns each trace ID. A simple method, “take the ID number and divide by the number of machines, keep the remainder”, reshuffles almost every trace whenever a machine is added or removed. A consistent hashing ring avoids this. Picture machines sitting at fixed seats around a round table, with each trace ID landing at a spot on the table and belonging to the next machine clockwise. When a new machine sits down, it takes over only the spots between itself and its neighbor, and the rest stay put. With ten machines, adding an eleventh moves roughly one eleventh of the traces, not nearly all of them.

Concurrency and back pressure

Many network threads write spans into memory at once. Spans from different traces rarely collide, but a single enormous trace with thousands of spans can become a hot spot where many threads fight for the same record. Good systems limit the number of spans kept per trace, evaluate policies on a pool of workers, and refuse new data politely when memory is nearly full. This refusal is called back pressure. It passes the “slow down” signal back up the chain, so each tier protects itself instead of collapsing.

Numbers turn a diagram into a plan. Let us size the pipeline for our made-up platform, step by step.

We keep the same platform as before, with 5,000 traces per second on average, 40 spans per trace, and 500 bytes per span. We also add three new assumptions that you must measure in your own system: peak traffic is twice the average, one sampling machine can handle about 50,000 spans per second, and compression shrinks data to about a quarter of its size.

Eight machines, each with room for several gigabytes, would handle this peak. A careful team would round up for safety, perhaps choosing machines with 8 gigabytes each, and would test the guess with real load.

The surprise: the hop can cost more than the storage

Now look at the zone-crossing traffic. Suppose the pipeline runs in three availability zones. A span’s trace is owned by a sampling machine in a random zone, so about two thirds of spans must cross a zone border. On an average day, the compressed stream is about 25 megabytes per second, and two thirds of it crossing zones is about 17 megabytes per second, or about 1,440 gigabytes per day. If we assume a made-up combined charge of two cents per gigabyte for crossing zones, that is roughly $29 per day.

Compare that with the hot backend from Chapter 10. At 150 gigabytes per day and our made-up ten cents per gigabyte, ingestion costs $15 per day. In this example, the network hop costs about twice as much as the backend ingest after sampling. The lesson is not that these exact numbers are right. The lesson is that sampling the backend is not the only big line on the bill, so a complete design also trims data before it travels.

Sampling removes whole traces. Several other levers shrink each trace, and they often save more with less risk.

Be careful with span-level dropping

It is tempting to keep the trace but drop low-value child spans from the middle of it. The danger is that the surviving spans point to parents that no longer exist, and tracing tools may show broken trees or float the orphans to the top. Unless your tooling clearly supports re-attaching children to a surviving ancestor, prefer removing spans at the source by turning off the noisy instrumentation, which keeps each tree whole.

Spans versus events

A span can carry small notes called events, such as an exception message with its stack trace. Stack traces can be large, and a busy failing service may attach one to every failed call. Limit how many events and how long each may be, and keep the full details in logs that are linked to the trace by its ID. This shifts bulk text to the system that is cheapest at storing it.

What happens when different services, or different stages, sample at different rates, and you still want correct counts?

The problem

Imagine the front service keeps ten percent of traces and a downstream billing service keeps one percent. If each makes an independent random choice, then the billing service’s kept traces do not line up with the front service’s kept traces. You get fragments. Now imagine a head sample at ten percent followed by a tail stage that keeps half of what it sees. The overall chance that a healthy trace survives is five percent, and anyone counting later has to know that.

The idea: one luck number per trace

This property is what the word consistent means in this setting. It lets teams choose different rates for different needs without breaking traces apart.

Passing the cutoff along

To count correctly later, a downstream tool needs to know the cutoff that was used. The OpenTelemetry community has been defining how to record the sampling threshold inside the trace state, which travels with the trace. With this information, each kept span can say “I stand for this many spans” and a backend can add up correct totals even when different parts of the system used different rates, or when the rate changed during the day.

What to do while support is uneven

Support for these features varies between languages, SDK versions, Collector versions, and backends, and parts of the specification are still maturing. Check what your own versions offer before relying on it. If you cannot rely on it, a simple and safe fallback exists. Use one rule per bucket, as in Chapter 10, and store the keep probability for that bucket as an attribute on every kept trace’s root span. Then count by adding the weight of each kept trace, which gives you the same result with less elegance.

Short answers to the questions people ask most often.