Before learning how, let us be clear about why this question deserves a careful answer.

Observability tools exist to help you understand your system. But they run inside your system, using the same processors, the same memory and the same network as the program they are watching. Physicists call this the observer effect: the act of measuring can change what is being measured. In software, the effect is usually tiny, yet it is never exactly zero, and it can grow in surprising ways when instrumentation is added carelessly to the busiest parts of a program.

There are three good reasons to measure rather than assume. The first is trust: if you cannot say how much your telemetry costs, people who care about performance will, quite reasonably, resist adopting it. The second is cost control: processor time, memory and network traffic are paid for, and overhead multiplied by thousands of servers becomes real money. The third is safety: a badly configured instrumentation setup can cause slowdowns or memory pressure during exactly the moments when the system is already under stress.

Each term below comes with what it is, why it matters for this topic, and a picture to hold in your head.

To measure a cost, you must know where it hides, so let us follow a request through an instrumented program.

Instrumentation work happens in several places. Some of it happens on the thread that is serving the customer, and that part adds directly to latency. Some happens on background threads, and that part adds to processor and memory use but not directly to latency. Some happens on the network and in other programs, such as the Collector. A complete measurement looks at all of these.

Every sound overhead measurement is a variation on one simple experiment.

Everything else in this tutorial adds detail to those five steps: how to build the ladder of variants, how to generate realistic load, how to read the numbers, how to measure noise, and what to do with the answer. Keep the five-step shape in mind and the details will fall into place.

A cost can show up as slowness, as lost capacity, as extra processor use, as memory growth or as network traffic, and each needs its own measurement.

A number means nothing until you know what number would be too much.

Many overhead investigations go in circles because nobody agreed in advance what counts as acceptable. The measurement shows that processor use is up six percent, and one person calls that negligible while another calls it a scandal. The cure is to write the budget down before you run the test. A budget turns an argument about opinions into a check against a target.

A good budget has a few lines, each tied to something the business cares about. For a customer-facing service, the most important line is usually the slow tail of latency, because slow requests are what customers complain about. For a service that is cost-sensitive, processor and memory lines matter more. For a service with a strict capacity plan, the throughput line matters.

Most bad overhead numbers come from unfair comparisons, not from bad tools.

A fair experiment changes only the thing you are studying. In practice, many other things can change by accident, and each one can hide or exaggerate the effect of instrumentation. A short checklist of controlled factors prevents most of these accidents.

Comparing only the two extremes tells you that there is overhead, but not where it comes from; a ladder of variants tells you both.

Instead of two runs, design a ladder of several. Each step adds exactly one more piece of the instrumentation stack, so that the difference between neighbouring steps is the cost of that one piece. This is the single most useful technique for turning a vague “it seems slower” into a precise “this particular layer costs this much”.

flowchart TD
    A["Step 0: no instrumentation at all"] --> B["Step 1: API only, no SDK installed"]
    B --> C["Step 2: SDK installed, sampler drops everything"]
    C --> D["Step 3: SDK samples, spans created, no export"]
    D --> E["Step 4: spans exported to a local fast sink"]
    E --> F["Step 5: spans exported to the real Collector"]
    F --> G["Step 6: full production configuration"]
        

The traffic you send decides what you learn, so it must resemble the traffic your system really sees.

Instrumentation overhead depends heavily on what the application is doing. A request that makes a single quick database call creates a handful of spans. A request that fans out to twenty services creates dozens. If your test sends only simple requests, you will measure a very light overhead and be surprised in production. A good workload reflects the real mix: the common requests, the heavy ones, and the rare expensive ones.

Open loop versus closed loop

Load generators work in one of two ways. A closed loop generator has a fixed number of virtual users, and each user waits for its response before sending the next request. If the system slows down, the generator automatically sends fewer requests, which hides the slowdown. An open loop generator sends requests on a fixed schedule, regardless of whether earlier ones have finished, just as real customers do. Open loop generators give a much more honest picture of latency and of how the system behaves near its limits.

flowchart LR
    subgraph CL["Closed loop"]
        U1["Virtual user sends request"] --> W1["Waits for response"] --> U1
    end
    subgraph OL["Open loop"]
        T["Timer fires on schedule"] --> S["Sends request"] --> T
        S --> R["Response arrives whenever it arrives"]
    end
        

Averages hide the very requests that matter most, so latency must be measured with percentiles.

Suppose a service answers ninety-nine requests in ten milliseconds and one request in a full second. The average is about twenty milliseconds, which sounds fine and describes nobody’s real experience. Customers see either ten milliseconds or one second. Percentiles describe this honestly: the 50th percentile, or median, is ten milliseconds, and the 99th percentile is a full second. Instrumentation overhead often shows up first in the slow tail, for example when a garbage collection pause coincides with a burst of span creation.

Record histograms, not just summary numbers

The best practice is to record the full distribution of response times as a histogram, then compare the two histograms. A histogram lets you see whether the whole curve shifted slightly, which suggests a steady per-request cost, or whether only the slow tail grew, which suggests pauses or contention. It also lets you compute any percentile later without rerunning the experiment.

Latency at a comfortable load can hide a cost that appears only when the system is pushed to its limit.

Every service has a point beyond which adding more traffic stops increasing the number of completed requests and only makes the waiting longer. This is the saturation point. Instrumentation uses a little processor time on every request, so it moves the saturation point slightly lower. To see that, you must push the system to saturation with and without instrumentation and compare where the knee of the curve appears.

Processor cost is the overhead that most directly becomes money, so measure it at a fixed amount of work.

The cleanest way to compare processor use is to hold the request rate constant, so both variants do the same work, and then compare how many processor cores each one uses. If the instrumented version uses a little more at the same rate, the difference is the cost of the instrumentation. Dividing the difference by the request rate gives a very useful number: the extra processor time spent on each request.

Memory overhead has two faces: how much is held, and how much is created and thrown away.

Instrumentation holds memory in a few places. Spans that are still in progress live in memory until they end. Finished spans wait in a queue until they are exported. The SDK itself and any agent add a fixed amount. Because the specification asks implementations not to use unbounded memory, the queue has a limit, but that limit can still be a noticeable amount of memory when spans are large or numerous.

Instrumentation also creates many small, short-lived objects: spans, attribute lists, context objects, serialised buffers. In languages with automatic memory cleanup, these objects must later be collected. A high rate of object creation means more frequent cleanup, which can cause short pauses or extra background processor use. Often the allocation rate is a more telling number than the total memory held.

Instrumentation can slow down the first seconds of a program’s life, which matters more than people expect.

Automatic instrumentation often works by inspecting and modifying program code as it loads. That takes time at start-up, and it keeps extra information in memory afterwards. In a world of containers that start and stop frequently, because of scaling, deployments and restarts, a few extra seconds at start-up can slow rollouts and delay new capacity during a traffic surge.

The cost of telemetry does not stop at the edge of your application; it continues across the network and into the systems that receive it.

Every span that leaves the application becomes bytes on a network and work for a Collector and a backend. At small scale this is invisible. At large scale it is often the largest part of the bill. A full picture of overhead therefore includes how much data is sent and what it costs the receiving systems to handle it.

The project publishes benchmark guidance that you can borrow as a starting recipe.

OpenTelemetry’s specification includes a short benchmark description for its tracing SDKs. It defines one standard test scenario: spans with no parent, default settings and always-on sampling, each carrying a numeric attribute and one event, and each ended immediately after creation. The guidance asks implementers to report three things for this scenario, and gives a few rules for how to run it.

A few configuration choices have an outsized effect on overhead, so they belong on your ladder.

The tracing SDK’s batching span processor has four main settings, each with a documented default. The queue size, defaulting to two thousand and forty-eight spans, limits how many finished spans can wait for export. The schedule delay, defaulting to five seconds, controls how often a batch is sent. The export timeout, defaulting to thirty seconds, limits how long one export may take. The maximum batch size, defaulting to five hundred and twelve, limits how many spans go out in one request. When the queue is full, new spans are dropped rather than slowing the application.

Choosing how many traces to record changes almost every cost at once, so test it explicitly.

The SDK offers several standard samplers. The always-on sampler records every span. The always-off sampler records none. The ratio-based sampler records a chosen fraction of traces, deciding in a way that stays consistent across all services touched by one trace. The parent-based sampler follows the decision already made by the parent span, which keeps whole traces together. Each choice changes how many spans are fully recorded and exported, and therefore how much overhead the system carries.

Not every cost shrinks with sampling

When a span is not sampled, most of the heavy work is skipped: recording attributes and events, queueing, serialising and sending. But some work remains for every request, such as creating the context, making the sampling decision and passing headers to the next service. This residual cost is exactly what steps one and two of your ladder measure. A good result is one where the always-off configuration is very close to the uninstrumented baseline.

flowchart LR
    R["Request arrives"] --> S{"Sampling decision"}
    S -->|"Not sampled"| N["Light path: context only, no recording, no export"]
    S -->|"Sampled"| H["Heavy path: record, queue, batch, serialise, send"]
        

How much you attach to each span matters as much as how many spans you create.

A span can carry attributes, which are labelled facts such as the route name or a customer tier, and events, which are timestamped notes. Each attribute costs a little time to set and some memory to hold, and it adds bytes to every export. A span with three short attributes and a span with fifty long ones are very different loads, even though both count as one span.

The SDK specification therefore defines limits that cap how many attributes, events and links one span can carry, and how long values may be. When a limit is exceeded, extra data is discarded and a warning is logged. These limits are a safety net, but they are not a substitute for thoughtful instrumentation, because the discarded data has already cost some effort to produce.

Where you place spans, and how many, usually matters more than any setting inside the SDK.

There are three broad ways to add instrumentation. Automatic instrumentation watches common frameworks and libraries for you, giving wide coverage with little effort. Library instrumentation is built into libraries by their authors. Manual instrumentation is code you add yourself, at places that matter to your business. Each style produces a different number of spans per request, and span count is the single biggest driver of cost.

Tracing is only one signal, and the other two can be just as expensive in different ways.

Metrics are numbers aggregated over time. Recording a measurement is usually very cheap, because it only updates a running total. The danger lies elsewhere: every distinct combination of label values creates a separate series that must be stored and exported. A label with an unbounded number of possible values, such as a user identifier or a full web address, can create an enormous number of series. This is called a cardinality explosion, and it costs memory, network and storage far more than the act of recording ever did.

Logs are records of individual events. Their cost scales with volume, which can be enormous if applications log on every request. The specification itself notes that logging can use a great deal of memory by default and recommends giving applications ways to filter what is sent. Tests should include realistic log volumes, including bursts caused by error storms.

The small headers that travel with every call are cheap individually and noticeable in aggregate.

For a trace to span several services, each call carries a few headers with the trace identifier and the sampling decision. Reading and writing these headers costs a tiny amount of time on every outgoing and incoming call, whether or not the trace is sampled. For almost all systems this cost is very small, but it is worth confirming, particularly for services that make a very large number of internal calls per request.

Baggage is a separate feature that carries custom key-value pairs along with the trace context to every downstream service. It is useful, but because it is copied into the headers of every call, it can grow the size of each message and the work of reading and writing it. Many teams are surprised by how much baggage accumulates over time.

Small, focused tests and whole-system tests answer different questions, and you need both.

A micro-benchmark measures one tiny operation in isolation, such as the time to create and end a span. It is quick, repeatable and good at spotting changes between SDK versions. A macro-benchmark measures the whole application under realistic load, which captures interactions that no isolated test can, such as processor contention, memory pressure and garbage collection behaviour.

Test environments never perfectly match reality, so the final proof of overhead comes from real systems.

Production measurement answers the question that test environments cannot: what does instrumentation cost with real traffic, real data and real neighbours? The safest way to measure it is a controlled comparison inside production, using a small group of instances with the variant and a matching group without it, both receiving the same live traffic.

A measured difference is only real if it is bigger than the random variation between identical runs.

No two runs of the same test give exactly the same numbers. Machines vary, networks vary, and the program itself varies. This natural variation is called noise. Suppose that two identical runs differ by two percent in processor use. If your instrumented variant uses three percent more than the baseline, then three percent is not clearly bigger than the noise, and you cannot honestly claim an effect. You need a way to find out how big the noise is.

The A/A test: measuring the noise floor

The simplest tool is an A/A test. Run the baseline against itself several times and look at how much the results vary. This shows the size of natural noise for your setup. Any difference between baseline and variant smaller than that noise should be treated as no detectable effect, not as zero effect.

Let us follow one complete experiment through every component involved, to see how the pieces fit.

sequenceDiagram
    participant L as Load generator
    participant A as Application under test
    participant S as SDK and exporter
    participant C as Collector or local sink
    participant M as Metrics store
    L->>A: Send steady open-loop traffic
    A->>S: Create spans and record data
    S->>C: Export batches in the background
    A-->>L: Responses with timings
    A->>M: Report processor and memory use
    C->>M: Report received and dropped counts
    L->>M: Store latency histograms
    M->>M: Compare baseline and variant runs
        

Many tools can do each job, so it is more useful to know the categories than to memorise names.

Numbers make ideas stick. The figures below are invented for teaching, so please do not treat them as a promise about any real system.

Imagine an online shop with a service that handles checkout. During a busy hour it receives about two thousand requests every second. The team wants to switch on tracing, metrics and logs through OpenTelemetry. Before they do, they set a budget: the instrumented service may add at most five percent to the slowest requests, at most ten percent more processor time, and at most fifteen percent more memory.

They run the experiment ladder from earlier chapters. The first run is the baseline with no instrumentation. The second run has the library installed but switched off. The third run has full tracing at one hundred percent sampling. The fourth run keeps tracing but samples only ten percent of requests. All four runs use the same machine type, the same replayed traffic and the same duration, and each is repeated several times.

Now do the arithmetic. Full tracing raised the slow-request latency from forty to forty-six milliseconds, which is fifteen percent. It raised processor use from six to seven point one cores, about eighteen percent. Memory grew from eight hundred to nine hundred eighty megabytes, about twenty-two percent. All three exceed the budget. With ten percent sampling, latency rose by two and a half percent, processor use by five percent and memory by about nine percent. All three fit.

A result is only useful when you understand what it does and does not say.

The first lesson is to look at both absolute and relative change. A rise of six milliseconds sounds small, yet on a request that normally takes forty it is fifteen percent. A rise of two milliseconds on a request that takes two seconds is invisible. Always write down both numbers so that readers can judge for themselves.

The second lesson is to think in cost per request. If instrumentation adds a fixed amount of processor time to each request, then a service handling more requests pays more in total. A second way to say this is that overhead usually grows with the number of spans, metric updates and log lines per request, not with the number of requests alone. A request that touches twenty internal steps and records a span for each pays much more than one that records a single span.

The third lesson is about shape. Averages hide trouble. If most requests are unchanged but a few become much slower because an export buffer fills up, the average barely moves while the slow tail grows. Look at percentiles, and look at the whole distribution, not at one summary number.

When the budget is exceeded, change the biggest levers first.

Many teams start by tuning small settings and are disappointed. The better habit is to work from the largest source of cost to the smallest. The timeline below follows that order.

Bad measurements are worse than none, because they create confidence in the wrong answer.

A short list of habits separates calm measurement from chaos.

A result that cannot be repeated by someone else is only an opinion.

A good report is short, honest and complete enough that a colleague could rerun the experiment. Start with the question and the budget. Then describe the setup: machine type, number of processor cores, language and runtime versions, OpenTelemetry library versions, configuration, sampling setting and the workload shape. After that show the results as a table with the spread between runs, followed by a plain sentence stating whether the budget was met.

Next, state the limits. Say what you did not test, such as other languages, much higher traffic or unusual failures. This is not weakness; it protects readers from using your result somewhere it does not apply. Finally, store the raw data and the exact configuration in a place the team can find later, because the question will come back after the next upgrade.

Overhead is not a one-time question. It can creep up every release unless something watches for it.

Because instrumentation changes with every new span, attribute and library upgrade, overhead tends to rise slowly, like weight gained a little at a time. The cure is to run a small, repeatable test inside your delivery pipeline. It does not need to be perfect; it needs to be consistent. If the result moves outside an agreed band compared with the last accepted result, the pipeline raises a warning or blocks the change.

flowchart LR
    A["Code or library change"] --> B["Build service"]
    B --> C["Run baseline and instrumented tests"]
    C --> D["Compare with last accepted result"]
    D --> E{"Within the budget band?"}
    E -->|"Yes"| F["Accept and store the result"]
    E -->|"No"| G["Warn or block and investigate"]
    G --> H["Fix or update the budget with approval"]
    H --> C
        

The diagram shows the loop. A change triggers a build. The pipeline runs the baseline and the instrumented versions under the same conditions, then compares the outcome with the last accepted result. If it fits, the new result becomes the reference. If not, a person investigates, and either fixes the cause or deliberately updates the budget with approval so that the decision is visible.

A few repeatable patterns make the whole subject easier to live with.

Different kinds of systems feel overhead in different places.

No single method answers every question, so choose by the question you are asking.

When numbers surprise you, the cause is usually one of a small set of familiar things.

OpenTelemetry gives guidance, but you remain responsible for your own numbers.

The project publishes specification material about performance and benchmarking, and the individual language projects often keep their own benchmark results and suggestions. These are helpful starting points, yet they come from the maintainers’ own machines, workloads and versions. They cannot tell you how your service behaves.

Remember too that implementations differ. A setting that exists in one language may be named differently, or may not exist, in another. Defaults also change from release to release. Whenever you upgrade a library, treat the previous measurements as out of date until you repeat them. Check the current documentation for your language before relying on any specific setting described in a general guide.

It is the extra time, processor work, memory, network traffic and start-up cost that your service pays because it creates and sends telemetry data.

No. The right limit depends on your service targets and business. Set your own budget in advance, then test against it.

Usually very little, but check. The disabled-library step on the ladder tells you exactly how much.

Typically creating many spans, attaching many attributes and exporting everything. Reducing the volume through sampling and careful choice of what to record usually helps most.

No. Unsampled requests may still pay a small decision cost. Sampling mostly removes the larger cost of recording and exporting the data.

Averages hide the slowest requests, which are the ones users notice and where queue and pause problems appear first.

Yes, carefully, using a canary that compares an instrumented share of machines with an uninstrumented share, ideally after lab tests have passed.

Each language library is built differently, with its own runtime, memory model and defaults, so each needs its own measurement.

Yes, but in a separate process. It moves work off the application and adds its own resource cost and network hop, which you should measure and plan for.

Before the first rollout, after each major upgrade or instrumentation change, and ideally automatically on every change using a small regression check.

They are a useful guide to what is typical, but your workload, hardware and settings are different, so measure your own service.

Work down the list of levers: sample less, remove low-value spans, limit attributes and dimensions, batch exports, and offload work to a Collector. Then measure again.

The same idea of a span costs different amounts in different languages, because the languages themselves work differently.

Some languages run inside a managed runtime that cleans up unused memory automatically through a process called garbage collection. Every span, attribute and event is a small object. Many small objects make the cleaner work harder, and the cleaner can briefly pause the program. In such languages, overhead often appears as extra pauses and a fatter slow tail.

Other languages manage memory directly, or through ownership rules. There the cost shows up more as processor work and copying, and less as pauses. Interpreted languages add their own cost, since each instruction takes longer to run, and instrumentation code is instructions too. Languages that run many tasks at once may lose time when instrumentation uses locks that tasks must queue for.

Overhead is not only a technical number; it becomes machines, bills and headroom.

Suppose full tracing adds eighteen percent processor use to a service. If the service runs on one hundred machines, the fleet needs roughly eighteen more machines to carry the same traffic at the same comfort level. That cost is real, and it belongs in the plan next to the cost of storing and viewing the data.

Capacity planning therefore asks three questions. How much extra resource does instrumentation need per unit of traffic? How much headroom do we keep so that a sudden spike does not push the service past its limit? And how will the overhead change as traffic and detail grow? Answering them early avoids unpleasant surprises, such as a service that was comfortable on Monday and struggling on Friday because a new release added many spans.

You do not need to do everything at once. Grow the habit in small, sensible steps.

Overhead does not stop at the edge of your application; the data still has to travel and be handled.

Many teams place a Collector between applications and the final destination. This moves heavy work out of the application, but it does not make the work disappear. The Collector uses processor time to receive, change and forward data, memory to hold queues, and network bandwidth in both directions. If you measure only the application, you will believe the cost is smaller than it truly is.

To measure the pipeline fairly, give the Collector the same treatment as the application. Run it on machines like those you will use in production. Send it realistic volumes of traces, metrics and logs. Record processor use, memory, queue depth, the number of items dropped, and the delay between arrival and delivery. Then raise the volume step by step until you find the point where it starts to fall behind. That point, with a safety margin, tells you how many Collector instances you need.

Measuring overhead is a calm, repeatable habit, not a one-time scare.

You began with a doctor who must not disturb the patient while checking their health. You learned that instrumentation costs time, processor work, memory, network traffic and start-up speed, and that these costs come mainly from creating, enriching and exporting data. You learned to set a budget, build a fair experiment, climb an overhead ladder, use a realistic workload, and read percentiles rather than averages. You saw how sampling, careful instrumentation, bounded attributes and batching reduce cost, and how pipeline checks catch slow creeping increases.

Remember that overhead is a trade, not a sin. Telemetry buys you the ability to see failures, slow requests and strange behaviour that would otherwise stay hidden. The goal is never zero overhead; the goal is a known, agreed and affordable overhead that gives your team clear insight. When the cost is measured honestly and kept within a written budget, the debate stops being a matter of opinion and becomes a matter of evidence that anyone can inspect, repeat and improve over time.

Finally, share what you learn. A short note describing your budget, your method and your latest numbers helps neighbouring teams avoid repeating your mistakes, and invites them to correct yours.