How Do You Suppress or Disable Specific Instrumentation in OpenTelemetry?
A plain-English guide to switching off the telemetry you do not want, from a single noisy route to a whole library, at every layer from your application to your collector, without accidentally breaking the traces you do want.
Imagine you install a very eager security camera system in your house. It records the front door, which is useful. But it also records the ceiling fan, the cat’s water bowl, the refrigerator door, the clock on the wall and every flicker of the window curtains. After a week, you have a mountain of footage, a huge storage bill, and no way to find the one clip that actually matters. The cameras are not broken. They are simply recording too much of the wrong things.
OpenTelemetry instrumentation can behave in exactly the same way. Once you switch on automatic instrumentation, it starts recording everything it knows how to record: every web request, every database call, every file read, every health check from the load balancer. Much of it is valuable. Some of it is noise, some of it is expensive, and some of it is private. The skill you need is the skill of choosing what to switch off.
In this guide you will learn what instrumentation is, why you might want to suppress or disable parts of it, and the many layers at which you can do it: from turning off a whole signal, to disabling one library, to ignoring one URL, to filtering data in the collector. You will also learn the most dangerous trap, which is that switching off the wrong thing can quietly break your traces. No prior knowledge is assumed. Every new term is explained the first time it appears.
Read the first five chapters in order, because they build the vocabulary and the map. After that, you can jump straight to the layer you need, or to the chapters on mistakes and the FAQ.
1The Big Picture: When Too Much Telemetry Becomes a Problem
Before we learn how to switch things off, we need to understand why anyone would want to. It starts with how easy it is to produce too much.
When a program runs in production, you cannot stand next to it and watch it. So you ask the program to send you information about itself. This information is called telemetry. The most common kinds are traces, which follow one request through many services, metrics, which are numbers measured over time, and logs, which are timestamped messages.
OpenTelemetry offers a wonderful shortcut: automatic instrumentation. You attach an agent or install a few packages, and your program starts producing telemetry for web servers, database drivers, message queues and many other libraries, without you writing any measuring code. This is a huge time saver. But “everything, automatically” has a flip side. It does not know what matters to you.
How much is too much?
Consider a small online shop. A load balancer checks whether each server is alive by calling a health check address every five seconds. With ten servers, that is two calls per second, or about 172,800 calls per day, and every one of them can produce a trace with several spans. None of those traces will ever help anyone debug anything. Yet they take up network bandwidth, processing time in the application, space in the backend and money on the monthly invoice.
Now add a library that records every single file read, a database driver that records every tiny internal query, and a framework that creates spans for every internal function call. In a busy system, it is entirely possible for more than half of the telemetry to be completely unhelpful.
Think of a newspaper. A good editor does not print every sentence that every reporter ever wrote. The editor cuts the boring parts, removes the repeated parts and keeps the stories that readers need. Suppressing instrumentation is the editor’s job for your telemetry: it is not about hiding the truth, it is about keeping the paper readable and affordable.
2The Basic Vocabulary: Instrumentation, Signals and Agents
A handful of small words appear in every conversation about this topic. Learn them once and the rest becomes easy.
Instrumentation
Instrumentation is code that measures what a program is doing and produces telemetry about it. The word is borrowed from the instruments on a ship or an airplane: the dials, gauges and meters that tell the pilot what is happening. In software, instrumentation creates spans, increments counters and writes log records.
Manual and automatic instrumentation
Manual instrumentation means you, the developer, write the measuring code yourself, for example “start a span called calculate price, do the work, end the span.” You are in complete control, because nothing happens unless you wrote it.
Automatic instrumentation, also called auto-instrumentation, means that a ready-made tool adds measuring code for you, for popular libraries and frameworks. It does this by attaching an agent when the program starts, by wrapping library functions at runtime, or by shipping small add-on packages. You get telemetry from the web framework, the HTTP client, the database driver and more, with almost no effort.
Instrumentation library
Each piece of automatic instrumentation usually lives in its own small package, called an instrumentation library, which knows how to measure one thing, for example one web framework or one database driver. A system may have dozens of them active at once. Each can usually be switched on or off separately, which is the key fact that makes suppression possible.
Signals
Traces, metrics and logs are called signals. A single instrumentation library may produce more than one signal. For example, an HTTP server library may produce a span for each request and also a metric for request duration.
SDK, API and exporter
The API is the set of functions that instrumentation calls to create telemetry. The SDK is the engine behind the API that collects, processes and sends the telemetry. An exporter is the part of the SDK that sends the finished telemetry to another place, such as a collector. Suppression can happen in any of these places, as you will see.
Instrumentation
Code that observes what the program does and produces telemetry about it.
Auto-instrumentation
Measuring code added for you by an agent or add-on packages, covering popular libraries without code changes.
Instrumentation library
A package that measures one specific thing, such as one web framework. It can usually be turned off on its own.
Signals
Traces, metrics and logs. Each can be controlled separately.
Think of a building with many security sensors: door sensors, window sensors, motion sensors and smoke detectors. Each sensor is an instrumentation library. The alarm panel in the hallway is the SDK. The phone line to the security company is the exporter. You can unplug one noisy motion sensor without touching the smoke detectors, or you can silence the phone line entirely. Knowing which part you want to quiet is the whole game.
3How Instrumentation Creates Telemetry
To switch something off at the right place, you must first understand the road that telemetry takes from your code to your dashboard.
graph LR
A["Your Application Code"] --> B["Instrumentation Library"]
B --> C["OpenTelemetry API"]
C --> D["SDK: samplers, processors, views"]
D --> E["Exporter"]
E --> F["Collector: receivers, processors, exporters"]
F --> G["Backend and Dashboards"]
Read the figure from left to right. Your application does its normal work, such as handling a web request. An instrumentation library watches that work and calls the OpenTelemetry API to say “a request started” or “a database call finished.” The SDK behind the API decides what to do with those calls: a sampler may decide to drop a trace, processors may change or filter spans, and views may reshape metrics. The exporter then sends whatever is left to a collector, which can filter and change it again before it reaches the backend that stores it and draws your dashboards.
Every box in this road is a place where something can be switched off or filtered. And each choice has a different price. The earlier you stop something, the more you save, because the work, memory and network use are avoided completely. The later you stop it, the more you have already paid for.
What an instrumentation library actually does
- Hooks in. It attaches itself to the library it watches, for example by wrapping a function in a web framework.
- Creates spans or measurements. When the watched function runs, it starts and ends spans, adds attributes such as the HTTP method and the route, and records metrics.
- Handles context. It reads trace headers from incoming requests and writes trace headers into outgoing ones. This is called context propagation, and it is what ties the spans of different services into one trace.
- Hands data to the API. It does not send anything over the network itself. The SDK and exporter do that.
That third point, context handling, is the one that matters most for suppression. An instrumentation library often does two jobs at once: it produces telemetry and it passes along the trace identity. If you switch off the whole library, you may stop both jobs, and that can break traces in other services. We will give this problem its own chapter.
Imagine a postal worker who both writes a log of every parcel and also sticks the tracking label on each parcel. If you tell the worker to stop keeping the log because it is too long, you must be careful not to also stop the labels. Otherwise the parcels travel on with no labels, and the next sorting center cannot tell where they came from.
4Why Suppress or Disable Instrumentation?
There are several very different reasons for switching something off. Knowing your reason helps you choose the right tool.
Boring, repeated telemetry
Health checks, readiness probes, metrics scrapes and internal heartbeats produce huge numbers of identical traces that nobody ever reads.
Storage and network bills
Every span has a price: processing, bandwidth, storage and search. Removing low-value spans lowers the bill directly.
Overhead in the application
Measuring takes a little time and memory. On a very latency-sensitive path, or in a very chatty library, that overhead can matter.
Sensitive data in attributes
Some instrumentation captures database statements, URLs or headers that may contain personal or secret data.
Two tools measuring the same thing
A proxy and the application, or two libraries, may both create a span for the same call, making traces confusing and doubling cost.
A buggy or incompatible library
Occasionally, a specific instrumentation library does not work well with a particular version of a framework, and switching it off is the safest quick fix.
Too many unique labels
Some metrics carry labels with huge numbers of values, such as raw URLs with identifiers. These multiply the number of series and the cost.
Compliance and ownership rules
A team may be told not to send certain telemetry outside the company, or may not own the library whose behavior is being measured.
Suppress, disable, drop and filter: what is the difference?
People use these words loosely, but there is a useful way to separate them.
- Disable usually means switching off a whole component, such as an instrumentation library or the entire SDK, so that it does nothing at all.
- Suppress usually means stopping telemetry for a specific situation while the component stays active, for example “do not create spans while inside this block of code” or “ignore requests to this address.”
- Drop means that telemetry has been created but is thrown away before it is stored, for example by a sampler or a view.
- Filter means selecting which telemetry to keep based on rules, for example in a collector.
In everyday speech all four may be called “suppression.” This guide uses the wider meaning, and points out the differences where they matter.
Switch off at the narrowest level that solves your problem. Turning off one URL pattern is safer than turning off a whole library, and turning off one library is safer than turning off the whole SDK.
5The Layers of Control: A Map
There is no single “off switch” in OpenTelemetry. There are many switches, each at a different layer. This chapter gives you the map, and the following chapters visit each stop.
graph TD
A["Deployment level: do not attach the agent, operator annotations"] --> B["SDK level: disable the SDK or a signal"]
B --> C["Instrumentation library level: disable one library"]
C --> D["Request level: ignore specific URLs or operations"]
D --> E["Code level: suppress inside a block of code"]
E --> F["SDK processing: sampler, span processor, metric views"]
F --> G["Collector level: filter, transform, sample"]
G --> H["Backend level: ingestion rules and retention"]
The figure runs from the top, where the cut is earliest, to the bottom, where it is latest. A cut near the top prevents everything below it from happening. A cut near the bottom happens after the work and the transport have already been paid for.
| Layer | What you switch off | Savings | Risk to traces |
|---|---|---|---|
| Deployment | The whole agent or auto-instrumentation for a service | Highest | High, the service disappears from traces and may break propagation |
| SDK or signal | The whole SDK, or all traces, metrics or logs | High | High for the signal switched off |
| Instrumentation library | One library, such as a file system or one framework | High for that library | Medium, may stop propagation if it is an HTTP or messaging library |
| Request level | Specific URLs, routes or operations | Medium to high | Low, only those requests are affected |
| Code level | Telemetry inside a chosen block of code | Medium | Low and very targeted |
| SDK processing | Chosen traces or metrics through samplers, processors and views | Medium | Low if the sampler is used, context still travels |
| Collector | Chosen spans, metrics, logs or attributes after they arrive | Lower, work and network are already spent upstream | Low, but it can cut holes in traces if done carelessly |
| Backend | Stored data by rules or age | Lowest | Low |
How to use the map
Ask three questions every time. First: what exactly do I want to stop? A single route? One library? One kind of attribute? Second: do I still need trace propagation through this part? Third: where is the cheapest place that still solves the problem safely? The answers point you to a layer.
Think of a water system. You can close the main valve of the whole city, close the valve of one street, close the tap of one house, or put a filter on the kitchen faucet. Closing the city valve is the biggest cut and the biggest risk. A filter on the faucet is the smallest. The wise plumber closes the smallest valve that fixes the leak.
6Level One: Turning Off the SDK or a Whole Signal
The biggest switches are the simplest. They are also the blunt ones, so use them only when you really mean to switch off a lot.
Disabling the whole SDK
The OpenTelemetry specification defines a standard environment variable called OTEL_SDK_DISABLED. When it is set to true, a compliant SDK behaves as though OpenTelemetry were not installed. Instrumentation calls still happen, but the SDK does nothing with them: no spans are recorded, no metrics are kept and nothing is exported. It is the “main fuse” for telemetry.
This is useful in a few cases. You might switch the SDK off in a unit test environment, on a developer’s laptop, in a temporary emergency when telemetry is suspected of harming performance, or in a deployment where no backend is available. Because it is an environment variable, it can be set without changing code, and removed again just as easily.
Support for this variable depends on the language and the SDK version. Some SDKs honor it fully, others only partly. After setting it, verify with a test that nothing is being exported. Never assume.
Turning off one signal
You often want to stop one signal but keep the others. OpenTelemetry defines a standard setting for the exporter of each signal. Setting the exporter to the special value none means “do not export this signal anywhere.”
| Setting | Value | Effect |
|---|---|---|
OTEL_TRACES_EXPORTER | none | Traces are not exported. Metrics and logs continue. |
OTEL_METRICS_EXPORTER | none | Metrics are not exported. Traces and logs continue. |
OTEL_LOGS_EXPORTER | none | Logs are not exported. Traces and metrics continue. |
OTEL_SDK_DISABLED | true | Everything is switched off. |
This is the right tool when a whole signal is not needed. For example, a team that only wants traces can set the metrics exporter to none, so that the application does not spend effort collecting and sending metrics. Remember that setting an exporter to none stops the sending, but in some SDKs the instrumentation may still create the telemetry in memory before it is thrown away. If you need to avoid even the creation, look for a setting that disables the instrumentation itself, which the next chapter covers.
Switching off the SDK is like turning off the main electrical switch of a house: everything goes dark. Setting one exporter to none is like turning off the lights in a single floor. The house still works, the wiring is still there, but one part is dark.
When to use these big switches
- In tests, where telemetry would only add noise and slowness.
- In a quick emergency, where you need to rule out telemetry as the cause of a performance problem.
- For a service that really needs no telemetry of one kind, such as no metrics.
- As a safe default for a new environment until the backend is ready.
They are not the right tool for removing one noisy route or one chatty library. For that, you need a narrower switch.
7Level Two: Disabling Specific Instrumentation Libraries
This is the switch most people are looking for. It turns off one instrumentation library and leaves everything else running.
Every auto-instrumentation distribution has a list of the libraries it can instrument, each with a short name such as “flask,” “redis,” “http” or “kafka.” Most distributions let you either disable chosen names from the full list, or disable everything and enable only the names you want. The exact setting names differ by language, but the idea is the same.
Per-library enable switches
The Java agent has a setting for each instrumentation, named with the pattern otel.instrumentation.[name].enabled, or the matching environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED. Setting it to false disables that one. A common default setting lets you switch off all instrumentation first and then switch on only the ones you want.
Disabled instrumentations list
The Python auto-instrumentation tool reads OTEL_PYTHON_DISABLED_INSTRUMENTATIONS, a comma-separated list of the names to skip, such as a web framework or an HTTP client library.
Enabled and disabled lists
The Node.js auto-instrumentation bundle reads OTEL_NODE_ENABLED_INSTRUMENTATIONS and OTEL_NODE_DISABLED_INSTRUMENTATIONS. A very common use is disabling the noisy file system instrumentation.
Per-instrumentation switches
The .NET auto-instrumentation offers environment variables that follow a pattern of signal and instrumentation name, to enable or disable each individual instrumentation. In code-based setups, you simply do not add the instrumentation you do not want.
Opt-in by design
In Go, most instrumentation is added by wrapping handlers and clients on purpose. To suppress it, you do not wrap that handler or client. There is nothing to switch off because nothing was switched on.
Kubernetes injection control
When using the OpenTelemetry Operator, injection is controlled by annotations on the workload. Removing or changing the annotation stops the agent from being added to that workload.
Two styles: deny list and allow list
- Deny list. Everything is on, except the names you list. This is easy to start with, but when you upgrade, newly added instrumentations switch themselves on, which can surprise you.
- Allow list. Everything is off, except the names you list. This is stricter and more predictable. It is a good choice for mature systems where you know exactly what telemetry you want.
Which libraries are commonly disabled?
| Library type | Why people disable it | Caution |
|---|---|---|
| File system access | Creates many tiny spans for every file read or write | Usually safe to disable, rarely needed for propagation |
| Low-level DNS and network sockets | Very detailed, rarely useful, adds many spans | Usually safe |
| Internal framework layers | Spans for every middleware or controller step | Check dashboards that rely on those spans |
| Logging appenders and bridges | Logs are already collected another way | Make sure logs still reach their destination |
| Database drivers | Statements may contain sensitive data or be too numerous | Consider masking instead of disabling |
| HTTP server or client libraries | Duplicate spans, or a bug | High risk: may stop trace propagation |
| Messaging libraries | Too many spans for high-volume queues | High risk: may break the link between producers and consumers |
The exact names and settings are specific to each language and each release. Names change, new ones are added, and some are renamed. Always check the documentation for the version you run, and verify by looking at the telemetry afterward. A misspelled name is usually ignored silently, which makes you think you switched something off when you did not.
8Level Three: Ignoring Specific Requests, URLs and Routes
Often you do not want to remove a whole library. You want to remove just the boring requests. This is the most common and the safest kind of suppression.
Instrumentation for web servers and HTTP clients usually lets you give it a list of patterns, or a small function, to decide which requests to ignore. If a request matches, the library simply does not create a span for it. All other requests are still measured as normal.
Typical things to ignore
- Health check and readiness probe addresses.
- Address paths that serve metrics for scraping.
- Static files such as images, style sheets and scripts.
- Internal status pages and administrative pings.
- Favicon requests from browsers.
How it is done in different ecosystems
Excluded URLs settings
The Python instrumentations read comma-separated lists of patterns from settings such as OTEL_PYTHON_EXCLUDED_URLS for all libraries, or library-specific variants such as one for the web framework or one for the HTTP client. The patterns are regular expressions matched against the address.
Ignore hooks
The Node.js HTTP instrumentation accepts hook functions to ignore incoming and outgoing requests. The function receives the request and returns true when the request should be skipped.
Filter options
The ASP.NET Core and HTTP client instrumentations have filter options. You supply a function that receives the request and returns whether to record it.
Filters on the HTTP wrapper
The Go HTTP instrumentation wrapper accepts filter options. A filter returns false for requests that should not be traced.
Agent settings and sampling
Java users most often combine agent settings with a sampler rule or a collector filter, because the options for ignoring particular addresses depend on the framework and the version. Check the agent documentation for the current options.
Sampler rule as a fallback
If the library offers no filter, a custom sampler that drops spans for given addresses achieves the same result, as Chapter 10 explains.
Server side and client side are different
A subtle point: ignoring a URL on the server side means “when my service receives a request for this address, do not trace it.” Ignoring a URL on the client side means “when my service calls this address, do not trace the call.” These are separate settings. A common beginner surprise is to configure the server-side ignore list and then still see spans for outgoing calls to the same address from another service. Decide which side you mean, and set both when needed.
Think of a hotel doorman who writes in a visitor book. You can instruct the doorman: “Do not write down the postman.” That is the server side: the hotel ignores a certain kind of visitor. But the postman’s own company also keeps a delivery log. To remove the hotel from the delivery log, you must change the postman’s company’s rule, which is the client side.
Patterns that are too broad can hide real traffic. A pattern meant to skip the health check address should match exactly that address, not every address that happens to contain the word “health.” Test the pattern against real URLs before deploying it.
9Level Four: Suppressing Inside a Block of Code
Sometimes you want telemetry to pause for a specific stretch of code. OpenTelemetry’s SDKs have a built-in idea for this, called context-based suppression.
Recall that the context is the small bag of information that travels with a piece of work. Some SDKs let you put a special marker into that bag, which means “do not create telemetry while this marker is present.” Instrumentation libraries check the context before creating spans, and if they see the marker, they skip their work. When the block of code finishes, the marker is removed, and normal instrumentation resumes.
In the Python ecosystem, this is offered as a helper that you wrap around a section of code. In the JavaScript ecosystem, there is a function that returns a new context in which tracing is suppressed. Other SDKs have similar tools or internal markers. The exact helpers differ between languages and versions, so check your SDK’s documentation.
The classic example: do not trace the tracer
Telemetry exporters send data over the network, often using an HTTP client library. If that HTTP client is itself instrumented, then sending a batch of spans would create a new span for the sending, which would be exported in the next batch, which would create another span, and so on. This feedback loop would flood the system. To prevent it, exporters run their sending inside a suppressed context, so the HTTP client does not trace its own telemetry traffic. You get this protection for free in good SDKs, but you can use the same idea for your own code.
sequenceDiagram
participant App as Application Code
participant Ctx as Context
participant Lib as Instrumented HTTP Client
App->>Ctx: Mark telemetry as suppressed
App->>Lib: Call a monitoring endpoint
Lib->>Ctx: Is suppression marked?
Ctx-->>Lib: Yes
Note over Lib: No span is created
App->>Ctx: Remove the mark
App->>Lib: Call a business service
Lib->>Ctx: Is suppression marked?
Ctx-->>Lib: No
Note over Lib: Span is created as usual
Good uses for code-level suppression
- Internal housekeeping calls. A background task that polls a configuration service every second and would otherwise create thousands of identical traces.
- Calls to monitoring and telemetry systems. Any code that talks to the observability system itself.
- Bulk operations. A loop that reads a million rows and would otherwise create a million tiny spans. Suppress inside the loop, and create one span around the whole loop instead.
- Sensitive sections. A short section of code that handles private data, where you prefer no automatic spans to be created.
It is like putting a “Do not disturb” sign on a hotel room door. While the sign is hanging, the cleaning staff leaves the room alone. When you take the sign down, normal service resumes. The sign belongs to the guest, and it only affects that room for that time.
Suppression must be tied to a block that ends cleanly, even when an error occurs. If the marker is left on by mistake, all later telemetry in that flow disappears silently. Use the language’s built-in structure for this, such as a scoped block, and never set the marker without a guaranteed way to remove it.
10Level Five: Dropping Inside the SDK with Samplers, Processors and Views
The SDK has its own tools for thinning telemetry out. They are the best choice when you want to remove telemetry but keep trace identities flowing.
Samplers: dropping whole traces by rule
A sampler decides, when a span starts, whether the trace will be recorded and exported. A custom sampler can look at the name of the span and its starting attributes, and decide to drop requests for health checks or other noisy routes, while keeping everything else. This is called head-based sampling, because the decision is made at the head of the trace.
The great advantage of dropping through a sampler is that the trace context still travels. A dropped span becomes a lightweight placeholder, the trace ID and span ID are still passed along, and other services still see a consistent picture. Chapter 13 explains why this matters.
Span processors: changing or filtering spans
A span processor receives every span at its start and its end. It can add attributes, remove sensitive attributes, or in some designs filter spans out before export. Processors are handy for scrubbing private data. They are less ideal for dropping spans, because by the time a processor sees a span, the span has already been created, and the application has already paid the cost of creating it. Dropping a parent span in a processor while keeping its children can also leave orphan spans in the trace, so be careful.
Metric views: reshaping or removing metrics
For metrics, the SDK offers views. A view is a rule that matches instruments by name, type or source, and changes how they are handled. Views can drop an instrument entirely, keep only chosen attributes to reduce the number of series, rename a metric or change its aggregation. If one instrumentation library produces a metric with a label that has thousands of values, a view that removes that label can cut the cost sharply without touching the library.
Sampler
Best for dropping whole traces by rule at the start. Keeps context flowing.
Span processor
Best for adding, changing or removing attributes. Use with care for dropping.
View
Best for removing a metric, removing a label or changing aggregation without touching the instrumentation.
Log record processor
Best for filtering or changing log records before export, where the SDK supports it.
It happens inside your own process, before anything crosses the network, so the savings are large. And it preserves context propagation. For many noisy-route problems, a sampler rule is the best balance of safety and savings.
11Level Six: Filtering in the OpenTelemetry Collector
When you cannot change the application, or when you want one central place to apply rules, the collector is a powerful tool.
The OpenTelemetry Collector is a separate program that receives telemetry from many applications, processes it, and forwards it to one or more backends. Its processing stage is made of small building blocks called processors. Several of them are useful for suppression.
Filter processor
Drops spans, metrics or logs that match conditions you write, for example spans whose route attribute equals the health check address. Conditions use a small expression language that the collector understands.
Attributes and transform processors
Add, rename, mask or delete attributes. This is the right tool for removing sensitive values or high-cardinality labels from everything passing through.
Sampling processors
A probabilistic sampler keeps a fixed share of traces by trace ID. A tail sampler keeps traces based on what happened in them, such as errors or slowness.
Memory limiter and batch
Protect the collector itself and group data efficiently. They do not remove chosen telemetry, but they matter when volume is high.
Strengths of collector-level filtering
- No application change. You can fix noise in a service you cannot redeploy.
- One central policy. A single rule can apply to every service in the company.
- Fast to change. Updating a collector configuration is usually quicker than rebuilding and releasing many services.
- Language independent. The same rule works regardless of the language of the source service.
Limits of collector-level filtering
- The work is already done. The application has created, recorded and sent the telemetry. You save backend storage, but not the application’s effort or the network between the application and the collector.
- It can cut holes in traces. Dropping a parent span while keeping its children, or dropping spans from the middle of a trace, leaves orphans and broken trees. Filter whole traces, or leaf spans that have no children, to stay safe.
- It cannot undo propagation. The collector sees telemetry after the fact, so it cannot change which trace headers were sent between services.
Think of a sorting center in a post office. The center can throw away junk mail before it reaches your house, which saves you from a pile of paper at home. But the junk mail has already been printed, stamped and carried to the center. The senders still paid for printing and delivery up to that point.
12Level Seven: Controlling Instrumentation at Deployment Time
The earliest cut of all happens before the application even starts. If the agent is never attached, no instrumentation ever runs.
Automatic instrumentation has to be introduced into a program somehow. Depending on the language and platform, that is done by adding an agent file to the start-up command, by adding a wrapper command, by installing packages, or by letting a Kubernetes component inject it. Each of these entry points is also an exit point: if you remove it, instrumentation disappears.
Ways instrumentation is introduced, and how to switch it off
| How it is introduced | How to suppress it | Notes |
|---|---|---|
| An agent file added to the start command | Remove the agent from the command, or use the agent’s own enabled setting | The Java agent has a setting to switch the whole agent off, which is handy for quick tests |
| A wrapper command that starts your program | Start the program without the wrapper | Common in the Python world |
| Packages and start-up code | Remove the registration code or the specific package | Common in Node.js and .NET |
| Kubernetes injection by an operator | Remove or change the annotation on the workload, or restrict it to chosen containers | Keeps decisions in deployment files, visible to reviewers |
| A base container image that includes the agent | Build a variant without it, or use settings to disable | Be careful when many teams share the base image |
Why deployment-level control is attractive
- Maximum savings. Nothing is attached, nothing runs, no memory is used.
- Visible and reviewable. The decision lives in the deployment files, not hidden in code.
- Easy to roll back. Adding the annotation or the agent again restores everything.
Why it is also the riskiest
A service with no instrumentation at all is a hole in every trace that passes through it. It will not extract incoming trace headers and will not inject outgoing ones, so the trace is cut in two at that point. Use deployment-level control only for services that really are not part of any traces you care about, such as isolated batch tools, or for a temporary experiment.
Before removing the agent from a service, list the services that call it and the services it calls. If any of them participate in traces that matter, removing the agent creates a gap in the middle of those traces.
Imagine a relay race in which one runner decides not to take part. The baton is dropped. The runners before and after are fine, but the race cannot be recorded as one race anymore. A service that opts out of instrumentation is that runner.
13The Hidden Danger: Disabling Telemetry Versus Breaking Propagation
This is the most important chapter of the guide. Many painful incidents come from confusing “stop recording spans” with “stop passing the trace along.”
As we saw earlier, instrumentation libraries often do two jobs. Job one is producing telemetry: creating spans, metrics and logs. Job two is propagating context: reading trace headers from incoming messages and writing them into outgoing ones, so that the trace identity flows from service to service.
When you disable a whole HTTP server or HTTP client library, you usually switch off both jobs at once. The server library no longer reads incoming trace headers, so the service starts a brand new trace for each request. The client library no longer writes trace headers, so the next service cannot continue the trace. Your traces fall apart into fragments.
graph LR
A["Service A: instrumented"] -->|"trace headers sent"| B["Service B: HTTP library disabled"]
B -->|"no trace headers sent"| C["Service C: instrumented"]
A --> T1["Trace 1 contains A only"]
B --> T2["No spans from B"]
C --> T3["Trace 2 starts at C as a new root"]
In the figure, service B has its HTTP instrumentation switched off. Service A sends trace headers, but B does not read them and produces no spans. When B calls C, it sends no trace headers, so C starts a fresh trace. Instead of one connected journey, the backend now shows two disconnected stories, and nobody can see that they belong together.
Two different ways to remove a span
| Method | Is a span created? | Is context passed on? | Result |
|---|---|---|---|
| Disable the whole HTTP library | No | Usually no | Trace is split |
| Drop with a sampler | A non-recording placeholder, not exported | Yes, with the sampled flag turned off | Trace consistently absent, no fragments |
| Ignore a URL in the library | No span for that request | Depends on the library, usually the request is treated as untraced | Fine for requests that are truly isolated, like health checks |
| Filter in the collector | Yes, but removed later | Yes | Trace intact up to the filter, but you must avoid orphaning |
The safe rules
- Prefer narrow switches. Ignore a route or drop with a sampler, before disabling a whole HTTP or messaging library.
- If you must disable a library, look for a setting that keeps propagation. Some SDKs let you disable span creation while keeping the propagators running. Check the documentation.
- Remember the propagators are separate. Context propagation is performed by propagator components configured in the SDK. As long as they are configured and used by something that reads and writes headers, trace identity can still flow. But if the instrumentation that calls the propagators is switched off, nothing calls them.
- Test after every change. Send a request through a path that includes the changed service and check that the trace is still one tree.
Think of a courier who both writes a diary of every parcel and stamps the tracking number on each one. You can tell the courier to stop writing the diary, which is safe. But if you tell the courier to stop doing parcel paperwork entirely, the stamp goes too, and the next warehouse has no idea where the parcel came from. Always ask which of the two jobs you are stopping.
14Deciding Where to Cut: A Practical Decision Guide
With so many layers, how do you choose? Use the questions in this chapter as a short checklist.
graph TD
A["I want to remove some telemetry"] --> B{"What exactly do I want to remove?"}
B -->|"A few specific requests, like health checks"| C["Ignore them in the library, or drop with a sampler rule"]
B -->|"One noisy library, like file access"| D["Disable that instrumentation library"]
B -->|"One signal, like all metrics"| E["Set that signal exporter to none"]
B -->|"A sensitive or high-cardinality attribute"| F["Remove it with a processor, a view or the collector"]
B -->|"Everything for this service"| G["Disable the SDK or do not attach the agent"]
B -->|"A noisy part of my own code"| H["Suppress inside that code block"]
D --> I{"Is it an HTTP or messaging library?"}
I -->|"Yes"| J["Check propagation, prefer narrower options"]
I -->|"No"| K["Safe to disable, then verify"]
The five questions
What exactly is the problem?
Name it precisely: “health check traces are 40 percent of our span volume,” not “there is too much telemetry.” A precise problem has a precise switch.
Can I narrow it?
Is it one route, one library, one signal, one attribute or the whole service? Always pick the narrowest description that is still true.
Do I still need propagation here?
If this service sits in the middle of traces that matter, avoid switches that stop context from flowing.
Where is the cheapest safe place?
Earlier layers save more, but are riskier or blunter. Pick the earliest layer that is also safe for your case.
How will I prove it worked?
Decide in advance how you will check, for example by comparing span counts before and after, and by inspecting a trace through the changed service.
A quick lookup table
| I want to remove … | Best first choice | Fallback |
|---|---|---|
| Health check traces | Library URL ignore setting, or sampler rule | Collector filter |
| Noisy file system or low-level spans | Disable that instrumentation library | Collector filter on span names |
| All metrics from a service | Metrics exporter set to none | Views that drop the instruments |
| A high-cardinality metric label | A view that removes the label | Collector attribute processing |
| Sensitive attribute values | Configure the library to not capture them, or a processor to remove them | Collector attribute processing |
| Duplicate spans from two layers | Disable the less useful layer or use the duplicate suppression setting | Collector filter, carefully |
| A chatty internal loop | Code-level suppression | Sampler rule on span name |
| Telemetry from a whole non-critical service | Do not attach the agent | SDK disabled setting |
15A Step-by-Step Example: Silencing Health Check Noise
Let us walk through a complete example, from noticing the problem to proving the fix. The numbers are invented, but the steps are the ones real teams follow.
A team runs an order service behind a load balancer. Their tracing bill has doubled in two months while customer traffic grew by only ten percent. They suspect that something is producing a lot of spans for little value.
Measure what is in the data
The team asks their tracing tool for the top span names by count. The top entry is a span named after the health check address, accounting for roughly 45 percent of all spans from this service. Every other name is far lower.
Confirm that the data is useless
They open a few of these traces. Each has one span, takes a millisecond and always succeeds. No alert, dashboard or investigation uses them. Real failures of the health check are already detected by the load balancer itself.
Choose the narrowest switch
The problem is one specific address on the server side. They choose the HTTP server library’s setting for ignoring addresses, with a pattern that matches exactly that one health check address. They do not disable the whole library, since the service needs normal request tracing and propagation.
Try it in a test environment
They set the ignore pattern in the staging deployment, send a few real requests and a few health check calls, and look at the output on a console or a debug exporter. Health checks produce no spans. Real requests produce the usual spans.
Check that traces are still whole
They send a real request through the gateway, the order service and the payment service, and confirm that the trace has spans from all three in one tree. The ignore pattern did not affect propagation for ordinary requests.
Roll out and measure
They deploy the change to production and compare the span count per minute before and after. It falls by about 45 percent, as predicted, while the number of real order traces is unchanged.
Write it down
They note the change, the reason and the expected effect in their documentation, and share the pattern with other teams that use the same load balancer.
The same seven steps work for almost any suppression task: measure, confirm, choose the narrowest switch, test, check propagation, roll out, and document.
16Suppressing Duplicate Spans
Sometimes the problem is not that there is too much telemetry, but that the same thing is recorded twice. Duplicates confuse readers and double the cost.
Duplicate spans appear when two layers of instrumentation both measure the same operation. There are several common causes.
Two libraries in one stack
A web framework is built on top of a lower-level server, and both are instrumented. Each creates a server span for the same request, producing two nested spans that say the same thing.
Auto and manual together
A developer adds manual spans around a call that the automatic instrumentation already measures. The trace now shows the same call twice.
Proxy plus application
A service mesh sidecar creates spans for each call, and the application also creates spans for the same call. Each has some unique value, but together they produce overlap.
Two agents attached
A vendor agent and an OpenTelemetry agent are both attached by mistake, and both patch the same library, measuring each call twice, sometimes with confusing results.
How to remove duplicates
- Use the built-in duplicate suppression. Some agents, such as the Java agent, have a mechanism, often called a span suppression strategy, that avoids creating a second span of the same kind when one is already active for the same operation. It is usually on by default in a safe form, and some versions allow you to change the strategy. Check your agent’s documentation.
- Disable the less valuable layer. If a lower-level library duplicates a better, higher-level one, disable the lower-level one. Check that propagation is not affected.
- Remove redundant manual spans. If automatic instrumentation already covers a call, remove your own span around it, or turn your span into a child with useful extra attributes.
- Pick one agent. Never attach two tracing agents to the same process. Choose one and remove the other.
- Decide who owns the span in a proxy plus application setup. Often the sidecar owns network-level spans and the application owns business-level spans. Disable the overlapping ones on one side.
Imagine two reporters sent by two newspapers to the same event, each writing the same story. The editor gets two versions of the same article. The fix is to choose one reporter, or to let each cover a different angle. Duplicate suppression is that editorial decision.
Two spans that look alike may carry different attributes. One may have the route, the other the user agent. Before disabling one of them, check what unique value each provides, so you keep the one that is more useful.
17Suppressing Sensitive and High-Cardinality Data
Sometimes you do not want to remove a whole span, only a risky or expensive detail inside it. This is often a better answer than switching off the instrumentation.
Sensitive data in attributes
Automatic instrumentation records attributes such as the full address of a request, the text of a database statement, the names of message queues and sometimes headers. Some of these can contain personal data or secrets: an email address inside a web address, a card number inside a query, an access token in a header. Once recorded, such data is copied to the collector, the backend, backups and every person who can read the traces.
You have several options, from best to last resort.
- Do not capture it at all. Many instrumentations capture request and response headers only when you ask them to. Leave those settings off for sensitive headers. Some database instrumentations sanitize statements by replacing literal values with placeholders, and this is normally on by default. Keep it on.
- Remove it in the SDK. A span processor can remove or mask chosen attributes before the span is exported. This keeps the data inside your process.
- Remove it in the collector. Attribute and transform processors can delete or hash chosen attributes for all services at once.
- Disable the instrumentation. If nothing else works, switching off the library that captures the data is the safest option, with the cost of losing its other telemetry.
High-cardinality data
Cardinality is the number of different values a label can take. A label such as the HTTP method has a handful of values, which is fine. A label such as the full address including a user identifier can have millions of values. For metrics, each unique combination of label values becomes a separate time series, so a high-cardinality label can create millions of series, slow down the backend and inflate the bill.
The remedies are similar. Prefer the route template, such as “orders by identifier,” over the raw address. Use metric views to drop or reduce labels in the SDK. Use collector processors to remove or group values. And avoid putting unbounded values, such as user identifiers or request identifiers, into metric labels in the first place. They belong in traces and logs, which are built for high-detail data.
Illustration: the leaky address
A team notices that their traces contain full web addresses with the customer’s email address in the query string, such as a password reset link. They do not want to disable HTTP tracing. Instead, they add a processor in the collector that removes the query string part of the address attribute from all spans. Traces remain useful, since the route and method are still there, but the private data no longer leaves the collector. This is an invented example to show the thinking.
Data removed in the collector has already traveled over the network from the application. If the data must never leave the application, remove it inside the SDK or do not capture it at all.
18Real-World Use Cases
Here are situations where teams reach for suppression. The scenarios are illustrative examples meant to build intuition, not descriptions of any particular company’s internal systems.
1. Health checks and readiness probes
An orchestrator calls a health address every few seconds on every instance. The team ignores that address in the HTTP server instrumentation, and the span volume drops by a large share at once.
2. A chatty file system library
A Node.js service that reads many small files produces thousands of spans per request from the file system instrumentation. The team disables that single instrumentation and keeps the HTTP and database ones. Traces become readable again.
3. Metrics scrape endpoints
A monitoring server scrapes each application every fifteen seconds. Each scrape would produce a trace. The team ignores the metrics address so that monitoring traffic is not traced.
4. Batch jobs with millions of tiny operations
A nightly import job reads a million rows and would create a million database spans. The developers wrap the loop in code-level suppression and create one span around the whole job, with a count of rows as an attribute.
5. Compliance and privacy
A company handling payments must never store card details outside a secure zone. The team turns off statement capture for the payment database driver, and adds a collector rule that deletes any attribute with certain names, as a second line of defense.
6. Duplicate server spans
After upgrading a framework, traces show two server spans for every request. The team uses the agent’s duplicate suppression setting and removes a manual span that duplicated one made by automatic instrumentation.
7. Emergency performance switch
During an incident, the team suspects that telemetry overhead is adding latency. They switch the SDK off for one service with a single setting, observe the effect, and restore it after confirming that the problem lay elsewhere.
8. Cardinality explosions in metrics
A web framework metric carries the raw address as a label, which multiplies series after a marketing campaign sends traffic to thousands of unique addresses. A metric view removes the label, and the bill returns to normal.
19Design Patterns for Suppression
A design pattern is a proven way of solving a problem that keeps coming back. These are the patterns teams use most often when taming instrumentation.
Start narrow, widen only if needed
Begin with the smallest switch, such as ignoring one route. If the problem remains, move to a wider switch, such as one library. Never begin with the biggest switch.
Allow list for mature services
For stable services, list exactly which instrumentations are on, and turn everything else off. Upgrades then cannot silently add new telemetry.
Drop with a sampler to keep propagation
When a service sits in the middle of important traces, remove noisy requests with a sampler rule rather than by disabling libraries, so context keeps flowing.
Central policy in the collector
Put company-wide rules, such as dropping health checks and removing sensitive attributes, in the collector, so no individual team has to remember them.
Defense in depth
For sensitive data, combine layers: do not capture it at the source, remove it in the SDK, and delete it in the collector. If one layer fails, another catches it.
Summarize instead of suppress
When an inner loop creates thousands of spans, suppress the inner spans and create one outer span with counts and totals as attributes. You keep the story, drop the noise.
Shared base configuration
Put the agreed switches in a shared base image, template or configuration file, so every service starts from the same sensible baseline.
Feature flag for telemetry
Keep a switch that can turn heavy telemetry on temporarily for debugging and off again afterward, without a new release.
The “summarize instead of suppress” pattern in more detail
Imagine a function that processes a file of ten thousand records and calls a database for each one. Automatic instrumentation creates ten thousand database spans. The trace becomes unreadable, and the backend is flooded. If you suppress all of them, you lose information about the loop. The better pattern is to suppress the inner spans, and wrap the whole loop in a single span that records useful summary values: how many records were processed, how many failed, and how long the loop took. One meaningful span replaces ten thousand meaningless ones.
A good meeting note does not copy every sentence spoken. It records the decisions, the actions and the total time. Summarizing instrumentation is the same: keep the headline, drop the transcript.
The “defense in depth” pattern in more detail
For privacy-sensitive data, a single control is fragile. Someone might upgrade a library and a default might change. Someone might add a new attribute without telling anyone. By placing a control at each layer, you ensure that a mistake at one layer is caught by another. The source does not capture, the SDK processor masks anything that slipped through, and the collector deletes any attribute whose name matches a sensitive pattern. The cost is a little extra configuration, and the benefit is far less chance of a data leak.
20Advantages, Disadvantages and Trade-offs
Suppression is helpful, but it is not free. Every piece of telemetry you remove is information you may later wish you had. An honest look at both sides helps you decide how far to go.
Advantages
- Lower cost for storage, network and processing.
- Cleaner, more readable traces and dashboards.
- Less overhead in the application on busy paths.
- Better privacy when sensitive details are not captured.
- Fewer duplicate spans and confusing overlaps.
- Faster searches in the backend because there is less data.
- A quick safety valve when telemetry itself is causing a problem.
Disadvantages
- Lost visibility: a removed span cannot help during a later investigation.
- Broken traces if propagation is affected by mistake.
- Silent failures when a setting name is wrong and nothing changes.
- Hidden configuration that nobody remembers, creating confusion later.
- Inconsistency when different services suppress different things.
- Upgrades can reintroduce or rename switches, undoing your work.
- Rules that are too broad may hide real problems along with the noise.
The central trade-off: savings versus visibility
Every cut moves a slider between “see everything and pay for it” and “see less and pay less.” The best position is not at either end. It is the point where the telemetry you keep answers the questions you actually ask, and the telemetry you drop is the telemetry nobody has ever opened. The test is simple: if nobody has used a certain kind of telemetry in an investigation or a dashboard in the last few months, it is a good candidate for suppression.
Suppress versus sample
It is worth comparing suppression with sampling, because they overlap. Sampling keeps a share of every kind of trace, chosen at random or by rule. Suppression removes specific kinds of telemetry entirely. Use suppression for telemetry that is never useful, such as health checks. Use sampling for telemetry that is useful but too plentiful, such as ordinary customer requests. The two work well together: suppress the useless, sample the plentiful.
| Question | Suppression | Sampling |
|---|---|---|
| What does it remove? | Specific kinds of telemetry, always | A share of all telemetry, by chance or rule |
| Best for | Noise, duplicates, sensitive details | Plentiful but valuable traffic |
| Effect on rare events | None, other events are kept fully | Rare events may be missed |
| Effect on counts | Counts of removed kinds are zero | Counts must be scaled by the rate |
| Configuration style | Names, patterns and rules | A rate or policy |
21Common Mistakes and Anti-patterns
These mistakes appear again and again in real systems. Each looks harmless and each can silently damage your telemetry.
The mistake
Disabling a whole HTTP or messaging instrumentation library to remove a few noisy requests.
Why it hurts
The library also reads and writes trace headers. Switching it off splits traces and creates orphan roots in the services around it.
Better approach
Ignore the specific routes in the library, or drop them with a sampler rule, so the library keeps working for everything else.
The mistake
Setting a suppression option with a misspelled or outdated name and assuming it worked.
Why it hurts
Most SDKs silently ignore unknown settings. The noise continues, but the team believes it is gone. Weeks pass before anyone checks the bill.
Better approach
Always verify the effect in a test environment, and compare telemetry volume before and after in production.
The mistake
Using a very broad pattern to ignore addresses, such as anything that contains a common word.
Why it hurts
Real, important requests that happen to match the pattern disappear. A failing payment endpoint could be hidden by a rule meant for a health check.
Better approach
Match the exact address, and test the pattern against a list of real addresses from your system.
The mistake
Filtering telemetry only in the backend or collector and calling the problem solved.
Why it hurts
The application still creates, records and sends everything. You save storage but still pay for CPU, memory and network. For sensitive data, it has already left the application.
Better approach
Cut as early as is safe. Use the collector for central rules and as a second line of defense, not as the only line.
The mistake
Dropping parent spans while keeping children, or dropping spans from the middle of a trace.
Why it hurts
The remaining spans point to parents that no longer exist. Trace viewers show broken trees, missing roots and confusing timing.
Better approach
Drop whole traces with a sampler, or drop only leaf spans that have no children.
The mistake
Leaving code-level suppression switched on after an error, or wrapping too much code in a suppressed block.
Why it hurts
Everything inside the block, including important calls to other services, becomes invisible, and the trace loses its middle.
Better approach
Keep suppressed blocks small and always scoped so the marker is removed even on failure.
The mistake
Making ad hoc changes in one service without telling anyone, so that services behave differently and nobody remembers why.
Why it hurts
Traces look different across services, investigations are confused, and later engineers undo the change or copy it blindly.
Better approach
Keep suppression settings in version control with a comment explaining the reason, and share common rules through a base configuration.
The mistake
Using suppression to hide a problem that should be fixed, such as silencing a flood of error spans instead of fixing the error.
Why it hurts
The problem remains, but you can no longer see it. Users suffer while the dashboards look calm.
Better approach
Suppress noise that is normal and harmless. Fix noise that signals a defect.
22Best Practices
These habits keep your suppression rules safe, understandable and effective. Think of them as a checklist for every change.
Measure before you cut
Look at real data first: top span names, top metrics, volume by service. Cut what the data shows is large and useless.
Use the narrowest switch
Ignore a route before disabling a library, and disable a library before disabling the SDK.
Protect propagation
Prefer methods that keep context flowing, such as samplers, especially for services in the middle of traces.
Put it in version control
Store settings in deployment files with a comment on the reason. Avoid manual changes on running machines.
Prove the effect
Check in a test environment, then compare span and metric volumes before and after the production rollout.
Share common rules
Define company-wide rules once, in a base configuration or in the collector, instead of in every team’s own way.
Layer your defenses
For sensitive data, do not capture it, mask it in the SDK, and delete it in the collector.
Revisit regularly
Upgrades change names and defaults. Review your rules after each major upgrade and every few months.
A short pre-change checklist
- Can I describe exactly what I want to remove, and why?
- Is there a narrower switch than the one I plan to use?
- Will this affect context propagation for any trace that matters?
- Have I checked the exact name of the setting for my language and version?
- Do I know how I will prove that it worked and that nothing else changed?
- Is the change written down, with its reason, where my team can find it?
- Do I know how to undo it quickly?
23Testing and Validating Your Suppression Rules
A suppression rule that does nothing looks the same as one that works, until the bill arrives. And a rule that does too much looks the same as a quiet system, until an outage. Testing is how you tell the difference.
Test one: the “before and after” count
Before the change, record the number of spans, metric series or log records per minute for the affected service. After the change, record them again, with similar traffic. The difference should match what you expected. If you removed health checks that were 45 percent of spans, the count should drop by about 45 percent. If it does not change, your rule is not working. If it drops much more, your rule is too broad.
Test two: look at the raw output
In a test environment, send the telemetry to a place where you can read it directly. The collector has a debug exporter that prints what it receives, and most SDKs offer a console exporter. Send a request that should be suppressed and one that should not. Confirm that the first produces nothing and the second produces the usual spans, with the usual attributes.
Test three: follow a trace across the changed service
Send a request through a path that includes the changed service. Open the trace in your tracing tool. Check that there is one tree, with the same trace ID from start to finish, and that the changed service’s spans are present as expected. This is the test that catches propagation damage.
Test four: check the edges of the pattern
For rules based on patterns, build a list of addresses or names from real traffic, including near misses. Run them through the pattern and see which are matched. You want exactly the ones you intended and no others. A few minutes spent here prevents a rule from hiding something important.
Test five: check metrics and logs too
Disabling an instrumentation library may also remove metrics or log records that dashboards and alerts depend on. List the dashboards and alert rules that use telemetry from the affected library, and make sure they still show data after the change.
Volume drop matches
The reduction in spans or series matches what you predicted from the data.
Raw output is correct
Suppressed requests produce nothing. Normal requests produce normal telemetry.
Traces stay whole
A request across several services still forms one trace tree.
Pattern edges are right
The rule matches exactly the intended addresses or names.
Dashboards still work
No dashboard or alert silently lost its data source.
Rollback is ready
You know the exact steps to undo the change within minutes.
24Monitoring and Troubleshooting: When Suppression Does Not Behave
Sooner or later a rule will not do what you expect. Either the noise stays, or something important vanishes. Here is a calm way to find out why.
Case one: the noise did not go away
Was the setting actually applied?
Check the running process or container, not just the file you edited. Confirm that the environment variable is present and spelled correctly, and that the service was restarted after the change.
Is the name right for this language and version?
Compare the setting name with the documentation for the exact version in use. Names and patterns change between releases.
Is the telemetry coming from where you think?
Look at the attributes of the noisy spans: which library created them, which service, which side, server or client. You may have silenced the wrong library or the wrong side.
Is a second source producing the same thing?
A proxy, a second agent or a manual span might be creating the same telemetry. Silencing one source leaves the other.
Does the pattern really match?
Test the pattern against the exact string that the library compares, which may include or exclude query parameters, host names or prefixes.
Case two: something important disappeared
Check propagation first
Open a trace that should have crossed the changed service. If it stops there, or a new root begins after it, the change has probably broken context flow.
Look for a pattern that is too broad
Compare the list of suppressed addresses or names with the list of missing telemetry. A pattern may be catching more than intended.
Check for a stuck suppression marker
If code-level suppression was used, look for paths where the marker might not have been removed, such as error handling paths.
Check the collector rules
A central filter rule may be dropping telemetry from services that were never meant to be affected. Review the conditions and test them on sample data.
Undo and bisect
If the cause is unclear, undo the latest change, confirm that telemetry returns, then reapply changes one at a time until the culprit appears.
Measuring the health of your rules over time
Treat suppression as something to observe. Track the volume of telemetry per service per day and watch for sudden changes. A drop that nobody planned is a warning that a rule is too broad or that an upgrade changed a default. A rise after an upgrade may mean that a switch was renamed and no longer applies. A short weekly look at these numbers catches most problems early.
Keep a “known good” example request that exercises the main path of your system, and run it after every change to telemetry settings. If its trace is complete, most of the important parts still work.
25Governance, Security and Cost
Suppression rules are decisions about what your company chooses to see. In a larger organization, those decisions need a little structure.
Governance: who decides what is switched off?
When many teams manage their own services, it is tempting to let each team switch off whatever it likes. That works for a while, then causes trouble: one team disables HTTP instrumentation and breaks traces for three other teams, another hides errors to make a dashboard look clean, and a third has a rule that nobody remembers. A light structure prevents this.
- Company defaults. A platform or observability team defines a baseline: which instrumentations are on, which routes are ignored everywhere, which attributes are always removed.
- Team overrides. Individual teams may adjust within limits, for example ignoring a route unique to their service. Overrides are kept in version control and reviewed.
- Protected switches. Some switches, such as disabling HTTP or messaging libraries, or turning off the SDK in production, may require a review, because of their effect on other teams’ traces.
- A record of reasons. Every rule has a comment explaining the reason, the owner and the date. Rules without a reason get questioned and eventually removed.
Security
- Suppression is not secrecy. Removing a span from a dashboard does not remove the underlying action. Do not use telemetry suppression to hide activity from auditors. Audit logs are a separate system with their own rules.
- Security monitoring needs its own signals. If you suppress telemetry for noisy routes, make sure that security tools do not depend on that telemetry. Suspicious requests to health or admin addresses could be exactly what a security team wants to see.
- Protect the controls. Anyone who can change environment variables or collector rules can hide activity or change what is captured. Limit that access and log changes.
- Remove sensitive data early. Data that never leaves the application cannot be leaked from the collector, the network or the backend.
Privacy
Suppression is one of the strongest privacy tools, because data that is never captured cannot be misused. Review what your instrumentations capture by default, and compare it with your privacy rules. Turn off capture of anything you do not need, and mask or remove the rest at the earliest safe point.
Cost
Telemetry cost is roughly the number of spans, metric series and log records, multiplied by their size and the time they are kept. Suppression reduces the first factor, often dramatically, for very little effort. Typical high-return targets are:
Keep a short list of “top ten sources of telemetry by volume” for each service, and review it every few months. Often the top two or three entries account for most of the cost, and each can be handled with a single narrow rule.
Show teams the cost of their own telemetry. When engineers can see that a single noisy library costs more than a small server, they remove it quickly and gladly.
26Frequently Asked Questions
Short, direct answers to the questions beginners ask most often.
OTEL_SDK_DISABLED variable to true, or do not attach the agent at all. Verify that your SDK honors the variable, since support depends on the language and version.27Summary and Key Takeaways
You started with a security camera that recorded the ceiling fan and ended with a full map of how to quiet OpenTelemetry instrumentation safely. Here is everything that matters in one place.
OpenTelemetry offers many switches at many layers, from turning off the whole SDK, to disabling one library, to ignoring one route, to suppressing inside a block of code, to dropping in samplers and views, to filtering in the collector. The earlier you cut, the more you save, but the greater the risk of breaking something. The wise path is to measure first, choose the narrowest switch that solves the problem, protect trace propagation, test the result and write down the reason. Done well, suppression gives you clearer traces, lower bills, better privacy and a calmer on-call life.
Key Takeaways
- Many layers, no single switch: you can suppress at deployment, SDK, library, request, code, sampler and view, and collector levels. Each has different savings and risks.
- Start narrow: ignore a route before disabling a library, and disable a library before disabling the SDK. Use the biggest switches only for the biggest problems.
- Protect propagation: disabling an HTTP or messaging library can stop trace headers and split traces. Prefer ignore rules and sampler rules, which keep context flowing.
- Know the standard switches:
OTEL_SDK_DISABLEDturns off the SDK, and setting a signal’s exporter to none turns off that signal. Language-specific names for disabling libraries and ignoring URLs differ, so check your version. - Cut early, filter centrally: early cuts save the most, while collector filters give one central place for company rules and a second line of defense for sensitive data.
- Fix, do not hide: suppress harmless noise and duplicates. If the noise signals a real defect, fix the defect.
- Verify and document: compare volumes before and after, follow a trace across the changed service, keep settings in version control and write down why each rule exists.