How Do You Suppress or Disable Specific Instrumentation in OpenTelemetry?

How Do You Suppress or Disable Specific Instrumentation in OpenTelemetry?

A plain-English guide to switching off the telemetry you do not want, from a single noisy route to a whole library, at every layer from your application to your collector, without accidentally breaking the traces you do want.

Imagine you install a very eager security camera system in your house. It records the front door, which is useful. But it also records the ceiling fan, the cat’s water bowl, the refrigerator door, the clock on the wall and every flicker of the window curtains. After a week, you have a mountain of footage, a huge storage bill, and no way to find the one clip that actually matters. The cameras are not broken. They are simply recording too much of the wrong things.

OpenTelemetry instrumentation can behave in exactly the same way. Once you switch on automatic instrumentation, it starts recording everything it knows how to record: every web request, every database call, every file read, every health check from the load balancer. Much of it is valuable. Some of it is noise, some of it is expensive, and some of it is private. The skill you need is the skill of choosing what to switch off.

In this guide you will learn what instrumentation is, why you might want to suppress or disable parts of it, and the many layers at which you can do it: from turning off a whole signal, to disabling one library, to ignoring one URL, to filtering data in the collector. You will also learn the most dangerous trap, which is that switching off the wrong thing can quietly break your traces. No prior knowledge is assumed. Every new term is explained the first time it appears.

i
How to read this guide

Read the first five chapters in order, because they build the vocabulary and the map. After that, you can jump straight to the layer you need, or to the chapters on mistakes and the FAQ.

1The Big Picture: When Too Much Telemetry Becomes a Problem

Before we learn how to switch things off, we need to understand why anyone would want to. It starts with how easy it is to produce too much.

When a program runs in production, you cannot stand next to it and watch it. So you ask the program to send you information about itself. This information is called telemetry. The most common kinds are traces, which follow one request through many services, metrics, which are numbers measured over time, and logs, which are timestamped messages.

OpenTelemetry offers a wonderful shortcut: automatic instrumentation. You attach an agent or install a few packages, and your program starts producing telemetry for web servers, database drivers, message queues and many other libraries, without you writing any measuring code. This is a huge time saver. But “everything, automatically” has a flip side. It does not know what matters to you.

How much is too much?

Consider a small online shop. A load balancer checks whether each server is alive by calling a health check address every five seconds. With ten servers, that is two calls per second, or about 172,800 calls per day, and every one of them can produce a trace with several spans. None of those traces will ever help anyone debug anything. Yet they take up network bandwidth, processing time in the application, space in the backend and money on the monthly invoice.

Now add a library that records every single file read, a database driver that records every tiny internal query, and a framework that creates spans for every internal function call. In a busy system, it is entirely possible for more than half of the telemetry to be completely unhelpful.

172,800
health check calls per day from just ten servers in our example
0
of them help debug a real problem
1 line
of configuration is often enough to remove them
Analogy

Think of a newspaper. A good editor does not print every sentence that every reporter ever wrote. The editor cuts the boring parts, removes the repeated parts and keeps the stories that readers need. Suppressing instrumentation is the editor’s job for your telemetry: it is not about hiding the truth, it is about keeping the paper readable and affordable.

The goal is not the most telemetry. The goal is the most useful telemetry per dollar and per millisecond.

2The Basic Vocabulary: Instrumentation, Signals and Agents

A handful of small words appear in every conversation about this topic. Learn them once and the rest becomes easy.

Instrumentation

Instrumentation is code that measures what a program is doing and produces telemetry about it. The word is borrowed from the instruments on a ship or an airplane: the dials, gauges and meters that tell the pilot what is happening. In software, instrumentation creates spans, increments counters and writes log records.

Manual and automatic instrumentation

Manual instrumentation means you, the developer, write the measuring code yourself, for example “start a span called calculate price, do the work, end the span.” You are in complete control, because nothing happens unless you wrote it.

Automatic instrumentation, also called auto-instrumentation, means that a ready-made tool adds measuring code for you, for popular libraries and frameworks. It does this by attaching an agent when the program starts, by wrapping library functions at runtime, or by shipping small add-on packages. You get telemetry from the web framework, the HTTP client, the database driver and more, with almost no effort.

Instrumentation library

Each piece of automatic instrumentation usually lives in its own small package, called an instrumentation library, which knows how to measure one thing, for example one web framework or one database driver. A system may have dozens of them active at once. Each can usually be switched on or off separately, which is the key fact that makes suppression possible.

Signals

Traces, metrics and logs are called signals. A single instrumentation library may produce more than one signal. For example, an HTTP server library may produce a span for each request and also a metric for request duration.

SDK, API and exporter

The API is the set of functions that instrumentation calls to create telemetry. The SDK is the engine behind the API that collects, processes and sends the telemetry. An exporter is the part of the SDK that sends the finished telemetry to another place, such as a collector. Suppression can happen in any of these places, as you will see.

Measuring code

Instrumentation

Code that observes what the program does and produces telemetry about it.

Ready-made

Auto-instrumentation

Measuring code added for you by an agent or add-on packages, covering popular libraries without code changes.

One small piece

Instrumentation library

A package that measures one specific thing, such as one web framework. It can usually be turned off on its own.

Kinds of data

Signals

Traces, metrics and logs. Each can be controlled separately.

Analogy

Think of a building with many security sensors: door sensors, window sensors, motion sensors and smoke detectors. Each sensor is an instrumentation library. The alarm panel in the hallway is the SDK. The phone line to the security company is the exporter. You can unplug one noisy motion sensor without touching the smoke detectors, or you can silence the phone line entirely. Knowing which part you want to quiet is the whole game.

3How Instrumentation Creates Telemetry

To switch something off at the right place, you must first understand the road that telemetry takes from your code to your dashboard.

graph LR
  A["Your Application Code"] --> B["Instrumentation Library"]
  B --> C["OpenTelemetry API"]
  C --> D["SDK: samplers, processors, views"]
  D --> E["Exporter"]
  E --> F["Collector: receivers, processors, exporters"]
  F --> G["Backend and Dashboards"]
    
Figure 1: The road that telemetry travels, and the points where it can be stopped.

Read the figure from left to right. Your application does its normal work, such as handling a web request. An instrumentation library watches that work and calls the OpenTelemetry API to say “a request started” or “a database call finished.” The SDK behind the API decides what to do with those calls: a sampler may decide to drop a trace, processors may change or filter spans, and views may reshape metrics. The exporter then sends whatever is left to a collector, which can filter and change it again before it reaches the backend that stores it and draws your dashboards.

Every box in this road is a place where something can be switched off or filtered. And each choice has a different price. The earlier you stop something, the more you save, because the work, memory and network use are avoided completely. The later you stop it, the more you have already paid for.

What an instrumentation library actually does

  • Hooks in. It attaches itself to the library it watches, for example by wrapping a function in a web framework.
  • Creates spans or measurements. When the watched function runs, it starts and ends spans, adds attributes such as the HTTP method and the route, and records metrics.
  • Handles context. It reads trace headers from incoming requests and writes trace headers into outgoing ones. This is called context propagation, and it is what ties the spans of different services into one trace.
  • Hands data to the API. It does not send anything over the network itself. The SDK and exporter do that.

That third point, context handling, is the one that matters most for suppression. An instrumentation library often does two jobs at once: it produces telemetry and it passes along the trace identity. If you switch off the whole library, you may stop both jobs, and that can break traces in other services. We will give this problem its own chapter.

Analogy

Imagine a postal worker who both writes a log of every parcel and also sticks the tracking label on each parcel. If you tell the worker to stop keeping the log because it is too long, you must be careful not to also stop the labels. Otherwise the parcels travel on with no labels, and the next sorting center cannot tell where they came from.

4Why Suppress or Disable Instrumentation?

There are several very different reasons for switching something off. Knowing your reason helps you choose the right tool.

Noise

Boring, repeated telemetry

Health checks, readiness probes, metrics scrapes and internal heartbeats produce huge numbers of identical traces that nobody ever reads.

Cost

Storage and network bills

Every span has a price: processing, bandwidth, storage and search. Removing low-value spans lowers the bill directly.

Performance

Overhead in the application

Measuring takes a little time and memory. On a very latency-sensitive path, or in a very chatty library, that overhead can matter.

Privacy

Sensitive data in attributes

Some instrumentation captures database statements, URLs or headers that may contain personal or secret data.

Duplicates

Two tools measuring the same thing

A proxy and the application, or two libraries, may both create a span for the same call, making traces confusing and doubling cost.

Stability

A buggy or incompatible library

Occasionally, a specific instrumentation library does not work well with a particular version of a framework, and switching it off is the safest quick fix.

Cardinality

Too many unique labels

Some metrics carry labels with huge numbers of values, such as raw URLs with identifiers. These multiply the number of series and the cost.

Policy

Compliance and ownership rules

A team may be told not to send certain telemetry outside the company, or may not own the library whose behavior is being measured.

Suppress, disable, drop and filter: what is the difference?

People use these words loosely, but there is a useful way to separate them.

  • Disable usually means switching off a whole component, such as an instrumentation library or the entire SDK, so that it does nothing at all.
  • Suppress usually means stopping telemetry for a specific situation while the component stays active, for example “do not create spans while inside this block of code” or “ignore requests to this address.”
  • Drop means that telemetry has been created but is thrown away before it is stored, for example by a sampler or a view.
  • Filter means selecting which telemetry to keep based on rules, for example in a collector.

In everyday speech all four may be called “suppression.” This guide uses the wider meaning, and points out the differences where they matter.

i
A first principle

Switch off at the narrowest level that solves your problem. Turning off one URL pattern is safer than turning off a whole library, and turning off one library is safer than turning off the whole SDK.

5The Layers of Control: A Map

There is no single “off switch” in OpenTelemetry. There are many switches, each at a different layer. This chapter gives you the map, and the following chapters visit each stop.

graph TD
  A["Deployment level: do not attach the agent, operator annotations"] --> B["SDK level: disable the SDK or a signal"]
  B --> C["Instrumentation library level: disable one library"]
  C --> D["Request level: ignore specific URLs or operations"]
  D --> E["Code level: suppress inside a block of code"]
  E --> F["SDK processing: sampler, span processor, metric views"]
  F --> G["Collector level: filter, transform, sample"]
  G --> H["Backend level: ingestion rules and retention"]
    
Figure 2: The layers of control, from the earliest and cheapest cut to the latest and most expensive.

The figure runs from the top, where the cut is earliest, to the bottom, where it is latest. A cut near the top prevents everything below it from happening. A cut near the bottom happens after the work and the transport have already been paid for.

LayerWhat you switch offSavingsRisk to traces
DeploymentThe whole agent or auto-instrumentation for a serviceHighestHigh, the service disappears from traces and may break propagation
SDK or signalThe whole SDK, or all traces, metrics or logsHighHigh for the signal switched off
Instrumentation libraryOne library, such as a file system or one frameworkHigh for that libraryMedium, may stop propagation if it is an HTTP or messaging library
Request levelSpecific URLs, routes or operationsMedium to highLow, only those requests are affected
Code levelTelemetry inside a chosen block of codeMediumLow and very targeted
SDK processingChosen traces or metrics through samplers, processors and viewsMediumLow if the sampler is used, context still travels
CollectorChosen spans, metrics, logs or attributes after they arriveLower, work and network are already spent upstreamLow, but it can cut holes in traces if done carelessly
BackendStored data by rules or ageLowestLow

How to use the map

Ask three questions every time. First: what exactly do I want to stop? A single route? One library? One kind of attribute? Second: do I still need trace propagation through this part? Third: where is the cheapest place that still solves the problem safely? The answers point you to a layer.

Analogy

Think of a water system. You can close the main valve of the whole city, close the valve of one street, close the tap of one house, or put a filter on the kitchen faucet. Closing the city valve is the biggest cut and the biggest risk. A filter on the faucet is the smallest. The wise plumber closes the smallest valve that fixes the leak.

6Level One: Turning Off the SDK or a Whole Signal

The biggest switches are the simplest. They are also the blunt ones, so use them only when you really mean to switch off a lot.

Disabling the whole SDK

The OpenTelemetry specification defines a standard environment variable called OTEL_SDK_DISABLED. When it is set to true, a compliant SDK behaves as though OpenTelemetry were not installed. Instrumentation calls still happen, but the SDK does nothing with them: no spans are recorded, no metrics are kept and nothing is exported. It is the “main fuse” for telemetry.

This is useful in a few cases. You might switch the SDK off in a unit test environment, on a developer’s laptop, in a temporary emergency when telemetry is suspected of harming performance, or in a deployment where no backend is available. Because it is an environment variable, it can be set without changing code, and removed again just as easily.

!
Check your SDK

Support for this variable depends on the language and the SDK version. Some SDKs honor it fully, others only partly. After setting it, verify with a test that nothing is being exported. Never assume.

Turning off one signal

You often want to stop one signal but keep the others. OpenTelemetry defines a standard setting for the exporter of each signal. Setting the exporter to the special value none means “do not export this signal anywhere.”

SettingValueEffect
OTEL_TRACES_EXPORTERnoneTraces are not exported. Metrics and logs continue.
OTEL_METRICS_EXPORTERnoneMetrics are not exported. Traces and logs continue.
OTEL_LOGS_EXPORTERnoneLogs are not exported. Traces and metrics continue.
OTEL_SDK_DISABLEDtrueEverything is switched off.

This is the right tool when a whole signal is not needed. For example, a team that only wants traces can set the metrics exporter to none, so that the application does not spend effort collecting and sending metrics. Remember that setting an exporter to none stops the sending, but in some SDKs the instrumentation may still create the telemetry in memory before it is thrown away. If you need to avoid even the creation, look for a setting that disables the instrumentation itself, which the next chapter covers.

Analogy

Switching off the SDK is like turning off the main electrical switch of a house: everything goes dark. Setting one exporter to none is like turning off the lights in a single floor. The house still works, the wiring is still there, but one part is dark.

When to use these big switches

  • In tests, where telemetry would only add noise and slowness.
  • In a quick emergency, where you need to rule out telemetry as the cause of a performance problem.
  • For a service that really needs no telemetry of one kind, such as no metrics.
  • As a safe default for a new environment until the backend is ready.

They are not the right tool for removing one noisy route or one chatty library. For that, you need a narrower switch.

7Level Two: Disabling Specific Instrumentation Libraries

This is the switch most people are looking for. It turns off one instrumentation library and leaves everything else running.

Every auto-instrumentation distribution has a list of the libraries it can instrument, each with a short name such as “flask,” “redis,” “http” or “kafka.” Most distributions let you either disable chosen names from the full list, or disable everything and enable only the names you want. The exact setting names differ by language, but the idea is the same.

Java agent

Per-library enable switches

The Java agent has a setting for each instrumentation, named with the pattern otel.instrumentation.[name].enabled, or the matching environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED. Setting it to false disables that one. A common default setting lets you switch off all instrumentation first and then switch on only the ones you want.

Python

Disabled instrumentations list

The Python auto-instrumentation tool reads OTEL_PYTHON_DISABLED_INSTRUMENTATIONS, a comma-separated list of the names to skip, such as a web framework or an HTTP client library.

Node.js

Enabled and disabled lists

The Node.js auto-instrumentation bundle reads OTEL_NODE_ENABLED_INSTRUMENTATIONS and OTEL_NODE_DISABLED_INSTRUMENTATIONS. A very common use is disabling the noisy file system instrumentation.

.NET

Per-instrumentation switches

The .NET auto-instrumentation offers environment variables that follow a pattern of signal and instrumentation name, to enable or disable each individual instrumentation. In code-based setups, you simply do not add the instrumentation you do not want.

Go

Opt-in by design

In Go, most instrumentation is added by wrapping handlers and clients on purpose. To suppress it, you do not wrap that handler or client. There is nothing to switch off because nothing was switched on.

Operator

Kubernetes injection control

When using the OpenTelemetry Operator, injection is controlled by annotations on the workload. Removing or changing the annotation stops the agent from being added to that workload.

Two styles: deny list and allow list

  • Deny list. Everything is on, except the names you list. This is easy to start with, but when you upgrade, newly added instrumentations switch themselves on, which can surprise you.
  • Allow list. Everything is off, except the names you list. This is stricter and more predictable. It is a good choice for mature systems where you know exactly what telemetry you want.

Which libraries are commonly disabled?

Library typeWhy people disable itCaution
File system accessCreates many tiny spans for every file read or writeUsually safe to disable, rarely needed for propagation
Low-level DNS and network socketsVery detailed, rarely useful, adds many spansUsually safe
Internal framework layersSpans for every middleware or controller stepCheck dashboards that rely on those spans
Logging appenders and bridgesLogs are already collected another wayMake sure logs still reach their destination
Database driversStatements may contain sensitive data or be too numerousConsider masking instead of disabling
HTTP server or client librariesDuplicate spans, or a bugHigh risk: may stop trace propagation
Messaging librariesToo many spans for high-volume queuesHigh risk: may break the link between producers and consumers
!
Names are not universal

The exact names and settings are specific to each language and each release. Names change, new ones are added, and some are renamed. Always check the documentation for the version you run, and verify by looking at the telemetry afterward. A misspelled name is usually ignored silently, which makes you think you switched something off when you did not.

8Level Three: Ignoring Specific Requests, URLs and Routes

Often you do not want to remove a whole library. You want to remove just the boring requests. This is the most common and the safest kind of suppression.

Instrumentation for web servers and HTTP clients usually lets you give it a list of patterns, or a small function, to decide which requests to ignore. If a request matches, the library simply does not create a span for it. All other requests are still measured as normal.

Typical things to ignore

  • Health check and readiness probe addresses.
  • Address paths that serve metrics for scraping.
  • Static files such as images, style sheets and scripts.
  • Internal status pages and administrative pings.
  • Favicon requests from browsers.

How it is done in different ecosystems

Python

Excluded URLs settings

The Python instrumentations read comma-separated lists of patterns from settings such as OTEL_PYTHON_EXCLUDED_URLS for all libraries, or library-specific variants such as one for the web framework or one for the HTTP client. The patterns are regular expressions matched against the address.

Node.js

Ignore hooks

The Node.js HTTP instrumentation accepts hook functions to ignore incoming and outgoing requests. The function receives the request and returns true when the request should be skipped.

.NET

Filter options

The ASP.NET Core and HTTP client instrumentations have filter options. You supply a function that receives the request and returns whether to record it.

Go

Filters on the HTTP wrapper

The Go HTTP instrumentation wrapper accepts filter options. A filter returns false for requests that should not be traced.

Java

Agent settings and sampling

Java users most often combine agent settings with a sampler rule or a collector filter, because the options for ignoring particular addresses depend on the framework and the version. Check the agent documentation for the current options.

Any language

Sampler rule as a fallback

If the library offers no filter, a custom sampler that drops spans for given addresses achieves the same result, as Chapter 10 explains.

Server side and client side are different

A subtle point: ignoring a URL on the server side means “when my service receives a request for this address, do not trace it.” Ignoring a URL on the client side means “when my service calls this address, do not trace the call.” These are separate settings. A common beginner surprise is to configure the server-side ignore list and then still see spans for outgoing calls to the same address from another service. Decide which side you mean, and set both when needed.

Analogy

Think of a hotel doorman who writes in a visitor book. You can instruct the doorman: “Do not write down the postman.” That is the server side: the hotel ignores a certain kind of visitor. But the postman’s own company also keeps a delivery log. To remove the hotel from the delivery log, you must change the postman’s company’s rule, which is the client side.

i
Tip: match the exact path

Patterns that are too broad can hide real traffic. A pattern meant to skip the health check address should match exactly that address, not every address that happens to contain the word “health.” Test the pattern against real URLs before deploying it.

9Level Four: Suppressing Inside a Block of Code

Sometimes you want telemetry to pause for a specific stretch of code. OpenTelemetry’s SDKs have a built-in idea for this, called context-based suppression.

Recall that the context is the small bag of information that travels with a piece of work. Some SDKs let you put a special marker into that bag, which means “do not create telemetry while this marker is present.” Instrumentation libraries check the context before creating spans, and if they see the marker, they skip their work. When the block of code finishes, the marker is removed, and normal instrumentation resumes.

In the Python ecosystem, this is offered as a helper that you wrap around a section of code. In the JavaScript ecosystem, there is a function that returns a new context in which tracing is suppressed. Other SDKs have similar tools or internal markers. The exact helpers differ between languages and versions, so check your SDK’s documentation.

The classic example: do not trace the tracer

Telemetry exporters send data over the network, often using an HTTP client library. If that HTTP client is itself instrumented, then sending a batch of spans would create a new span for the sending, which would be exported in the next batch, which would create another span, and so on. This feedback loop would flood the system. To prevent it, exporters run their sending inside a suppressed context, so the HTTP client does not trace its own telemetry traffic. You get this protection for free in good SDKs, but you can use the same idea for your own code.

sequenceDiagram
  participant App as Application Code
  participant Ctx as Context
  participant Lib as Instrumented HTTP Client
  App->>Ctx: Mark telemetry as suppressed
  App->>Lib: Call a monitoring endpoint
  Lib->>Ctx: Is suppression marked?
  Ctx-->>Lib: Yes
  Note over Lib: No span is created
  App->>Ctx: Remove the mark
  App->>Lib: Call a business service
  Lib->>Ctx: Is suppression marked?
  Ctx-->>Lib: No
  Note over Lib: Span is created as usual
    
Figure 3: A suppression marker in the context silences instrumentation only while it is present.

Good uses for code-level suppression

  • Internal housekeeping calls. A background task that polls a configuration service every second and would otherwise create thousands of identical traces.
  • Calls to monitoring and telemetry systems. Any code that talks to the observability system itself.
  • Bulk operations. A loop that reads a million rows and would otherwise create a million tiny spans. Suppress inside the loop, and create one span around the whole loop instead.
  • Sensitive sections. A short section of code that handles private data, where you prefer no automatic spans to be created.
Analogy

It is like putting a “Do not disturb” sign on a hotel room door. While the sign is hanging, the cleaning staff leaves the room alone. When you take the sign down, normal service resumes. The sign belongs to the guest, and it only affects that room for that time.

!
Always remove the sign

Suppression must be tied to a block that ends cleanly, even when an error occurs. If the marker is left on by mistake, all later telemetry in that flow disappears silently. Use the language’s built-in structure for this, such as a scoped block, and never set the marker without a guaranteed way to remove it.

10Level Five: Dropping Inside the SDK with Samplers, Processors and Views

The SDK has its own tools for thinning telemetry out. They are the best choice when you want to remove telemetry but keep trace identities flowing.

Samplers: dropping whole traces by rule

A sampler decides, when a span starts, whether the trace will be recorded and exported. A custom sampler can look at the name of the span and its starting attributes, and decide to drop requests for health checks or other noisy routes, while keeping everything else. This is called head-based sampling, because the decision is made at the head of the trace.

The great advantage of dropping through a sampler is that the trace context still travels. A dropped span becomes a lightweight placeholder, the trace ID and span ID are still passed along, and other services still see a consistent picture. Chapter 13 explains why this matters.

Span processors: changing or filtering spans

A span processor receives every span at its start and its end. It can add attributes, remove sensitive attributes, or in some designs filter spans out before export. Processors are handy for scrubbing private data. They are less ideal for dropping spans, because by the time a processor sees a span, the span has already been created, and the application has already paid the cost of creating it. Dropping a parent span in a processor while keeping its children can also leave orphan spans in the trace, so be careful.

Metric views: reshaping or removing metrics

For metrics, the SDK offers views. A view is a rule that matches instruments by name, type or source, and changes how they are handled. Views can drop an instrument entirely, keep only chosen attributes to reduce the number of series, rename a metric or change its aggregation. If one instrumentation library produces a metric with a label that has thousands of values, a view that removes that label can cut the cost sharply without touching the library.

Traces

Sampler

Best for dropping whole traces by rule at the start. Keeps context flowing.

Traces

Span processor

Best for adding, changing or removing attributes. Use with care for dropping.

Metrics

View

Best for removing a metric, removing a label or changing aggregation without touching the instrumentation.

Logs

Log record processor

Best for filtering or changing log records before export, where the SDK supports it.

i
Why SDK-level dropping is a sweet spot

It happens inside your own process, before anything crosses the network, so the savings are large. And it preserves context propagation. For many noisy-route problems, a sampler rule is the best balance of safety and savings.

11Level Six: Filtering in the OpenTelemetry Collector

When you cannot change the application, or when you want one central place to apply rules, the collector is a powerful tool.

The OpenTelemetry Collector is a separate program that receives telemetry from many applications, processes it, and forwards it to one or more backends. Its processing stage is made of small building blocks called processors. Several of them are useful for suppression.

Drop

Filter processor

Drops spans, metrics or logs that match conditions you write, for example spans whose route attribute equals the health check address. Conditions use a small expression language that the collector understands.

Change

Attributes and transform processors

Add, rename, mask or delete attributes. This is the right tool for removing sensitive values or high-cardinality labels from everything passing through.

Choose

Sampling processors

A probabilistic sampler keeps a fixed share of traces by trace ID. A tail sampler keeps traces based on what happened in them, such as errors or slowness.

Limit

Memory limiter and batch

Protect the collector itself and group data efficiently. They do not remove chosen telemetry, but they matter when volume is high.

Strengths of collector-level filtering

  • No application change. You can fix noise in a service you cannot redeploy.
  • One central policy. A single rule can apply to every service in the company.
  • Fast to change. Updating a collector configuration is usually quicker than rebuilding and releasing many services.
  • Language independent. The same rule works regardless of the language of the source service.

Limits of collector-level filtering

  • The work is already done. The application has created, recorded and sent the telemetry. You save backend storage, but not the application’s effort or the network between the application and the collector.
  • It can cut holes in traces. Dropping a parent span while keeping its children, or dropping spans from the middle of a trace, leaves orphans and broken trees. Filter whole traces, or leaf spans that have no children, to stay safe.
  • It cannot undo propagation. The collector sees telemetry after the fact, so it cannot change which trace headers were sent between services.
Analogy

Think of a sorting center in a post office. The center can throw away junk mail before it reaches your house, which saves you from a pile of paper at home. But the junk mail has already been printed, stamped and carried to the center. The senders still paid for printing and delivery up to that point.

12Level Seven: Controlling Instrumentation at Deployment Time

The earliest cut of all happens before the application even starts. If the agent is never attached, no instrumentation ever runs.

Automatic instrumentation has to be introduced into a program somehow. Depending on the language and platform, that is done by adding an agent file to the start-up command, by adding a wrapper command, by installing packages, or by letting a Kubernetes component inject it. Each of these entry points is also an exit point: if you remove it, instrumentation disappears.

Ways instrumentation is introduced, and how to switch it off

How it is introducedHow to suppress itNotes
An agent file added to the start commandRemove the agent from the command, or use the agent’s own enabled settingThe Java agent has a setting to switch the whole agent off, which is handy for quick tests
A wrapper command that starts your programStart the program without the wrapperCommon in the Python world
Packages and start-up codeRemove the registration code or the specific packageCommon in Node.js and .NET
Kubernetes injection by an operatorRemove or change the annotation on the workload, or restrict it to chosen containersKeeps decisions in deployment files, visible to reviewers
A base container image that includes the agentBuild a variant without it, or use settings to disableBe careful when many teams share the base image

Why deployment-level control is attractive

  • Maximum savings. Nothing is attached, nothing runs, no memory is used.
  • Visible and reviewable. The decision lives in the deployment files, not hidden in code.
  • Easy to roll back. Adding the annotation or the agent again restores everything.

Why it is also the riskiest

A service with no instrumentation at all is a hole in every trace that passes through it. It will not extract incoming trace headers and will not inject outgoing ones, so the trace is cut in two at that point. Use deployment-level control only for services that really are not part of any traces you care about, such as isolated batch tools, or for a temporary experiment.

!
Think about the neighbors

Before removing the agent from a service, list the services that call it and the services it calls. If any of them participate in traces that matter, removing the agent creates a gap in the middle of those traces.

Analogy

Imagine a relay race in which one runner decides not to take part. The baton is dropped. The runners before and after are fine, but the race cannot be recorded as one race anymore. A service that opts out of instrumentation is that runner.

13The Hidden Danger: Disabling Telemetry Versus Breaking Propagation

This is the most important chapter of the guide. Many painful incidents come from confusing “stop recording spans” with “stop passing the trace along.”

As we saw earlier, instrumentation libraries often do two jobs. Job one is producing telemetry: creating spans, metrics and logs. Job two is propagating context: reading trace headers from incoming messages and writing them into outgoing ones, so that the trace identity flows from service to service.

When you disable a whole HTTP server or HTTP client library, you usually switch off both jobs at once. The server library no longer reads incoming trace headers, so the service starts a brand new trace for each request. The client library no longer writes trace headers, so the next service cannot continue the trace. Your traces fall apart into fragments.

graph LR
  A["Service A: instrumented"] -->|"trace headers sent"| B["Service B: HTTP library disabled"]
  B -->|"no trace headers sent"| C["Service C: instrumented"]
  A --> T1["Trace 1 contains A only"]
  B --> T2["No spans from B"]
  C --> T3["Trace 2 starts at C as a new root"]
    
Figure 4: Disabling the HTTP library in the middle service splits one trace into two unrelated traces.

In the figure, service B has its HTTP instrumentation switched off. Service A sends trace headers, but B does not read them and produces no spans. When B calls C, it sends no trace headers, so C starts a fresh trace. Instead of one connected journey, the backend now shows two disconnected stories, and nobody can see that they belong together.

Two different ways to remove a span

MethodIs a span created?Is context passed on?Result
Disable the whole HTTP libraryNoUsually noTrace is split
Drop with a samplerA non-recording placeholder, not exportedYes, with the sampled flag turned offTrace consistently absent, no fragments
Ignore a URL in the libraryNo span for that requestDepends on the library, usually the request is treated as untracedFine for requests that are truly isolated, like health checks
Filter in the collectorYes, but removed laterYesTrace intact up to the filter, but you must avoid orphaning

The safe rules

  • Prefer narrow switches. Ignore a route or drop with a sampler, before disabling a whole HTTP or messaging library.
  • If you must disable a library, look for a setting that keeps propagation. Some SDKs let you disable span creation while keeping the propagators running. Check the documentation.
  • Remember the propagators are separate. Context propagation is performed by propagator components configured in the SDK. As long as they are configured and used by something that reads and writes headers, trace identity can still flow. But if the instrumentation that calls the propagators is switched off, nothing calls them.
  • Test after every change. Send a request through a path that includes the changed service and check that the trace is still one tree.
Analogy

Think of a courier who both writes a diary of every parcel and stamps the tracking number on each one. You can tell the courier to stop writing the diary, which is safe. But if you tell the courier to stop doing parcel paperwork entirely, the stamp goes too, and the next warehouse has no idea where the parcel came from. Always ask which of the two jobs you are stopping.

Stopping the diary is safe. Stopping the stamp breaks the chain.

14Deciding Where to Cut: A Practical Decision Guide

With so many layers, how do you choose? Use the questions in this chapter as a short checklist.

graph TD
  A["I want to remove some telemetry"] --> B{"What exactly do I want to remove?"}
  B -->|"A few specific requests, like health checks"| C["Ignore them in the library, or drop with a sampler rule"]
  B -->|"One noisy library, like file access"| D["Disable that instrumentation library"]
  B -->|"One signal, like all metrics"| E["Set that signal exporter to none"]
  B -->|"A sensitive or high-cardinality attribute"| F["Remove it with a processor, a view or the collector"]
  B -->|"Everything for this service"| G["Disable the SDK or do not attach the agent"]
  B -->|"A noisy part of my own code"| H["Suppress inside that code block"]
  D --> I{"Is it an HTTP or messaging library?"}
  I -->|"Yes"| J["Check propagation, prefer narrower options"]
  I -->|"No"| K["Safe to disable, then verify"]
    
Figure 5: A decision guide for choosing the narrowest safe switch.

The five questions

1

What exactly is the problem?

Name it precisely: “health check traces are 40 percent of our span volume,” not “there is too much telemetry.” A precise problem has a precise switch.

2

Can I narrow it?

Is it one route, one library, one signal, one attribute or the whole service? Always pick the narrowest description that is still true.

3

Do I still need propagation here?

If this service sits in the middle of traces that matter, avoid switches that stop context from flowing.

4

Where is the cheapest safe place?

Earlier layers save more, but are riskier or blunter. Pick the earliest layer that is also safe for your case.

5

How will I prove it worked?

Decide in advance how you will check, for example by comparing span counts before and after, and by inspecting a trace through the changed service.

A quick lookup table

I want to remove …Best first choiceFallback
Health check tracesLibrary URL ignore setting, or sampler ruleCollector filter
Noisy file system or low-level spansDisable that instrumentation libraryCollector filter on span names
All metrics from a serviceMetrics exporter set to noneViews that drop the instruments
A high-cardinality metric labelA view that removes the labelCollector attribute processing
Sensitive attribute valuesConfigure the library to not capture them, or a processor to remove themCollector attribute processing
Duplicate spans from two layersDisable the less useful layer or use the duplicate suppression settingCollector filter, carefully
A chatty internal loopCode-level suppressionSampler rule on span name
Telemetry from a whole non-critical serviceDo not attach the agentSDK disabled setting

15A Step-by-Step Example: Silencing Health Check Noise

Let us walk through a complete example, from noticing the problem to proving the fix. The numbers are invented, but the steps are the ones real teams follow.

A team runs an order service behind a load balancer. Their tracing bill has doubled in two months while customer traffic grew by only ten percent. They suspect that something is producing a lot of spans for little value.

1

Measure what is in the data

The team asks their tracing tool for the top span names by count. The top entry is a span named after the health check address, accounting for roughly 45 percent of all spans from this service. Every other name is far lower.

2

Confirm that the data is useless

They open a few of these traces. Each has one span, takes a millisecond and always succeeds. No alert, dashboard or investigation uses them. Real failures of the health check are already detected by the load balancer itself.

3

Choose the narrowest switch

The problem is one specific address on the server side. They choose the HTTP server library’s setting for ignoring addresses, with a pattern that matches exactly that one health check address. They do not disable the whole library, since the service needs normal request tracing and propagation.

4

Try it in a test environment

They set the ignore pattern in the staging deployment, send a few real requests and a few health check calls, and look at the output on a console or a debug exporter. Health checks produce no spans. Real requests produce the usual spans.

5

Check that traces are still whole

They send a real request through the gateway, the order service and the payment service, and confirm that the trace has spans from all three in one tree. The ignore pattern did not affect propagation for ordinary requests.

6

Roll out and measure

They deploy the change to production and compare the span count per minute before and after. It falls by about 45 percent, as predicted, while the number of real order traces is unchanged.

7

Write it down

They note the change, the reason and the expected effect in their documentation, and share the pattern with other teams that use the same load balancer.

i
Tip

The same seven steps work for almost any suppression task: measure, confirm, choose the narrowest switch, test, check propagation, roll out, and document.

16Suppressing Duplicate Spans

Sometimes the problem is not that there is too much telemetry, but that the same thing is recorded twice. Duplicates confuse readers and double the cost.

Duplicate spans appear when two layers of instrumentation both measure the same operation. There are several common causes.

Cause 1

Two libraries in one stack

A web framework is built on top of a lower-level server, and both are instrumented. Each creates a server span for the same request, producing two nested spans that say the same thing.

Cause 2

Auto and manual together

A developer adds manual spans around a call that the automatic instrumentation already measures. The trace now shows the same call twice.

Cause 3

Proxy plus application

A service mesh sidecar creates spans for each call, and the application also creates spans for the same call. Each has some unique value, but together they produce overlap.

Cause 4

Two agents attached

A vendor agent and an OpenTelemetry agent are both attached by mistake, and both patch the same library, measuring each call twice, sometimes with confusing results.

How to remove duplicates

  • Use the built-in duplicate suppression. Some agents, such as the Java agent, have a mechanism, often called a span suppression strategy, that avoids creating a second span of the same kind when one is already active for the same operation. It is usually on by default in a safe form, and some versions allow you to change the strategy. Check your agent’s documentation.
  • Disable the less valuable layer. If a lower-level library duplicates a better, higher-level one, disable the lower-level one. Check that propagation is not affected.
  • Remove redundant manual spans. If automatic instrumentation already covers a call, remove your own span around it, or turn your span into a child with useful extra attributes.
  • Pick one agent. Never attach two tracing agents to the same process. Choose one and remove the other.
  • Decide who owns the span in a proxy plus application setup. Often the sidecar owns network-level spans and the application owns business-level spans. Disable the overlapping ones on one side.
Analogy

Imagine two reporters sent by two newspapers to the same event, each writing the same story. The editor gets two versions of the same article. The fix is to choose one reporter, or to let each cover a different angle. Duplicate suppression is that editorial decision.

!
Look before you cut

Two spans that look alike may carry different attributes. One may have the route, the other the user agent. Before disabling one of them, check what unique value each provides, so you keep the one that is more useful.

17Suppressing Sensitive and High-Cardinality Data

Sometimes you do not want to remove a whole span, only a risky or expensive detail inside it. This is often a better answer than switching off the instrumentation.

Sensitive data in attributes

Automatic instrumentation records attributes such as the full address of a request, the text of a database statement, the names of message queues and sometimes headers. Some of these can contain personal data or secrets: an email address inside a web address, a card number inside a query, an access token in a header. Once recorded, such data is copied to the collector, the backend, backups and every person who can read the traces.

You have several options, from best to last resort.

  • Do not capture it at all. Many instrumentations capture request and response headers only when you ask them to. Leave those settings off for sensitive headers. Some database instrumentations sanitize statements by replacing literal values with placeholders, and this is normally on by default. Keep it on.
  • Remove it in the SDK. A span processor can remove or mask chosen attributes before the span is exported. This keeps the data inside your process.
  • Remove it in the collector. Attribute and transform processors can delete or hash chosen attributes for all services at once.
  • Disable the instrumentation. If nothing else works, switching off the library that captures the data is the safest option, with the cost of losing its other telemetry.

High-cardinality data

Cardinality is the number of different values a label can take. A label such as the HTTP method has a handful of values, which is fine. A label such as the full address including a user identifier can have millions of values. For metrics, each unique combination of label values becomes a separate time series, so a high-cardinality label can create millions of series, slow down the backend and inflate the bill.

The remedies are similar. Prefer the route template, such as “orders by identifier,” over the raw address. Use metric views to drop or reduce labels in the SDK. Use collector processors to remove or group values. And avoid putting unbounded values, such as user identifiers or request identifiers, into metric labels in the first place. They belong in traces and logs, which are built for high-detail data.

Illustration: the leaky address

A team notices that their traces contain full web addresses with the customer’s email address in the query string, such as a password reset link. They do not want to disable HTTP tracing. Instead, they add a processor in the collector that removes the query string part of the address attribute from all spans. Traces remain useful, since the route and method are still there, but the private data no longer leaves the collector. This is an invented example to show the thinking.

!
Remove at the earliest safe point

Data removed in the collector has already traveled over the network from the application. If the data must never leave the application, remove it inside the SDK or do not capture it at all.

18Real-World Use Cases

Here are situations where teams reach for suppression. The scenarios are illustrative examples meant to build intuition, not descriptions of any particular company’s internal systems.

1. Health checks and readiness probes

An orchestrator calls a health address every few seconds on every instance. The team ignores that address in the HTTP server instrumentation, and the span volume drops by a large share at once.

2. A chatty file system library

A Node.js service that reads many small files produces thousands of spans per request from the file system instrumentation. The team disables that single instrumentation and keeps the HTTP and database ones. Traces become readable again.

3. Metrics scrape endpoints

A monitoring server scrapes each application every fifteen seconds. Each scrape would produce a trace. The team ignores the metrics address so that monitoring traffic is not traced.

4. Batch jobs with millions of tiny operations

A nightly import job reads a million rows and would create a million database spans. The developers wrap the loop in code-level suppression and create one span around the whole job, with a count of rows as an attribute.

5. Compliance and privacy

A company handling payments must never store card details outside a secure zone. The team turns off statement capture for the payment database driver, and adds a collector rule that deletes any attribute with certain names, as a second line of defense.

6. Duplicate server spans

After upgrading a framework, traces show two server spans for every request. The team uses the agent’s duplicate suppression setting and removes a manual span that duplicated one made by automatic instrumentation.

7. Emergency performance switch

During an incident, the team suspects that telemetry overhead is adding latency. They switch the SDK off for one service with a single setting, observe the effect, and restore it after confirming that the problem lay elsewhere.

8. Cardinality explosions in metrics

A web framework metric carries the raw address as a label, which multiplies series after a marketing campaign sends traffic to thousands of unique addresses. A metric view removes the label, and the bill returns to normal.

19Design Patterns for Suppression

A design pattern is a proven way of solving a problem that keeps coming back. These are the patterns teams use most often when taming instrumentation.

Pattern

Start narrow, widen only if needed

Begin with the smallest switch, such as ignoring one route. If the problem remains, move to a wider switch, such as one library. Never begin with the biggest switch.

Pattern

Allow list for mature services

For stable services, list exactly which instrumentations are on, and turn everything else off. Upgrades then cannot silently add new telemetry.

Pattern

Drop with a sampler to keep propagation

When a service sits in the middle of important traces, remove noisy requests with a sampler rule rather than by disabling libraries, so context keeps flowing.

Pattern

Central policy in the collector

Put company-wide rules, such as dropping health checks and removing sensitive attributes, in the collector, so no individual team has to remember them.

Pattern

Defense in depth

For sensitive data, combine layers: do not capture it at the source, remove it in the SDK, and delete it in the collector. If one layer fails, another catches it.

Pattern

Summarize instead of suppress

When an inner loop creates thousands of spans, suppress the inner spans and create one outer span with counts and totals as attributes. You keep the story, drop the noise.

Pattern

Shared base configuration

Put the agreed switches in a shared base image, template or configuration file, so every service starts from the same sensible baseline.

Pattern

Feature flag for telemetry

Keep a switch that can turn heavy telemetry on temporarily for debugging and off again afterward, without a new release.

The “summarize instead of suppress” pattern in more detail

Imagine a function that processes a file of ten thousand records and calls a database for each one. Automatic instrumentation creates ten thousand database spans. The trace becomes unreadable, and the backend is flooded. If you suppress all of them, you lose information about the loop. The better pattern is to suppress the inner spans, and wrap the whole loop in a single span that records useful summary values: how many records were processed, how many failed, and how long the loop took. One meaningful span replaces ten thousand meaningless ones.

Analogy

A good meeting note does not copy every sentence spoken. It records the decisions, the actions and the total time. Summarizing instrumentation is the same: keep the headline, drop the transcript.

The “defense in depth” pattern in more detail

For privacy-sensitive data, a single control is fragile. Someone might upgrade a library and a default might change. Someone might add a new attribute without telling anyone. By placing a control at each layer, you ensure that a mistake at one layer is caught by another. The source does not capture, the SDK processor masks anything that slipped through, and the collector deletes any attribute whose name matches a sensitive pattern. The cost is a little extra configuration, and the benefit is far less chance of a data leak.

20Advantages, Disadvantages and Trade-offs

Suppression is helpful, but it is not free. Every piece of telemetry you remove is information you may later wish you had. An honest look at both sides helps you decide how far to go.

Advantages

  • Lower cost for storage, network and processing.
  • Cleaner, more readable traces and dashboards.
  • Less overhead in the application on busy paths.
  • Better privacy when sensitive details are not captured.
  • Fewer duplicate spans and confusing overlaps.
  • Faster searches in the backend because there is less data.
  • A quick safety valve when telemetry itself is causing a problem.

Disadvantages

  • Lost visibility: a removed span cannot help during a later investigation.
  • Broken traces if propagation is affected by mistake.
  • Silent failures when a setting name is wrong and nothing changes.
  • Hidden configuration that nobody remembers, creating confusion later.
  • Inconsistency when different services suppress different things.
  • Upgrades can reintroduce or rename switches, undoing your work.
  • Rules that are too broad may hide real problems along with the noise.

The central trade-off: savings versus visibility

Every cut moves a slider between “see everything and pay for it” and “see less and pay less.” The best position is not at either end. It is the point where the telemetry you keep answers the questions you actually ask, and the telemetry you drop is the telemetry nobody has ever opened. The test is simple: if nobody has used a certain kind of telemetry in an investigation or a dashboard in the last few months, it is a good candidate for suppression.

Suppress versus sample

It is worth comparing suppression with sampling, because they overlap. Sampling keeps a share of every kind of trace, chosen at random or by rule. Suppression removes specific kinds of telemetry entirely. Use suppression for telemetry that is never useful, such as health checks. Use sampling for telemetry that is useful but too plentiful, such as ordinary customer requests. The two work well together: suppress the useless, sample the plentiful.

QuestionSuppressionSampling
What does it remove?Specific kinds of telemetry, alwaysA share of all telemetry, by chance or rule
Best forNoise, duplicates, sensitive detailsPlentiful but valuable traffic
Effect on rare eventsNone, other events are kept fullyRare events may be missed
Effect on countsCounts of removed kinds are zeroCounts must be scaled by the rate
Configuration styleNames, patterns and rulesA rate or policy

21Common Mistakes and Anti-patterns

These mistakes appear again and again in real systems. Each looks harmless and each can silently damage your telemetry.

ANTI-PATTERN 01 Avoid
The mistake

Disabling a whole HTTP or messaging instrumentation library to remove a few noisy requests.

Why it hurts

The library also reads and writes trace headers. Switching it off splits traces and creates orphan roots in the services around it.

Better approach

Ignore the specific routes in the library, or drop them with a sampler rule, so the library keeps working for everything else.

ANTI-PATTERN 02 Avoid
The mistake

Setting a suppression option with a misspelled or outdated name and assuming it worked.

Why it hurts

Most SDKs silently ignore unknown settings. The noise continues, but the team believes it is gone. Weeks pass before anyone checks the bill.

Better approach

Always verify the effect in a test environment, and compare telemetry volume before and after in production.

ANTI-PATTERN 03 Avoid
The mistake

Using a very broad pattern to ignore addresses, such as anything that contains a common word.

Why it hurts

Real, important requests that happen to match the pattern disappear. A failing payment endpoint could be hidden by a rule meant for a health check.

Better approach

Match the exact address, and test the pattern against a list of real addresses from your system.

ANTI-PATTERN 04 Avoid
The mistake

Filtering telemetry only in the backend or collector and calling the problem solved.

Why it hurts

The application still creates, records and sends everything. You save storage but still pay for CPU, memory and network. For sensitive data, it has already left the application.

Better approach

Cut as early as is safe. Use the collector for central rules and as a second line of defense, not as the only line.

ANTI-PATTERN 05 Avoid
The mistake

Dropping parent spans while keeping children, or dropping spans from the middle of a trace.

Why it hurts

The remaining spans point to parents that no longer exist. Trace viewers show broken trees, missing roots and confusing timing.

Better approach

Drop whole traces with a sampler, or drop only leaf spans that have no children.

ANTI-PATTERN 06 Avoid
The mistake

Leaving code-level suppression switched on after an error, or wrapping too much code in a suppressed block.

Why it hurts

Everything inside the block, including important calls to other services, becomes invisible, and the trace loses its middle.

Better approach

Keep suppressed blocks small and always scoped so the marker is removed even on failure.

ANTI-PATTERN 07 Avoid
The mistake

Making ad hoc changes in one service without telling anyone, so that services behave differently and nobody remembers why.

Why it hurts

Traces look different across services, investigations are confused, and later engineers undo the change or copy it blindly.

Better approach

Keep suppression settings in version control with a comment explaining the reason, and share common rules through a base configuration.

ANTI-PATTERN 08 Avoid
The mistake

Using suppression to hide a problem that should be fixed, such as silencing a flood of error spans instead of fixing the error.

Why it hurts

The problem remains, but you can no longer see it. Users suffer while the dashboards look calm.

Better approach

Suppress noise that is normal and harmless. Fix noise that signals a defect.

22Best Practices

These habits keep your suppression rules safe, understandable and effective. Think of them as a checklist for every change.

Evidence

Measure before you cut

Look at real data first: top span names, top metrics, volume by service. Cut what the data shows is large and useless.

Scope

Use the narrowest switch

Ignore a route before disabling a library, and disable a library before disabling the SDK.

Safety

Protect propagation

Prefer methods that keep context flowing, such as samplers, especially for services in the middle of traces.

Config

Put it in version control

Store settings in deployment files with a comment on the reason. Avoid manual changes on running machines.

Verify

Prove the effect

Check in a test environment, then compare span and metric volumes before and after the production rollout.

Consistency

Share common rules

Define company-wide rules once, in a base configuration or in the collector, instead of in every team’s own way.

Privacy

Layer your defenses

For sensitive data, do not capture it, mask it in the SDK, and delete it in the collector.

Review

Revisit regularly

Upgrades change names and defaults. Review your rules after each major upgrade and every few months.

A short pre-change checklist

  • Can I describe exactly what I want to remove, and why?
  • Is there a narrower switch than the one I plan to use?
  • Will this affect context propagation for any trace that matters?
  • Have I checked the exact name of the setting for my language and version?
  • Do I know how I will prove that it worked and that nothing else changed?
  • Is the change written down, with its reason, where my team can find it?
  • Do I know how to undo it quickly?

23Testing and Validating Your Suppression Rules

A suppression rule that does nothing looks the same as one that works, until the bill arrives. And a rule that does too much looks the same as a quiet system, until an outage. Testing is how you tell the difference.

Test one: the “before and after” count

Before the change, record the number of spans, metric series or log records per minute for the affected service. After the change, record them again, with similar traffic. The difference should match what you expected. If you removed health checks that were 45 percent of spans, the count should drop by about 45 percent. If it does not change, your rule is not working. If it drops much more, your rule is too broad.

Test two: look at the raw output

In a test environment, send the telemetry to a place where you can read it directly. The collector has a debug exporter that prints what it receives, and most SDKs offer a console exporter. Send a request that should be suppressed and one that should not. Confirm that the first produces nothing and the second produces the usual spans, with the usual attributes.

Test three: follow a trace across the changed service

Send a request through a path that includes the changed service. Open the trace in your tracing tool. Check that there is one tree, with the same trace ID from start to finish, and that the changed service’s spans are present as expected. This is the test that catches propagation damage.

Test four: check the edges of the pattern

For rules based on patterns, build a list of addresses or names from real traffic, including near misses. Run them through the pattern and see which are matched. You want exactly the ones you intended and no others. A few minutes spent here prevents a rule from hiding something important.

Test five: check metrics and logs too

Disabling an instrumentation library may also remove metrics or log records that dashboards and alerts depend on. List the dashboards and alert rules that use telemetry from the affected library, and make sure they still show data after the change.

Check 1

Volume drop matches

The reduction in spans or series matches what you predicted from the data.

Check 2

Raw output is correct

Suppressed requests produce nothing. Normal requests produce normal telemetry.

Check 3

Traces stay whole

A request across several services still forms one trace tree.

Check 4

Pattern edges are right

The rule matches exactly the intended addresses or names.

Check 5

Dashboards still work

No dashboard or alert silently lost its data source.

Check 6

Rollback is ready

You know the exact steps to undo the change within minutes.

24Monitoring and Troubleshooting: When Suppression Does Not Behave

Sooner or later a rule will not do what you expect. Either the noise stays, or something important vanishes. Here is a calm way to find out why.

Case one: the noise did not go away

1

Was the setting actually applied?

Check the running process or container, not just the file you edited. Confirm that the environment variable is present and spelled correctly, and that the service was restarted after the change.

2

Is the name right for this language and version?

Compare the setting name with the documentation for the exact version in use. Names and patterns change between releases.

3

Is the telemetry coming from where you think?

Look at the attributes of the noisy spans: which library created them, which service, which side, server or client. You may have silenced the wrong library or the wrong side.

4

Is a second source producing the same thing?

A proxy, a second agent or a manual span might be creating the same telemetry. Silencing one source leaves the other.

5

Does the pattern really match?

Test the pattern against the exact string that the library compares, which may include or exclude query parameters, host names or prefixes.

Case two: something important disappeared

1

Check propagation first

Open a trace that should have crossed the changed service. If it stops there, or a new root begins after it, the change has probably broken context flow.

2

Look for a pattern that is too broad

Compare the list of suppressed addresses or names with the list of missing telemetry. A pattern may be catching more than intended.

3

Check for a stuck suppression marker

If code-level suppression was used, look for paths where the marker might not have been removed, such as error handling paths.

4

Check the collector rules

A central filter rule may be dropping telemetry from services that were never meant to be affected. Review the conditions and test them on sample data.

5

Undo and bisect

If the cause is unclear, undo the latest change, confirm that telemetry returns, then reapply changes one at a time until the culprit appears.

Measuring the health of your rules over time

Treat suppression as something to observe. Track the volume of telemetry per service per day and watch for sudden changes. A drop that nobody planned is a warning that a rule is too broad or that an upgrade changed a default. A rise after an upgrade may mean that a switch was renamed and no longer applies. A short weekly look at these numbers catches most problems early.

i
Tip

Keep a “known good” example request that exercises the main path of your system, and run it after every change to telemetry settings. If its trace is complete, most of the important parts still work.

25Governance, Security and Cost

Suppression rules are decisions about what your company chooses to see. In a larger organization, those decisions need a little structure.

Governance: who decides what is switched off?

When many teams manage their own services, it is tempting to let each team switch off whatever it likes. That works for a while, then causes trouble: one team disables HTTP instrumentation and breaks traces for three other teams, another hides errors to make a dashboard look clean, and a third has a rule that nobody remembers. A light structure prevents this.

  • Company defaults. A platform or observability team defines a baseline: which instrumentations are on, which routes are ignored everywhere, which attributes are always removed.
  • Team overrides. Individual teams may adjust within limits, for example ignoring a route unique to their service. Overrides are kept in version control and reviewed.
  • Protected switches. Some switches, such as disabling HTTP or messaging libraries, or turning off the SDK in production, may require a review, because of their effect on other teams’ traces.
  • A record of reasons. Every rule has a comment explaining the reason, the owner and the date. Rules without a reason get questioned and eventually removed.

Security

  • Suppression is not secrecy. Removing a span from a dashboard does not remove the underlying action. Do not use telemetry suppression to hide activity from auditors. Audit logs are a separate system with their own rules.
  • Security monitoring needs its own signals. If you suppress telemetry for noisy routes, make sure that security tools do not depend on that telemetry. Suspicious requests to health or admin addresses could be exactly what a security team wants to see.
  • Protect the controls. Anyone who can change environment variables or collector rules can hide activity or change what is captured. Limit that access and log changes.
  • Remove sensitive data early. Data that never leaves the application cannot be leaked from the collector, the network or the backend.

Privacy

Suppression is one of the strongest privacy tools, because data that is never captured cannot be misused. Review what your instrumentations capture by default, and compare it with your privacy rules. Turn off capture of anything you do not need, and mask or remove the rest at the earliest safe point.

Cost

Telemetry cost is roughly the number of spans, metric series and log records, multiplied by their size and the time they are kept. Suppression reduces the first factor, often dramatically, for very little effort. Typical high-return targets are:

Health checks
often a large share of spans from busy services
Low-level spans
file, network and internal calls with little insight
Label values
unbounded labels that multiply metric series

Keep a short list of “top ten sources of telemetry by volume” for each service, and review it every few months. Often the top two or three entries account for most of the cost, and each can be handled with a single narrow rule.

i
Tip

Show teams the cost of their own telemetry. When engineers can see that a single noisy library costs more than a small server, they remove it quickly and gladly.

26Frequently Asked Questions

Short, direct answers to the questions beginners ask most often.

Q1How do I turn off all OpenTelemetry telemetry for a service?
Set the standard OTEL_SDK_DISABLED variable to true, or do not attach the agent at all. Verify that your SDK honors the variable, since support depends on the language and version.
Q2How do I turn off only metrics, or only traces?
Set the exporter for that signal to none, using the standard exporter settings for traces, metrics or logs. The other signals continue to work.
Q3How do I disable one specific instrumentation library?
Use the language’s own setting: a per-instrumentation enable switch in the Java agent, a disabled list in Python and Node.js, per-instrumentation switches in .NET auto-instrumentation, or simply not adding the instrumentation in code-based setups such as Go.
Q4How do I stop health check requests from being traced?
Use the HTTP library’s ignore or filter option for that address, or a sampler rule, or a collector filter as a fallback. Prefer the first two because they save work in the application.
Q5Will disabling an instrumentation library break my traces?
It can, if the library is an HTTP or messaging library that also handles trace headers. Other libraries, such as file system instrumentation, are usually safe to disable. Always test with a request that crosses several services.
Q6What is the difference between disabling and sampling?
Disabling removes specific kinds of telemetry entirely. Sampling keeps a share of all telemetry, chosen at random or by rule. A sampler also keeps trace context flowing even for dropped traces, which disabling a library often does not.
Q7Why did my setting have no effect?
The most common causes are a misspelled or outdated name, a service that was not restarted, a setting placed in the wrong place, or noisy telemetry coming from a different library or a second source. SDKs usually ignore unknown settings silently, so always verify.
Q8What is context-based suppression?
It is a way to mark the current context so that instrumentation inside a block of code does not create telemetry. It is used, for example, so that exporters do not trace their own network calls. Several SDKs offer helpers for it, with different names.
Q9Can I remove telemetry without changing the application?
Yes, in two main ways: environment variable settings for agents and auto-instrumentation, and filters in the OpenTelemetry Collector. The collector cannot save application effort, only downstream cost.
Q10How do I remove sensitive values from spans?
Best: configure the instrumentation not to capture them. Next: remove them in the SDK with a span processor. Then: delete them in the collector with attribute or transform processors. As a last resort: disable the instrumentation that captures them.
Q11How do I reduce metric cardinality caused by one instrumentation?
Use a metric view to remove or reduce the high-cardinality label, or process the metric in the collector. Prefer route templates over raw addresses at the source if the instrumentation allows.
Q12What if two layers create duplicate spans?
Use the agent’s duplicate suppression setting if it has one, disable the less useful layer, remove redundant manual spans, and never attach two tracing agents to the same process.
Q13Is it safe to filter spans in the collector?
It is safe if you drop whole traces or leaf spans. Dropping parents while keeping children, or dropping spans from the middle of a trace, leaves orphans and broken trees.
Q14Should I use an allow list or a deny list for instrumentations?
A deny list is quick to start with, but new instrumentations may switch themselves on after upgrades. An allow list is more predictable and suits mature services where you know exactly what you want.
Q15Does suppression improve performance?
Yes, especially when you stop creating spans in very busy code paths. The saving is larger at earlier layers, because the work of creating, recording and sending the telemetry is avoided entirely.
Q16Can I turn instrumentation back on temporarily for debugging?
Yes, and it is a good practice. Keep the switches in configuration so that you can enable detailed telemetry for a short time and then switch it off again without a new release.
Q17How do I stop an exporter from tracing its own traffic?
Good SDKs already do this by sending telemetry inside a suppressed context. If you write a custom exporter or call monitoring systems from your own code, use the SDK’s suppression helper around those calls.
Q18How do I know that my suppression worked?
Compare telemetry volume before and after, read the raw output in a test environment, and follow a trace through the changed service to confirm it is still one tree. Do not rely only on the fact that a setting was applied.

27Summary and Key Takeaways

You started with a security camera that recorded the ceiling fan and ended with a full map of how to quiet OpenTelemetry instrumentation safely. Here is everything that matters in one place.

OpenTelemetry offers many switches at many layers, from turning off the whole SDK, to disabling one library, to ignoring one route, to suppressing inside a block of code, to dropping in samplers and views, to filtering in the collector. The earlier you cut, the more you save, but the greater the risk of breaking something. The wise path is to measure first, choose the narrowest switch that solves the problem, protect trace propagation, test the result and write down the reason. Done well, suppression gives you clearer traces, lower bills, better privacy and a calmer on-call life.

Key Takeaways

  • Many layers, no single switch: you can suppress at deployment, SDK, library, request, code, sampler and view, and collector levels. Each has different savings and risks.
  • Start narrow: ignore a route before disabling a library, and disable a library before disabling the SDK. Use the biggest switches only for the biggest problems.
  • Protect propagation: disabling an HTTP or messaging library can stop trace headers and split traces. Prefer ignore rules and sampler rules, which keep context flowing.
  • Know the standard switches: OTEL_SDK_DISABLED turns off the SDK, and setting a signal’s exporter to none turns off that signal. Language-specific names for disabling libraries and ignoring URLs differ, so check your version.
  • Cut early, filter centrally: early cuts save the most, while collector filters give one central place for company rules and a second line of defense for sensitive data.
  • Fix, do not hide: suppress harmless noise and duplicates. If the noise signals a real defect, fix the defect.
  • Verify and document: compare volumes before and after, follow a trace across the changed service, keep settings in version control and write down why each rule exists.