Before learning the tool, let us feel the problem it solves.

Telemetry is born messy

Telemetry is information that a system sends out about itself: timings, counts, error messages and the path of a request. When your code creates a piece of telemetry, such as a record of one database query, that record is raw. It may be missing useful details, like which cloud region it came from. It may contain things that should never leave the building, like a customer’s email address. It may be one of millions of nearly identical records that nobody needs to keep.

Now imagine sending each record one by one over the network the moment it is created. A busy service might create tens of thousands of records every second. Sending them individually would be like a shop owner driving to the post office for every single letter. It wastes effort, slows down the shop, and clogs the roads.

What we really need

We need a set of steps, placed between creation and storage, that can do several jobs. These jobs include grouping records into batches, adding missing labels, removing sensitive values, throwing away noise, protecting the system from running out of memory, and deciding which records are worth keeping. Doing all this inside your business code would be messy and hard to change. OpenTelemetry’s answer is to make these steps separate, reusable pieces called processors.

A short, honest answer first, and then the important twist.

A processor in OpenTelemetry is a component that sits in the path of telemetry data and does some work on it before it moves to the next step. That work might be changing the data, grouping it, filtering it, delaying it, or protecting the system. A processor does not create the original data, and it does not store the final data. It lives in the middle.

The twist: there are two kinds of processors

This is the point that confuses most beginners. The word “processor” appears in two different places in OpenTelemetry, and they behave differently even though they share a name.

A few words you need before we go deeper.

Signals

A signal is a type of telemetry. OpenTelemetry has three main ones. Traces describe the journey of one request through your system, made of timed steps called spans. Metrics are numbers measured over time, like requests per second. Logs are timestamped text records of events.

The API, the SDK, and the Collector

The API is the set of interfaces your code uses to ask for telemetry, like a wall socket that anything can plug into. The SDK is the real engine behind the API that records and sends the data. The Collector is a separate program that receives telemetry from many sources and forwards it to storage. Processors live in the SDK and in the Collector, but not in the API. The API never processes anything. It only describes what your code may ask for.

The Collector pipeline

The Collector is built from a small number of building blocks, and a processor is only one of them. Understanding the others shows you what a processor is not.

The processors that live inside your application, starting with tracing.

The job of a span processor

Inside the SDK, when your code starts a span, finishes a span, or asks the SDK to shut down, the SDK tells the registered span processors. A span processor is like a receptionist who is told about every visitor arriving and leaving. A processor can be told four main things.

Sampler or processor?

Beginners often mix up a sampler and a span processor. They are different pieces with different jobs. A sampler runs at the very start, when a span is created, and decides whether this span should be recorded at all. A span processor runs afterwards, on spans that exist, and decides what to do with them. In short, the sampler asks “should this exist?” and the processor asks “what do we do with it now?”

The two built-in kinds

Almost every SDK ships with two standard processors that hand finished spans to an exporter.

The most common processor in the world, and the one worth understanding properly.

The batch processor, in both the SDK and the Collector, follows the same simple idea. Finished items go into a waiting line, called a queue. A background worker takes items out of the queue in groups and sends each group to the next step. The queue has a size limit so that it cannot eat all your memory.

flowchart LR
    A["Span ends in application"] --> B["Batch processor"]
    B --> C{"Is the queue full?"}
    C -->|Yes| D["Drop the new span and count it as dropped"]
    C -->|No| E["Add span to queue"]
    E --> F{"Batch size reached or timer expired?"}
    F -->|No| G["Wait in queue"]
    F -->|Yes| H["Background worker takes a batch"]
    H --> I["Exporter sends batch over the network"]
    G --> F
    

The knobs you can turn

Most batch processors have a few settings with sensible defaults. The exact numbers can differ between languages and versions, so always check your own documentation, but the ideas are the same everywhere.

Why dropping data is actually a safety feature

It may sound alarming that a processor can drop data. But consider the alternative. If the backend is slow and the queue could grow without limit, your application would slowly use more and more memory until it crashed. The whole point of OpenTelemetry’s design is that observability must never be the reason your service goes down. A bounded queue chooses to lose a little telemetry rather than risk the real application.

Processors exist for logs too, but metrics work a little differently.

Log record processors

For logs, the SDK has log record processors. They work much like span processors. They are told when a log record is emitted, and they can change it or pass it along. Just like spans, there are simple and batch versions, and production systems normally use the batch kind. A typical use is to add the same shared labels, such as the service name or environment, to every log record.

Metrics do not use processors in the SDK

Here is a surprise. In the SDK, metrics do not flow through a “metric processor.” Instead, two other concepts shape metric data.

In the Collector, however, metrics do pass through processors, just like traces and logs. So if you need to filter or change metrics, you can do it there. This difference is a good reminder to be exact about which “processor” you mean.

The Collector offers many ready-made processors. Here are the ones every beginner should know.

Collector processors are configured, not coded. The operator writes a short configuration file that names the processors to use and sets their options. No application code changes. Some processors are in the core Collector, while many others are in a larger collection often called the contrib distribution. The maturity of each processor varies, so always check its status before depending on it in production.

Let us put all the pieces on one map.

flowchart LR
    subgraph APPS["Applications with OpenTelemetry SDK"]
        A1["Order Service"]
        A2["Payment Service"]
        A3["Inventory Service"]
    end
    A1 --> R["OTLP Receiver"]
    A2 --> R
    A3 --> R
    subgraph COL["OpenTelemetry Collector Pipeline"]
        R --> P1["Memory Limiter"]
        P1 --> P2["Kubernetes Attributes"]
        P2 --> P3["Filter"]
        P3 --> P4["Redaction"]
        P4 --> P5["Tail Sampling"]
        P5 --> P6["Batch"]
    end
    P6 --> E1["Tracing Backend"]
    P6 --> E2["Long Term Archive"]
    

Read the diagram from left to right. Three services send their spans to a receiver. The data then walks through six processors, one after another, in the exact order the operator wrote them. At the far end, exporters send it to different places. Each processor receives the output of the one before it, so if one processor drops a span, every processor after it never sees that span.

One pipeline per signal

In the Collector, you define separate pipelines for traces, metrics and logs. Each pipeline lists its own receivers, processors and exporters. A single processor can be named in several pipelines, but each pipeline gets its own working copy of it. Not every processor supports every signal, so always check which signals a processor can handle.

Agent and gateway layouts

Many teams run Collectors in two layers. A small agent Collector runs next to each application, on the same machine or in the same pod, and does light work like adding host details and batching. A central gateway Collector receives from all the agents and does heavier work like tail sampling and routing. This split keeps your applications light and your heavy logic in one place.

What actually happens to a span as it moves through processors?

sequenceDiagram
    participant App as Application SDK
    participant Rec as Collector Receiver
    participant Mem as Memory Limiter
    participant Fil as Filter
    participant Bat as Batch
    participant Exp as Exporter
    participant Back as Backend
    App->>App: Span ends and SDK batch processor queues it
    App->>Rec: Send batch using OTLP
    Rec->>Mem: Pass data
    Mem->>Mem: Check memory use
    Mem->>Fil: Pass data if memory is healthy
    Fil->>Fil: Drop health check spans
    Fil->>Bat: Pass remaining spans
    Bat->>Bat: Wait for batch size or timer
    Bat->>Exp: Send larger batch
    Exp->>Back: Deliver to storage
    

Step by step

Notice that the same idea, batching, appears twice: once in the SDK and once in the Collector. This is normal. The SDK batch makes the trip from application to Collector efficient, and the Collector batch makes the trip from Collector to backend efficient, after data has been filtered and sampled and the numbers have changed.

Because each processor works on the output of the one before it, the order you choose changes the result.

A sensible order and the reasons behind it

flowchart TD
    O1["1. Memory Limiter: protect the Collector first"] --> O2["2. Enrich: add Kubernetes and cloud details"]
    O2 --> O3["3. Filter: drop data nobody wants"]
    O3 --> O4["4. Redact and Transform: mask or reshape"]
    O4 --> O5["5. Sample: keep only what is valuable"]
    O5 --> O6["6. Batch: group for efficient sending"]
    

The smartest processor is also the one that is easiest to get wrong.

Normally, sampling is decided at the start of a trace, which is called head sampling. It is cheap and simple, but it is blind. At the start, you do not know yet whether this request will fail or be slow. If you keep ten percent at random, you will probably lose most of your interesting traces.

Tail sampling waits until the trace is finished and then decides. A tail sampling processor holds all the spans of each trace in memory for a short waiting time, then applies rules such as “keep every trace with an error,” “keep every trace slower than two seconds,” and “keep five percent of everything else.” This gives you the best of both: low cost and high value.

The hidden requirement

For tail sampling to work, all the spans of one trace must arrive at the same Collector instance. If the spans of one trace are spread across ten Collectors, no single one sees the whole story, and its decision will be wrong. In larger setups, teams place a Collector layer in front that routes spans by trace ID, so that every span with the same trace ID goes to the same sampling Collector. This routing is usually done by a special exporter that balances load using the trace ID.

Five everyday problems and the processor that solves each one.

The same job can often be done in two places. Which should you choose?

A useful rule: do in the SDK what only your code can know, and do in the Collector what is shared, changeable or needs a wider view. Keeping policy decisions out of application code means platform teams can adjust them in minutes without asking dozens of developers to redeploy.

No design is free of cost. Let us look honestly at both sides.

Key trade-offs

Processors sit on the road that all your data travels. That makes them powerful, and sensitive.

The pipeline that watches your systems also needs to be watched.

A processor that quietly drops data or refuses data is doing its job, but you must know it is happening. The Collector exposes its own metrics about its health, and the names of many processor-related metrics begin with a shared prefix that includes the word “processor.” Build a small dashboard and a few alerts around these signals.

• Refused data: a rising count from the memory limiter means the Collector is under pressure and senders are being told to wait.
• Dropped data: data discarded because queues were full or rules removed it. Compare drops caused by your own filter rules with drops caused by overload.
• Batch sizes and send timing: if batches are always tiny, your timeout may be too short. If they are always at the maximum, you may need more capacity.
• Queue depth: a queue that is always nearly full is a warning before data loss begins.
• Collector memory and CPU: heavy processors, especially tail sampling, can push these up.
• Silence alerts: alert when a service that normally sends telemetry suddenly sends none.
• Collector logs: configuration errors and rejected data often appear first in the Collector’s own logs.

Habits worth copying and traps worth avoiding.

Good design patterns

Anti-patterns

A practical checklist for a healthy processing pipeline.

Best practices

• Always use batching. Keep the batch processor in the SDK and in the Collector, and tune sizes and timeouts from real measurements.
• Protect first. Put the memory limiter at the start of every Collector pipeline.
• Keep the order intentional. Follow “protect, enrich, remove, hide, choose, pack” and write down why any exception exists.
• Test with sample data. Before deploying a new rule, run known samples through it in a test Collector and check what comes out.
• Roll out gradually. Apply new filtering or sampling rules to a small slice of traffic first, and compare counts before and after.
• Prefer allow lists for sensitive fields. Decide what you allow, rather than guessing everything you must block.
• Monitor the processors. Alert on refused data, dropped data and silent services.
• Flush on shutdown. Make sure applications call shutdown so the last batch is not lost, especially in short-lived jobs and serverless functions.
• Keep Collector versions current. Processor behaviour and names change over time, so read release notes before upgrades.
• Document your pipeline. A one-page diagram of the processors and why each exists saves hours during incidents.

Common mistakes beginners make

• Confusing the SDK span processor with the Collector span processor.
• Thinking a sampler and a processor are the same thing.
• Defining a processor in the configuration but forgetting to list it in the pipeline, so it never runs.
• Expecting a processor to modify a span after it has ended in the SDK. Ended spans are read-only.
• Forgetting that a processor may not support all three signals.
• Setting a batch timeout so long that engineers think the system is broken because data arrives late.
• Dropping data with a filter and then wondering why dashboards look empty.

How these ideas appear in tools and platforms you may already know.

A typical growth journey

Quick answers to questions beginners ask most.

If you remember only a few things, remember these.