If you only have two minutes, this chapter gives you the whole difference. The rest of the guide explains why it is true and how to act on it.

The Simple Span Processor sends each span to the exporter immediately, one at a time, right when the span ends. The code that ended the span waits while the export happens. Nothing is stored in between. It is easy to understand and easy to debug, but it slows your application and makes many tiny network calls.

The Batch Span Processor puts each ended span into a waiting line in memory, called a queue. A background worker takes spans out of the queue in groups and sends each group together. Your application does almost no waiting. In exchange, spans sit in memory for a short time, and if the line gets too long, extra spans are thrown away.

Before comparing the two processors, it helps to see why a processor exists at all. The reason is that a span lives and dies inside your program, and nobody can read it there.

When your application handles a request, the OpenTelemetry tracer creates spans as work begins and finishes them as work ends. Each finished span holds a name, a start time, an end time, a status, and a set of small notes called attributes. All of this sits in your program’s memory. If your program stops, the memory is gone. And even while it runs, nobody outside the program can read that memory.

So the spans must be carried out to a place where people can read them. That place might be a tracing backend, an OpenTelemetry Collector, or just your terminal screen. The part that actually sends spans to such a place is called an exporter. The exporter knows how to speak a particular language, such as the OTLP format over the network, and how to reach a particular destination.

Here is the gap. The tracer knows when a span ends. The exporter knows how to send. Something has to sit between them and decide when and how many at a time. That in-between part is the span processor. Without it, the tracer would have to guess how to talk to exporters, and every exporter would have to guess when to send. Having a processor in the middle gives you one clear place to make timing decisions.

Why timing matters so much

Sending data over a network is slow compared with ordinary work inside a program. Reading from memory takes billionths of a second. Sending a message to another machine takes thousandths of a second or more, which is a million times longer. If your program stops to send every single span, those delays add up fast. A request that creates twenty spans could pay for twenty network trips.

A handful of words appear in every conversation about processors. Learn these once and everything else becomes easy to follow.

Two small ideas hiding in the table

First, a span has two important moments: the moment it starts and the moment it ends. Processors are told about both. Only after a span ends is it complete and ready to export, because only then does it know its duration.

Second, a span processor and an exporter are different things that are easy to mix up. The processor decides when. The exporter decides how and where. You can pair either processor with almost any exporter. The Simple processor with a console exporter prints spans to your screen immediately. The Batch processor with an OTLP exporter sends groups of spans to a Collector.

A picture of the whole path makes the rest of the guide much easier. Follow a single span from birth to the backend.

flowchart LR
    CODE["Your application code"] --> TR["Tracer creates a span"]
    TR --> SP["Span starts and later ends"]
    SP --> PROC["Span processor"]
    PROC --> EXP["Span exporter"]
    EXP --> NET["OTLP over the network"]
    NET --> COL["Collector or backend"]
    COL --> UI["Trace viewer"]

Walking through the picture

Your application code asks a tracer to start a span. The tracer creates the span object and tells every registered processor that a span has started. Your code does its work. When the work is done, your code ends the span. The span now has a duration, and the tracer tells every registered processor that the span has ended.

What each processor does at that moment is the whole difference we are studying. The Simple processor hands the span to its exporter right away. The Batch processor adds the span to its queue. Either way, the exporter eventually turns the span into a message, such as OTLP, and sends it across the network to a Collector or a backend. The backend stores it, and a trace viewer such as a web dashboard draws it for a human.

Notice that the processor is not in the way of span creation. Creating a span is very cheap and happens in memory. The processor only matters at the end, when the question becomes how to ship the finished span.

Every span processor, built in or custom, promises to respond to the same four moments. Knowing them explains why both processors look so similar from the outside.

Why only the end matters for sending

A span can only be exported when it has ended, because a running span has no final duration and may still change. So the on-start moment is used for other jobs, such as stamping every span with a common attribute. The exporting processors, Simple and Batch, both ignore the on-start moment and act on the on-end moment.

Why flush and shutdown are in the contract

The Simple processor has nothing waiting, so flush and shutdown are quick for it. The Batch processor can have hundreds of spans in its queue, so these two moments are vital. They are the way you tell the Batch processor, “Stop waiting for the timer and send everything now.” Forgetting to call them is one of the most common causes of missing spans, a topic we return to in later chapters.

The Simple processor has almost no inner machinery. That is its charm, and also its limit.

When a span ends, the Simple processor does the following. First, it checks whether the span is sampled, meaning that the sampling rules decided this trace should be recorded. Unsampled spans are skipped. Second, it passes the span to the exporter by calling its export function with a list holding that one span. Third, it waits for the exporter to finish, in most implementations, so that spans are exported one at a time and in order. Then the call returns, and your application code continues.

sequenceDiagram
    participant App as Application thread
    participant SP as Simple processor
    participant EX as Exporter
    participant BE as Backend or Collector
    App->>SP: Span ends
    SP->>EX: Export one span
    EX->>BE: Network call with one span
    BE-->>EX: Acknowledge
    EX-->>SP: Done
    SP-->>App: Return and continue

Reading the sequence

Follow the arrows from top to bottom. The application thread ends a span and, without any pause, finds itself inside the processor, then inside the exporter, then waiting on the network. Only after the backend answers does control travel back up the chain. Every arrow on the way down and on the way back is time during which your application is not doing its real job.

Different language implementations differ in small details. Some call the exporter on the same thread and wait. Some hold a lock so that two threads cannot export at the same moment. The OpenTelemetry specification says that the Simple processor must not let export calls overlap, so the exporter can assume it is never called by two threads at once. The shared idea stays the same: the work happens right there, right then, on the thread that ended the span.

What the Simple processor does not have

It has no queue, no timer, no background worker, and no settings to tune. This is why it is called simple. There is nothing to configure, nothing waiting in memory, and nothing to forget to flush. If you ask it to flush, it has nothing to send. If you ask it to shut down, it only asks the exporter to shut down.

Every design trades one good thing for another. The Simple processor trades speed for clarity, and it is worth being honest about both sides.

How slow can it get?

Suppose a single export takes ten milliseconds, which is a modest figure for a call across a data center. A request that ends fifteen spans would add roughly one hundred fifty milliseconds of waiting, even though the real work might take forty. In this made-up example, tracing has made the service almost four times slower. If the backend has a bad moment and each export takes a full second, the same request would stall for fifteen seconds. Your users would blame your application, not your telemetry.

This effect is the reason the Simple processor is called a testing and learning tool. The cost is invisible when you have one request per second on a laptop. It becomes painful when you have a thousand requests per second.

The Batch processor adds three new parts to the story: a queue, a timer, and a background worker. Together they take sending out of the application’s way.

When a span ends, the Batch processor checks whether it is sampled. If it is, the processor places the span into an in-memory queue and returns right away. This step takes a tiny fraction of a millisecond. Your application thread goes back to its work.

Meanwhile, a separate background worker, usually a thread or an equivalent in the language, watches the queue. It wakes up in two situations. One is when a timer fires after a set delay. The other is when the queue has grown large enough to fill a whole batch. The worker then removes up to a maximum number of spans from the queue, hands them to the exporter as one list, and waits for the exporter to finish. Then it goes back to watching.

flowchart LR
    A1["Request thread 1 ends a span"] --> Q["In-memory queue"]
    A2["Request thread 2 ends a span"] --> Q
    A3["Request thread 3 ends a span"] --> Q
    Q --> W["Background worker"]
    T["Timer fires"] --> W
    F["Queue holds a full batch"] --> W
    W --> E["Exporter sends one batch"]
    E --> B["Collector or backend"]
    Q -. "queue full" .-> D["New spans dropped"]

Reading the picture

On the left, several request threads each finish spans and drop them into the same queue. Because the drop is a quick in-memory action, none of them waits for the network. In the middle, a single worker is woken by either the timer or a full batch. On the right, the worker gives the batch to the exporter, which sends it as one network message. The dotted line at the bottom is the safety valve: if the queue is already full when a new span arrives, the new span is dropped instead of making the application wait.

Why one worker is enough

Sending a batch of five hundred spans in one message is much cheaper than sending five hundred messages. So a single worker can keep up with an enormous number of spans. Also, one worker means the exporter sees only one export at a time, which keeps the exporter’s job simple.

The Batch processor has settings that the Simple processor does not. Four of them matter most, and they are the same across languages even if the names differ slightly.

Two rules that keep the settings sensible

The first rule is that the maximum batch size should never be larger than the maximum queue size. A batch cannot contain more spans than the line can hold. Many SDKs quietly adjust a bad combination, but do not count on it. The second rule is that the queue should be several times larger than one batch. If the queue is only barely bigger than a batch, a short pause in sending fills it up and spans are dropped.

Where the settings come from

Most SDKs let you set these values in code when you create the processor. The OpenTelemetry project also defines environment variables for them. Their names start with OTEL_BSP_, where BSP stands for batch span processor, followed by the setting name, such as the schedule delay, the export timeout, the maximum queue size, and the maximum export batch size. Using environment variables is popular because you can change behavior without rebuilding the application.

The Batch processor makes a deliberate choice that surprises many newcomers. When it cannot keep up, it throws spans away rather than slowing your application.

Imagine the worker is busy sending a batch, and the backend is slow. Meanwhile, request threads keep ending spans and adding them to the queue. The queue grows. If it reaches its maximum size, the next span that arrives has nowhere to go. The processor has two options. It could make the request thread wait until there is room, which would bring back the very slowness we wanted to escape. Or it could discard the new span. The OpenTelemetry design chooses to discard, because observability should never be the reason your product becomes unavailable.

What a drop looks like to you

Dropped spans do not raise an error in your application. A trace may arrive with missing pieces, or a whole request may not appear at all. Some SDKs log a warning or expose a counter of dropped spans. Others stay silent. This is why watching the queue, a topic of a later chapter, is so important. Without it you may spend hours wondering why a trace has a hole in it.

Why drops usually happen in bursts

A queue with 2,048 slots sounds large, but a busy service can fill it quickly. Suppose a service ends 4,000 spans per second during a traffic spike, and the exporter can send only 2,000 per second. The queue gains 2,000 spans every second and would be full in roughly one second. After that, half of all spans are dropped until the spike passes. Drops mean that you demanded more output than the exporter and network could deliver.

The worker can wake for three different reasons. Understanding each one explains why spans arrive when they do.

Reason one: the timer

Every few seconds, by default five, the worker wakes up and sends whatever is in the queue, up to the batch size limit. This guarantees that even a quiet service eventually reports its spans. Without a timer, a service that creates only three spans per minute would never fill a batch and would keep its spans forever.

Reason two: a full batch

When enough spans have piled up to fill a whole batch, the worker does not wait for the timer. It sends right away. This keeps busy services from letting the queue grow when there is work to do. So on a busy service, sends happen much more often than once every five seconds, and the timer matters little.

Reason three: flush or shutdown

When your code calls flush, or when the program shuts down, the worker sends everything it has, in as many batches as needed, as fast as it can. This is the only way to be sure the queue is empty at a chosen moment.

A small example with made-up numbers

A service ends 20 spans per second. With a batch size of 512, it would take about 25 seconds to fill one batch. So the five-second timer fires first, and each send carries roughly 100 spans. Another service ends 2,000 spans per second. A batch of 512 fills in about a quarter of a second, so sends happen about four times per second and the timer rarely fires. The same settings behave differently depending on traffic, which is a nice feature.

Now that both are understood on their own, this chapter lines them up across the questions that matter in real projects.

The central trade

If you read the table carefully, you see that the two processors make opposite trades on two axes. On the axis of application speed, Batch wins by a huge margin. On the axis of never losing a span at the moment of a crash, Simple wins, because it never holds anything back. You cannot have both. Production systems almost always choose speed and stability for the application, since a missing span is a minor loss and a slow checkout is a business problem.

The biggest practical difference is invisible in a list of settings. It is about which thread pays the price of sending.

A thread is a single line of work inside a program. A web server might use hundreds of threads at once, each serving one user’s request. When a thread is waiting for the network, it cannot do anything else. In many servers the number of threads is limited. If many threads wait on slow exports, there are fewer free threads for new requests, and users see delays or errors.

With the Simple processor, the request thread is the one that waits. Tracing therefore uses up your capacity to serve users. With the Batch processor, the request thread only adds a span to the queue. The background worker, which is separate from your capacity to serve users, pays for the network. Your server can handle just as many requests with tracing on as with tracing off, apart from a tiny cost.

The effect on tail latency

People who run busy systems care about tail latency, the slowest few percent of requests. Under the Simple processor, a single slow export makes one request slow. If the backend has a hiccup, many requests become slow at once. The tail of your latency chart then reflects the telemetry system’s health instead of your application’s health. This is a surprising and unpleasant coupling, and it is the main reason experienced engineers avoid the Simple processor outside of testing.

flowchart TB
    subgraph SIMPLE["Simple processor"]
      S1["Request thread does work"] --> S2["Request thread exports span and waits"]
      S2 --> S3["Request thread responds to user"]
    end
    subgraph BATCH["Batch processor"]
      B1["Request thread does work"] --> B2["Request thread enqueues span"]
      B2 --> B3["Request thread responds to user"]
      B2 -.-> B4["Worker exports later on its own thread"]
    end

Reading the picture

In the top box, the export step stands between doing the work and answering the user, so the user’s wait includes it. In the bottom box, the request thread finishes right after the quick enqueue step. The worker’s export, shown with a dotted arrow, happens off to the side and the user never feels it.

Numbers make ideas stick. The following example is invented for teaching, and the exact figures will differ in your system.

Case A: Simple processor

Each request must pay for 12 exports at 8 milliseconds each, which is 96 milliseconds of extra waiting. If real work takes 30 milliseconds, a request now takes 126 milliseconds, more than four times longer. The service needs many more threads to serve the same traffic, because each thread is held much longer. If the backend slows to 40 milliseconds per export, each request adds 480 milliseconds, and the service may run out of threads and begin rejecting users. The 3,600 network calls per second are also heavy for the backend.

Case B: Batch processor with defaults

The service produces 3,600 spans per second, so a batch of 512 fills in about 0.14 seconds. The worker sends about seven batches per second. Even if each batch takes 20 milliseconds to send, the worker is busy for only 140 milliseconds per second, which is comfortable. Request threads add 12 spans each, costing a tiny amount of time. Latency is nearly the same as with tracing off.

Case C: a backend hiccup for 10 seconds

The backend stops answering for 10 seconds. Spans keep arriving at 3,600 per second, but the queue holds only 2,048, which is about 0.57 seconds of traffic. So within the first second, the queue fills, and most spans for the next 9 or so seconds are dropped. Users notice nothing. The team later sees a gap in the traces. They could raise the queue size, but 10 seconds of spans would need around 36,000 slots, which costs memory. A better answer might be to sample fewer traces, or to place a Collector close to the service that absorbs short backend outages.

Nothing is free. The Batch processor spends a little memory to save a lot of time, and it is worth knowing what that exchange looks like.

Memory

A queue of 2,048 spans is not just 2,048 small numbers. Each span holds its name, its identifiers, timestamps, attributes, events, and links. A span with a few attributes might take a kilobyte or two. A span with large attributes, such as a long query text, can take much more. So a full queue may occupy several megabytes, or tens of megabytes in unusual cases. For most services this is small, but for tiny containers with strict limits, it deserves a thought. The Simple processor holds nothing, so it has no such cost.

CPU

The exporter must turn spans into a message, which takes processor time. Batching gives it an advantage here. Packing 512 spans into one message shares overhead such as headers and connection setup. Compressing a bigger message is also usually more efficient than compressing many tiny ones. So, counter-intuitively, the Batch processor often uses less total processor time than the Simple one, even though it has extra moving parts.

Network

One network message per span means many small packets and many connection round trips. One message per batch means fewer, larger packets, which use the network and the receiving system more efficiently. Receivers such as Collectors are designed with batches in mind and run best when they receive them.

Systems are judged by how they behave on bad days. Here are the common failures and how each processor responds.

Retries and what happens to failed batches

Many exporters retry a failed send a few times with growing pauses, especially for errors that look temporary. The retry behavior belongs to the exporter, not to the processor. Retries help with short hiccups. They also keep the worker busy longer, so the queue grows while it waits. In the end, if the send still fails, the batch is dropped. The span processors themselves do not store spans on disk, so there is no durable safety net.

A reasonable safety net

To protect against backend trouble without paying a latency cost, teams often send spans to a nearby Collector first. The Collector is a separate program with its own queues, its own retry behavior, and sometimes disk-backed buffers. The application’s Batch processor only has to hand batches to the local Collector, which is fast and nearly always available.

The Batch processor’s one real weakness is that spans waiting in the queue exist only in memory. When the program ends, so do they, unless you flush.

Two ways a program ends

A graceful exit is when your program is told to stop and has time to tidy up, such as when a deployment system sends a polite stop signal. A hard crash is when the program dies suddenly, because of a memory failure, a forced kill, or a power loss. For a graceful exit, you can call shutdown on the tracer provider, and the processor will flush its queue first. For a hard crash, nothing runs, and whatever sat in the queue is gone.

flowchart TB
    START["Program receives stop signal"] --> HOOK{"Is a shutdown hook registered?"}
    HOOK -- "yes" --> FL["Provider shutdown flushes the batch queue"]
    FL --> EXPORT["Exporter sends remaining spans"]
    EXPORT --> EXIT["Program exits cleanly"]
    HOOK -- "no" --> LOST["Queued spans are lost"]
    LOST --> EXIT2["Program exits"]
    CRASH["Hard crash or forced kill"] --> LOST

Why the lost spans often matter most

The spans that were in flight right before a crash are exactly the ones you want to read when investigating the crash. That irony is real. Teams reduce the risk in two ways. They call shutdown on every graceful exit, and they keep the scheduled delay short enough that the exposure window is small. A shorter delay means fewer spans are at risk, at the price of more frequent, smaller sends.

What the Simple processor offers here

The Simple processor has already sent everything that ended before the crash. Only the span being exported at that exact instant is at risk. That is a real advantage for crash debugging, but it does not justify the latency cost on production services. A better answer for crash visibility is to log crash details separately, and to keep the Batch delay modest.

Some programs live for a few seconds, or even a few milliseconds. For them, the timing of the Batch processor can quietly swallow every span.

Command-line tools and batch jobs

Imagine a small program that starts, does a task in two seconds, and exits. With the Batch processor and a five-second delay, the timer never fires before the program ends. If the program does not flush or shut down properly, the queued spans vanish, and the trace viewer shows nothing. The fix is to call shutdown or flush on the tracer provider right before exiting. This one habit prevents most “my spans never show up” tickets for scripts.

Serverless functions

Serverless platforms run your code only when a request arrives, and they may freeze the process the moment your function returns. A frozen process cannot run its background worker. So spans left in the Batch queue wait until the next time the function is thawed, which may be minutes later, or they may be lost if the environment is discarded. To avoid this, serverless setups usually flush at the end of every invocation, before returning the response. Some teams use the Simple processor here, accepting the small cost because each invocation creates few spans. Others flush a Batch processor explicitly and keep the cost low. The right choice depends on span count per invocation and the platform’s behavior, so check the platform’s documentation.

The Simple processor earns its place in the toolbox. In a test or in a first experiment, immediate and ordered output is exactly what you want.

Learning and first experiments

When you are trying OpenTelemetry for the first time, you want to run your program and see spans appear in the terminal at once. Pair the Simple processor with a console exporter, which prints spans as text, and you get that instantly. There is no waiting five seconds and no wondering whether a timer will fire.

Automated tests

A test of instrumentation wants to run some code and then check which spans were created. Most SDKs provide an in-memory exporter that keeps spans in a list. Paired with the Simple processor, the list is complete the moment the code under test finishes. With the Batch processor, the test must remember to flush first, or the list may be empty or incomplete and the test becomes unreliable. Using Simple keeps tests short and stable.

Local debugging of a single request

When you chase one strange request on your own machine, you may want spans to show up in the backend in real time. The Simple processor does that. Just do not leave it enabled when you commit the change to a shared environment.

Instead of memorizing rules, follow a path of questions. Most real cases are settled in two or three steps.

flowchart TB
    START["Which span processor should I use?"] --> Q1{"Is this a test or a local experiment?"}
    Q1 -- "yes" --> SIMPLE["Use the Simple processor"]
    Q1 -- "no" --> Q2{"Does the program run longer than a few seconds?"}
    Q2 -- "yes" --> BATCH["Use the Batch processor and wire in shutdown"]
    Q2 -- "no" --> Q3{"Serverless or short job?"}
    Q3 -- "yes" --> Q4{"Few spans per run?"}
    Q4 -- "yes" --> EITHER["Simple, or Batch with an explicit flush"]
    Q4 -- "no" --> BATCH2["Batch with an explicit flush before exit"]
    Q3 -- "no" --> BATCH

Walking the path

The first question catches the easy case. If it is a test or a local experiment, Simple is the friendly choice. If not, ask how long the program lives. A web service that runs for days should use Batch and make sure shutdown is wired in. A short program raises a second question. In a serverless function or a short job that makes only a few spans, the Simple processor is acceptable, because the extra waiting is small and nothing is left behind. If the program makes many spans per run, use Batch and flush before the exit.

Beyond the path: two extra checks

First, ask how bad it would be to lose a few spans. For most services, a few missing spans are fine. For auditing needs, tracing is not the right tool anyway. Second, ask where the backend is. A backend across the world pushes the Batch processor toward larger batches and a local Collector, and pushes the Simple processor into pain.

A tracer provider can hold a list of processors. Each processor sees every span event, and this opens useful combinations.

Processors that change spans on start

Besides the two exporting processors, you can write a custom processor that acts on the start moment. For example, it might add the current deployment version or the name of a feature flag to every span as it begins. Because it works at start, the span is still open and can be changed. Such a processor does not export anything, so it is paired with a Simple or Batch processor that does.

Processors that filter or redact

A custom processor can also decide to hide or alter data before it reaches the exporter. For example, it might remove an attribute that should not be shared. Whether this is possible on the end moment depends on the SDK, since ended spans are read-only. Some SDKs offer a wrapping approach, where one processor passes a modified copy of the span to the next. If you need dependable removal of sensitive fields, a common and more reliable place is the Collector, which has processors designed for the purpose.

Fan-out to several destinations

You can register one Batch processor for your production backend and a second Batch or Simple processor for another destination, such as a local file. Each has its own queue, so a slow destination affects only its own queue. This is a clean way to isolate trouble.

flowchart LR
    SPAN["Span events"] --> ENR["Enriching processor on start"]
    SPAN --> BP["Batch processor"]
    SPAN --> SP["Simple processor"]
    BP --> EX1["OTLP exporter to Collector"]
    SP --> EX2["Console exporter for debugging"]

In this picture, all three processors receive each span event. The enriching processor changes spans as they start. The Batch processor sends to the Collector. The Simple processor prints to the console, which might be turned on only for a debugging session.

The word batch appears in two different places in the OpenTelemetry world, and they are easy to confuse.

The Batch Span Processor lives inside your application’s SDK. It is the one this guide is about. It groups spans before they leave your program.

The OpenTelemetry Collector is a separate program that receives telemetry, changes it, and forwards it. The Collector also has a component named the batch processor, which groups telemetry before it is sent onward to the next destination. Although the names are close, they work at different places on the path, with different settings, and they solve related but separate problems.

Do I need both?

Commonly, yes. The SDK processor protects your application from the network. The Collector’s own batching protects the Collector’s exporters and the final backend from a flood of small messages that came from many applications. They are two links in the chain, each doing a separate job. The Collector’s components and defaults change from release to release, so check its current documentation before copying a configuration.

Sampling and processing are neighbors in the pipeline. Understanding their order stops a lot of confusion about missing spans.

Sampling is the decision of which traces to record. A sampler makes this decision when a span starts. If the sampler says “do not record,” the span is either not created at all, or created as a lightweight placeholder that is never exported. Only spans marked as sampled continue on to be exported.

The processors receive start and end events, but the export processors, Simple and Batch, export only sampled spans. This means sampling reduces the load on both processors. With a sampling rate of ten percent, the Batch queue sees ten percent of the spans, so it fills much more slowly. With the Simple processor, ten percent sampling means ten percent as many expensive exports, which is a big improvement, though still not enough to make Simple a production choice.

A common misunderstanding

Some people think that the Batch processor decides which spans to keep when the queue is crowded. It does not. It drops the newest arrivals when full, without looking at their importance. If you want to keep all errors and slow traces and drop the boring ones, you need a smarter sampling strategy, such as tail sampling in a Collector, where the whole trace is visible before the decision.

When many threads end spans at once, what order do spans leave in? The answer is gentler than many people fear, and it matters less than they think.

Order inside the Simple processor

Because the Simple processor exports at the moment of ending, the order of exports follows the order in which spans end, as far as threads can be ordered at all. When two threads end spans at nearly the same instant, one of them waits for the other, so exports never overlap. This keeps things orderly, though it can serialize your application threads, which is another reason for lower throughput.

Order inside the Batch processor

The queue preserves the order in which spans were added, so batches usually follow end order. But different applications, different threads, and retries mean the backend can still receive data in a slightly different order than it was produced. Backends are built to handle this. They sort spans by their timestamps and by their parent and child links, not by arrival order.

Why parent spans often arrive after children

A parent span ends after all its children, because it covers their whole duration. Since processors work on ended spans, the children are exported before the parent. A viewer must therefore wait for the parent, or draw the trace with a missing root for a moment. This is normal behavior, and backends cope with it by waiting briefly before showing a trace as complete.

sequenceDiagram
    participant C1 as Child span A
    participant C2 as Child span B
    participant P as Parent span
    participant Q as Processor
    C1->>Q: Ends first
    C2->>Q: Ends second
    P->>Q: Ends last
    Note over Q: Children reach the exporter before the parent

The picture shows two child spans ending first, then the parent. Whatever the processor, the exporter receives them in that order. A trace viewer reconstructs the tree afterward by following parent links, so the arrival order does not harm the final picture.

The thing that reports on your system needs reporting on too. A silent processor is one of the hardest problems to find.

What to watch

Where these numbers come from

Some SDK versions can expose the processor’s own metrics, such as queue size and processed spans, through OpenTelemetry’s metrics, and many log warnings when spans are dropped. Support differs across languages and versions, so check what yours offers. If the SDK does not provide them, a fallback is to watch for gaps from the other side. Compare the number of requests your service handled, taken from your web server’s own metrics, with the number of root spans that arrived at the backend. A persistent gap suggests drops or sampling.

If you use a Collector

The Collector exposes its own metrics about data received, refused, and sent. Watching them tells you whether your batches are arriving, and whether the next link in the chain is also struggling. Seeing trouble in both places points to the backend. Trouble only at the application side points to the network between the application and the Collector, or to sizing.

Processors handle every span your application creates, so they sit at a sensitive crossroads. A few careful habits keep that crossroads safe.

What is inside spans

Spans can carry attributes such as user identifiers, query text, URLs, and header values. If sensitive information such as a password, a card number, or a health detail ends up in a span, then the processor will faithfully deliver it to every destination. The processor does not know what is secret. The rule is to keep sensitive values out of spans in the first place, and to add a safety net where possible, such as removing known fields in a custom processor or in the Collector.

Spans waiting in memory

The Batch queue holds spans in memory for a short time. This is not a major security risk by itself, since the same memory already holds your data. But if a memory dump of the process is taken for debugging, queued spans are in it. Treat such dumps as sensitive.

Protecting the path out

• Encrypt in transit. Use TLS between the exporter and the Collector or backend, especially across networks you do not control.
• Authenticate the sender. Receivers should accept spans only from known senders, using tokens or certificates.
• Limit sizes. A sensible maximum batch size and attribute size limits protect against oversized messages and accidental data leaks of huge values.
• Prefer a local Collector. Applications talk to a Collector in the same trusted network, and only the Collector holds credentials for the outside world.

A denial of service angle

Because a Simple processor ties request speed to the backend, an attacker who can slow your backend, or flood your tracing with traffic, may indirectly slow your application. The Batch processor’s drop-instead-of-wait policy reduces this risk, since a flooded queue never blocks users.

Experienced teams repeat the same few habits. Each pattern below solves a real, common problem.

Why the first pattern deserves extra attention

Putting a Collector next to the application changes the failure story. If the far-away backend goes down for an hour, the Collector can buffer and retry, and the application never notices. The application’s Batch processor talks to something nearby that answers within a millisecond or two. This moves the difficult part of reliability out of every application and into one shared, well-tuned program.

The following habits look harmless and cause trouble. Learning them early saves many late-night investigations.

This chapter is the checklist to keep near your keyboard.

Best practices

• Use the Batch processor for every service that handles real traffic.
• Use the Simple processor only for tests, demos, and local debugging.
• Call shutdown on graceful exit, and flush at the end of short-lived work.
• Keep the default settings until you have a measured reason to change them.
• Keep the maximum batch size well below the queue size, and below what the receiver accepts.
• Send to a nearby Collector rather than straight across the internet.
• Use sampling to control volume before you resize queues.
• Monitor queue size, drops, and export duration.
• Keep sensitive data out of span attributes.

Common mistakes and their cures

The stories below are invented for teaching. They describe situations that teams meet often, and no real company is meant.

When spans are missing, work from the inside out. The checklist below follows the path a span takes, so the first break you find is the cause.

flowchart TB
    START["Spans are missing at the backend"] --> A{"Was the trace sampled?"}
    A -- "no" --> A1["Adjust the sampler or the sampling rate"]
    A -- "yes" --> B{"Does the program exit quickly?"}
    B -- "yes" --> B1["Add shutdown or flush before exit"]
    B -- "no" --> C{"Are drops reported or the queue full?"}
    C -- "yes" --> C1["Reduce volume or speed up exports then resize"]
    C -- "no" --> D{"Do export errors appear in logs?"}
    D -- "yes" --> D1["Fix endpoint or credentials or TLS or size limits"]
    D -- "no" --> E{"Is the Collector receiving data?"}
    E -- "no" --> E1["Check network path and ports"]
    E -- "yes" --> E2["Check Collector pipelines and the backend"]

Explaining the checklist

Start with sampling, because it is the cheapest to check and the most common cause. If traces were never sampled, nothing downstream matters. Next, ask whether the program ends before the timer can fire, which is the classic short-job problem. Third, look for drops or a full queue, which means output exceeds capacity. Fourth, search the application logs for export errors, such as a wrong address, a rejected credential, a failed secure connection, or a message that is too large. If none of these shows a problem, check the Collector and the backend, in that order.

A useful trick is to temporarily add a second processor, a Simple one with a console exporter, to see whether spans are being created and ended at all. If they print on the screen, the problem is downstream of the processor. If they do not, the problem is in your instrumentation or sampling. Remove the extra processor when you are done.

To make every earlier idea concrete, here is one span followed from the moment it ends until the backend acknowledges it.

The span does not travel alone

The span shares this journey with hundreds of neighbors. Its fate is tied to theirs: if the batch succeeds, all succeed, and if it fails, all fail together. This is one more reason to keep batches modest. A giant batch that fails loses a giant amount of data at once, while a smaller batch loses less per failure and is easier for receivers to accept.

Where each cost appears

Look again at the seven stages. Stages one to three happen on your application thread and take almost no time. Stages four to seven happen outside your application’s request path. The only real waiting a request ever sees is the brief handoff in stage three. This is the entire design benefit in one sentence: move the slow stages away from the people who are waiting.

The two processors exist in every mainstream OpenTelemetry SDK, but each language has its own style and its own small surprises.

The same ideas, different packaging

Whether you use Java, Python, JavaScript, Go, .NET, or another language, you will find a simple processor and a batch processor with the names you now know. The batch version takes the same four settings, though names and units differ a little. Some use milliseconds, others seconds. Some call the delay an interval. Reading the documentation of your own language for the precise names takes a minute and avoids a lot of confusion.

Automatic setup and agents

Many teams do not create processors by hand. They use an auto-instrumentation agent or an auto-configuration feature, which reads environment variables and builds a standard pipeline. In such setups, the default span processor is almost always the Batch processor, paired with an OTLP exporter. The environment variables that begin with OTEL_BSP_ let you tune the four settings without changing code. Variables that begin with OTEL_TRACES_EXPORTER pick the exporter. Check the documentation for your agent, because some agents offer an option to select the simple processor for debugging.

Frameworks that manage lifecycle

Some application frameworks create the tracer provider for you and call shutdown when the application stops. This is helpful and removes the need to wire your own hook. Others leave it to you. When you move to a new framework, one of the first checks should be whether shutdown is called, and the easiest way to test it is to stop the service and see whether the very last spans arrive.

A span processor decides when finished spans leave your program. The Simple Span Processor sends each span at once on the thread that ended it, which is easy to understand and ideal for learning and tests, but it ties your application’s speed to the network and the backend. The Batch Span Processor collects spans in a queue and sends them in groups from a background worker, which keeps your application fast and your network calls few, at the cost of some memory, a short delay, possible drops under overload, and the loss of queued spans on a hard crash. Both are only a step in the pipeline. Sampling decides which spans exist, the processor decides when to send them, and the exporter decides how and where.