Before we talk about blocking, we need to know what is being blocked. This chapter builds the vocabulary for everything that follows.

When you build a website, a mobile app, or any online service, it does not run on one computer sitting on your desk. It runs on many computers, often hundreds or thousands, spread across data centers. You cannot stand next to each machine and watch what it is doing. So the software has to describe its own behavior by sending out a steady stream of small notes. Those notes are called telemetry. The word comes from Greek roots meaning “measuring from far away.”

Telemetry comes in three main flavors, which OpenTelemetry calls signals.

OpenTelemetry, usually shortened to OTel, is a free, open standard and a collection of tools for creating and moving telemetry in the same way no matter which programming language or monitoring product you use. It is a project of the Cloud Native Computing Foundation, the same non-profit home that hosts Kubernetes. Because it is vendor neutral, you can write your instrumentation once and change your monitoring product later without rewriting your code.

The OTel API is the small set of calls your code makes to say “a step started” or “a number changed.” The SDK is the engine behind that API. The Collector is an optional but very common middle stop. And the act of sending telemetry out of the application toward another place is called exporting. Exporting is where our story takes place.

An export pipeline is a chain of stations. A problem at any station can slow every station before it.

Telemetry does not teleport. Each span, metric point or log line travels through a series of stations before a human can look at it on a screen. We call that whole chain an export pipeline. Let us follow one span, the record of a single step in a customer request, from birth to dashboard.

flowchart LR
A["Your App Code"] --> B["Tracer in the SDK"]
B --> C["Batch Processor Queue"]
C --> D["Exporter Worker"]
D --> E["Network Connection"]
E --> F["OpenTelemetry Collector"]
F --> G["Observability Backend"]
G --> H["Dashboard and Alerts"]
    

Here is what each station does, in simple terms.

• Your app code does its normal work, like checking a shopping cart, and a span is created for that step.
• The tracer is the part of the SDK that fills in the span with timing and labels, called attributes. When the step ends, the span is handed onward.
• The batch processor is a holding area. It collects finished spans into a waiting line, called a queue, so they can be sent in groups instead of one at a time.
• The exporter takes a group from the queue and sends it across the network.
• The Collector receives groups from many applications, may filter or enrich them, and passes them on.
• The backend is the storage and search system, such as Jaeger, Grafana Tempo, or a commercial monitoring product. It is where the dashboards read from.

Why batching exists at all

You might ask why the SDK does not simply send each span the moment it finishes. The answer is cost. Every network trip has fixed overhead: opening a connection, wrapping the data, waiting for a reply. If your service creates five thousand spans per second, five thousand tiny trips would waste huge effort. Grouping spans into batches of a few hundred makes each trip worth the overhead. This is a good idea, and it is also exactly what creates the conditions for head-of-line blocking, because the batch becomes the unit of waiting.

The pipeline repeats for every signal

Traces, metrics and logs each have their own path. Spans go through a batch span processor. Logs go through a batch log processor. Metrics usually go through a reader that wakes up on a timer, collects current values, and exports them. These paths are separate from each other inside the SDK, which turns out to be an important protection that we will use later.

A plain definition, three everyday pictures, and the two ingredients that always cause it.

In any waiting line, the person at the very front is the head of the line. If that person takes a very long time, everyone behind them waits too, even if those people only need a few seconds. Head-of-line blocking, often shortened to HOL blocking, is exactly this effect in computer systems: one slow or stuck item at the front of a queue holds up every item queued behind it.

Two ingredients are always present. First, the line is strictly ordered: items leave in the order they arrived, so a slow item cannot be skipped. Second, there is a single shared path, such as one counter or one courier, so nobody else can take a faster route. Remove either ingredient and the problem softens. Add more counters and the milk customer can move. Let customers with few items use an express lane and the milk customer escapes entirely.

Where engineers already know it

Head-of-line blocking is not new, and it is not unique to OpenTelemetry. Older versions of the web protocol, HTTP/1.1, could handle only one request at a time on each connection, so a slow download blocked the small ones behind it. Network switches can suffer it when packets for a busy destination hold up packets headed elsewhere. Even TCP, the reliable delivery layer under most internet traffic, delivers bytes strictly in order, so one lost packet makes all the later bytes wait until it is resent. Engineers invented newer protocols and designs such as HTTP/2, parallel connections and express lanes largely to fight this single problem.

Why telemetry is especially vulnerable

Telemetry pipelines are built to be cheap and invisible to the application. They run on the same machines as the work that pays the bills. They buffer data in memory. They are usually designed with one worker per exporter. All of those good design choices produce the two ingredients above, an ordered queue and a shared path. And since telemetry is written when everything is busy and healthy, the pipeline faces its biggest load at the moments when something unusual is happening and people most want the data.

Eight common hiding places, from inside your application all the way to the network.

The risk is not in one single spot. It shows up at many places along the pipeline, and each one has a different cause and a different fix. Here is a map of the eight most common ones. Later chapters take each in turn.

The batch processor is the most common place where head-of-line blocking starts. Let us open it up.

The batch span processor sits between the code that finishes a span and the exporter that ships it. It has three jobs: accept spans quickly so the application is never slowed down, group them into batches, and hand each batch to the exporter. Its design follows the OpenTelemetry specification, so the behavior is very similar in Java, Go, Python, JavaScript and other languages.

flowchart TB
S["A span ends in your code"] --> Q{"Is the queue full?"}
Q -- "Yes" --> X["Span is dropped and counted"]
Q -- "No" --> W["Span waits in the queue"]
W --> T{"Batch size reached or timer fired?"}
T -- "No" --> W
T -- "Yes" --> E["Exporter sends one batch"]
E --> R{"Did the export finish?"}
R -- "Yes" --> N["Take the next batch"]
R -- "Slow or stuck" --> H["Everything behind it waits"]
H --> Q
    

The four settings that matter

Four settings control the whole behavior. Their commonly used default values are shown below. Defaults can differ slightly between languages and versions, so always check the documentation of the SDK you actually run.

The sentence that explains the risk

The specification asks that a single exporter instance is not asked to run overlapping exports. In plain words: one courier, one bag at a time. This is a sensible rule. It keeps memory use predictable and avoids flooding the receiver. But look at the numbers in the table. If one trip hangs for the full 30 seconds, the queue has to absorb 30 seconds of new spans while it waits. And the queue only holds 2,048.

Simple division tells you how fast your pipeline can run and how quickly it fills when it slows down.

You do not need advanced mathematics to reason about head-of-line blocking. Two small formulas do almost all of the work.

Formula one: the speed limit of one courier

If the exporter sends one batch at a time, the most it can ever ship is the batch size divided by the time one trip takes. With the default batch of 512 spans, here is what different trip times allow.

Notice how brutally the speed limit falls. A trip that is merely ten times slower than normal makes the pipeline ten times slower. A service creating 3,000 spans per second runs happily at 100 milliseconds per trip, and drowns at one second per trip.

Formula two: how fast the waiting room fills

When shipping falls below the arrival rate, the difference piles up in the queue. In the worst case, when the courier is completely stuck, the time to fill a 2,048-span queue is simply the queue size divided by the arrival rate.

This is a version of a famous result called Little’s Law: the number of items waiting equals the arrival rate multiplied by how long each item waits. It is a perfect fit for queues of any kind. If items arrive at 2,000 per second and each waits for one second, then about 2,000 are always sitting in line.

Two nasty surprises

Let us replay an incident in slow motion, from the first hiccup to the long recovery.

Understanding the lifecycle of data during trouble is the best way to spot the problem early. The picture below shows the conversation between four actors: your application, the SDK queue, the exporter, and the Collector.

sequenceDiagram
participant App as Application
participant Q as SDK Queue
participant Ex as Exporter
participant Col as Collector
App->>Q: Spans arrive
Q->>Ex: Hand over batch 1
Ex->>Col: Send batch 1
Note over Col: The Collector becomes slow
App->>Q: More spans arrive
Note over Q: The queue keeps growing
App-xQ: New spans dropped when full
Col-->>Ex: Reply arrives after a long delay
Ex->>Q: Ready for batch 2
    

The six stages of a blocked pipeline

Why you often do not notice

Telemetry code is deliberately designed to never harm your application. When the queue is full, the SDK does not throw an error at your business logic. It quietly discards the span, usually with only an occasional internal warning. That is the right choice for protecting users, but it means the first sign of trouble is often a human saying “the graph looks weird.” This is why the monitoring chapter later in this guide insists that you also watch the health of the pipeline itself.

The Collector has its own queues and workers, and its own ways of blocking.

The OpenTelemetry Collector is a standalone program. Inside it, a pipeline is built from three kinds of parts. Receivers accept incoming data. Processors change or filter it. Exporters send it onward. Many pipelines can run side by side in one Collector, usually one per signal type.

flowchart LR
R1["OTLP Receiver"] --> M["Memory Limiter"]
M --> B["Batch Processor"]
B --> F["Fan-out to exporters"]
F --> Q1["Queue for Backend A"]
Q1 --> C1["Worker threads A"]
C1 --> D1["Backend A"]
F --> Q2["Queue for Backend B"]
Q2 --> C2["Worker threads B"]
C2 --> D2["Backend B"]
    

The exporter’s built-in helper

Most Collector exporters share a common helper that adds three features. A timeout limits how long a single send may take. A sending queue holds requests while workers are busy. And a retry mechanism re-sends failed requests after a pause. The sending queue has a number of consumers, which are worker threads that pull requests from the queue and send them. Typical defaults, at the time of writing, are a few seconds of timeout, around ten consumers, and a queue of roughly a thousand requests. Values change between versions, so confirm them in the documentation for your release.

Fan-out: when one pipeline feeds two places

If you configure a pipeline with two exporters, such as one sending to your main backend and one to cheap archive storage, the Collector hands the data to both. When the exporters have sending queues, the handoff is nearly instant because each just drops the request into its own queue. When a queue is switched off, the call goes straight to the network, and the Collector walks through the exporters one after another. Now a slow archive store makes the main backend wait, and the receiver waits too, because it cannot answer the application until all exporters have replied.

The memory limiter and the batch processor

The memory limiter is a safety valve. It watches how much memory the Collector uses, and when usage gets close to a limit it starts refusing new data instead of running out of memory and crashing. It is normally placed first in the pipeline. The batch processor groups many small incoming requests into bigger outgoing ones, commonly waiting a fraction of a second or until a few thousand items are collected. Because it hands data off quickly, it decouples receiving from sending. Some newer Collector versions are moving this batching into the exporter’s own queue, so watch release notes if you plan upgrades.

Backpressure: the pain travels backward

When the queue in front of a slow backend is full, the exporter refuses the next request. That refusal travels backward through the pipeline to the receiver, which tells the application’s SDK “I am busy, try later.” This signal is called backpressure. It is healthy and useful, because it stops the Collector from blowing up its memory. But it also means a slow backend at the far end of the chain can ripple all the way back to thousands of SDKs, each with its own courier now waiting.

Trying again sounds harmless. In a crowded system it can turn a small problem into a large one.

Networks fail in small ways all the time. A packet is lost, a server restarts, a connection resets. The standard answer is to retry: send the same request again after a short pause. OTLP defines which failures are worth retrying, and well-built exporters retry with exponential backoff, meaning the pause gets longer after each failure, such as one second, then two, then four, and so on.

Which failures are retried

The OTLP specification separates failures into two groups. Temporary ones may succeed on a second try. Permanent ones never will, so retrying them wastes effort and blocks workers.

When the server includes a hint about how long to wait, such as a standard retry delay in the response, a good client follows that hint instead of guessing. Spreading retries out in time with a little randomness, called jitter, matters just as much, as we will see next.

How retries create head-of-line blocking

In the Collector, a consumer that is retrying a request stays busy with it. It keeps waiting and re-sending, with growing pauses, until the request succeeds, is judged permanently failed, or hits the maximum retry time, which by default can be several minutes. Meanwhile it takes no new work. If a backend is flaky and every consumer gets stuck retrying, all counters at the airport are occupied by passengers who cannot be served. The queue behind them grows until it is full, and then backpressure starts flowing toward the applications.

The retry budget idea

Retries multiply load. If a service normally receives 1,000 requests per second and every failed request is retried three times, a failure that hits everyone can briefly demand up to 4,000 requests per second from a service that is already struggling. Mature teams limit total retry time, cap the number of retries, and prefer to drop old low-value data rather than keep workers busy with it. The goal is to remember that a retry is borrowed capacity, and the loan must be repaid by someone.

Even if your queues are perfect, the road underneath can create its own blocking.

OTLP can travel over two main transports. OTLP over gRPC uses HTTP/2, a modern protocol that can carry many conversations, called streams, at once over one connection. OTLP over HTTP sends ordinary web requests, using HTTP/1.1 or HTTP/2. The choice affects how blocking appears.

The sticky connection problem

gRPC connections are designed to live a long time. A typical network load balancer that works at the connection level picks one Collector when the connection opens and then sends everything on it to that same Collector for hours. If your fleet has fifty applications, and a handful happen to land on one Collector replica, that replica becomes busy while others sit idle. Worse, if that replica is the slow one, all those applications experience head-of-line blocking while the other replicas are healthy. The fixes are to use a balancer that understands requests rather than just connections, and to set a maximum connection age on the Collector so connections are periodically closed and re-opened, letting traffic spread out again.

Size matters

Bigger batches are more efficient, but they have two costs. They take longer to serialize, send and process, so each trip holds the courier longer. And receivers set a maximum message size. The Collector’s gRPC receiver commonly allows around four megabytes by default. A batch larger than the limit is refused, and because refusing is permanent for that exact batch, retrying it would never help. Keep batch size modest enough that a normal trip finishes in tens or hundreds of milliseconds. Turning on compression such as gzip reduces bytes on the wire at the price of some processor time, which is usually a good trade on busy networks.

Not all blocking comes from slow networks. Sometimes the data itself, or the people sending it, are the cause.

The poison pill

A poison pill is one piece of data that the system cannot handle, which causes repeated failure. Imagine one batch that contains a malformed value the backend rejects every time. If the system treats the rejection as temporary, it retries forever, holding a worker each time. The same happens with a batch that is simply too big for the receiver. Good pipelines classify errors correctly, drop permanent failures quickly, and count them so someone can investigate.

Heavy items that slow the line

Some metrics batches are heavy because of cardinality, a word that means the number of different combinations of labels. If a metric is labeled with a user identifier, and you have ten million users, you may create ten million separate series. A batch that contains them takes far longer to process than a batch of ordinary metrics, and everything queued behind it waits. Keeping labels to a small set of values is not only a storage-cost concern, it is a pipeline-health concern.

Valuable and worthless data in one line

Health checks and heartbeat endpoints can create thousands of nearly identical spans every minute. In a shared queue they stand in line with the rare error traces that you actually need during an incident. When the line is full, both are dropped equally. A filtering step that removes or samples the low-value spans early gives the valuable ones room to survive.

Noisy neighbors in shared Collectors

When many teams or customers share one Collector tier, a single bug in one service, such as a loop that emits a log line every microsecond, can flood the shared queue. The other teams, who did nothing wrong, now see delays and drops. This is the noisy neighbor problem. Per-team pipelines, separate queues, and rate limits turn a shared single lane into separate lanes, so one team’s flood stays in its own lane.

The security angle

A telemetry endpoint is a door into your infrastructure that accepts data from many sources. An attacker, or just a misconfigured client, can push huge volumes or oversized messages to make your queues fill, which is a form of denial of service that uses head-of-line blocking as its weapon. Basic hygiene helps: encrypt connections with TLS, require authentication for anything that is reachable beyond a trusted network, set message size and rate limits, and keep the memory limiter on. Do not expose a raw Collector receiver directly to the public internet.

You cannot fix what you cannot see. Here is what to watch, and what each signal tells you.

The pipeline carries your monitoring data, so it needs monitoring of its own. This is sometimes called monitoring the monitor. Three sources help: the SDK’s own diagnostic output, the Collector’s internal metrics, and the backend’s ingestion statistics.

The Collector exposes its own metrics, often on a local port in a format that Prometheus can read, and the names usually start with a common prefix and mention the exporter, receiver or processor involved. Exact names and suffixes vary by version, so inspect your own endpoint. The newer OpenTelemetry semantic conventions also define metrics for the SDK itself, such as queue size, queue capacity and processed counts, though some are still marked experimental and may not yet be present in every language.

Useful alarm ideas

• Queue use above about seventy percent for several minutes in a row.
• Any sustained rate of failed enqueues, since it means data is being lost right now.
• Receiver refusals above zero for more than a short burst.
• A sudden drop in the number of spans reaching the backend while your application traffic stays flat. This catches silent loss.

Every cure removes one of the two ingredients: strict ordering, or a single shared path.

Recall from Chapter 3 that blocking needs an ordered queue and a single shared path. Each pattern below breaks one of those two conditions. Taken together they form a layered defense, much like the compartments in a ship’s hull, where one flooded compartment cannot sink the whole vessel. This idea is known as a bulkhead.

flowchart LR
A["Application Services"] --> AG["Agent Collector on each host"]
AG --> GW["Gateway Collector Tier"]
GW --> P1["Critical Pipeline for errors and slow traces"]
GW --> P2["Bulk Pipeline for normal traces"]
GW --> P3["Metrics Pipeline"]
GW --> P4["Logs Pipeline"]
P1 --> B1["Primary Trace Backend"]
P2 --> B2["Low Cost Storage"]
P3 --> B3["Metrics Backend"]
P4 --> B4["Log Backend"]
    

Tail sampling needs special care

To decide whether a trace is interesting, tail sampling must see all spans of that trace in the same place. A special exporter can route spans by trace identifier, so every span of one trace lands on the same gateway replica. This creates a new risk, since one slow replica now holds up just the traces assigned to it. The exporter gives each destination its own queue, which keeps the damage limited to that slice. Tools such as Honeycomb’s Refinery are built around the same idea of sampling after seeing whole traces.

Every protection has a price. Here is an honest look at the three most popular ones.

There is no free fix. Each approach trades one risk for another, and a good architect states the trade openly.

Option A: Make the queue bigger

Option B: Add a disk-backed queue in the Collector

Option C: Add an agent tier and a gateway tier

The honest conclusion

A larger or persistent queue handles bursts. Parallelism, separation and sampling handle sustained overload. If your system is permanently shipping slower than it produces, no buffer will save it, and you must reduce the data or raise the capacity. Choose the tool that matches the kind of problem you actually have.

Four habits that look sensible on a whiteboard and fail in production.

Other frequent mistakes

• Never reading the Collector’s own metrics until a post-incident review.
• Raising the number of workers without checking whether the backend can accept more parallel writes, which only moves the jam.
• Running the Collector with no memory limiter and no resource limits in its container.
• Forgetting that development and production defaults may differ.
• Treating dropped spans as acceptable and invisible, rather than counting them.

A practical order of work that fits most teams, from measuring to protecting.

Resist the urge to change numbers at random. Follow a sequence, change one thing at a time, and watch the metrics from Chapter 12 after every change.

The knobs at a glance

Four illustrative situations, written as stories, showing how blocking appears and how teams respond.

The scenarios below are illustrative composites built from patterns that are widely discussed in the observability community. The numbers are examples chosen for clarity, not measurements from any named company.

Short answers to the questions people ask most often.