Sending data one piece at a time feels natural, and it is almost always the wrong choice once traffic grows.

What Is Telemetry?

Telemetry is the data a running system produces about itself so that humans can understand it from the outside. OpenTelemetry works with three kinds of it, usually called signals. Traces record the journey of one request through many services, and each step in that journey is a span. Metrics are numbers measured over time, such as requests per second or memory in use. Logs are timestamped text records of events.

All three signals share one problem. They are produced constantly, in tiny pieces, by code that is busy doing something else. A checkout service has no time to stop and negotiate with a monitoring server every time a step finishes.

The Hidden Cost of Sending One Item at a Time

Imagine a single span. After it is encoded into bytes it may be only a few hundred bytes long. Now picture what it takes to send that span alone. The program must turn the span into a wire format, wrap it in a network message with headers, hand it to the operating system, wait for the other side to acknowledge it, and then clean up. Each of those steps has a fixed price that does not shrink just because the payload is small.

The server on the other end pays too. It must accept the connection or reuse one, check who is calling, route the request, parse the bytes, and send back a reply. When millions of tiny requests arrive, the server spends most of its energy on ceremony instead of storing data.

What Batching Buys You

• Fewer network calls. One call carries hundreds of items, so the fixed price is shared by all of them.
• Better compression. Compression finds repeated patterns, and a batch of spans from the same service repeats names, attribute keys and resource details constantly. Bigger payloads shrink more.
• Less CPU. Encoding, system calls and thread wake-ups happen once per batch, not once per item.
• A calmer backend. A steady stream of medium-sized requests is easier to ingest than a storm of tiny ones.
• Work off the hot path. The application only drops the item into an in-memory queue, which is very fast. A separate background worker does the slow network part.

What Batching Costs You

Nothing is free. While items wait for their batch to fill, they are not visible to anyone, so batching adds delay. While they wait they also sit in memory, so batching spends RAM. And if the process crashes before the batch is sent, those items are gone, so batching adds a small risk of loss. Tuning at scale is the art of balancing these three costs against the efficiency you gain.

Why the Defaults Stop Working at Scale

The built-in defaults were chosen to be safe for a modest service. They work well for a few hundred spans per second. At tens of thousands of items per second, the same defaults can overflow their queues, send tiny batches during quiet hours, send oversized batches that the receiver rejects, or let memory grow until the process is killed. Each of these failures has a clear cause, and the rest of this guide explains them one by one.

Batching is not one setting in one place. It happens at several points along the road from your code to your dashboard.

Meet the Cast

Instrumentation is the code that creates telemetry, either written by hand or added automatically by a library. The SDK (software development kit) is the OpenTelemetry library inside your application that receives that telemetry, decides what to keep, and prepares it for shipping. Inside the SDK, a processor handles each item as it finishes, and an exporter is the part that actually sends data out of the process.

The exporter usually speaks OTLP, the OpenTelemetry Protocol, which is the standard format and wire protocol for telemetry. OTLP can travel over gRPC (a fast, binary, HTTP/2-based protocol, usually on port 4317) or over plain HTTP carrying Protobuf (usually on port 4318).

The receiver of that data is often the OpenTelemetry Collector, a separate program whose whole job is to receive, process and forward telemetry. A Collector is built from three kinds of parts. Receivers accept data. Processors change or filter it. Exporters send it onward. The chain of those parts for one signal is called a pipeline.

At scale, teams run two layers of Collectors. An agent Collector runs close to the application, for example one per server. A gateway Collector is a shared pool of Collectors that sits in front of the backend. The backend is the database and user interface where people finally look at the data.

flowchart LR
  A["Application code
creates spans, logs and metrics"] --> B["SDK batch processor
queue plus timer"]
  B -->|"OTLP over gRPC or HTTP"| C["Agent Collector
one per node"]
  C -->|"OTLP"| D["Gateway Collector pool
batch processor plus exporter queue"]
  D -->|"Backend protocol"| E["Backend
storage and query"]
    

The Five Places Batching Happens

Follow one span along the diagram above and you will meet batching five times. First, the SDK processor holds it in a queue until a batch is ready. Second, the exporter sends the batch as one request. Third, the Collector’s batch processor regroups what arrives from many senders into new batches. Fourth, the Collector’s exporter keeps its own queue of batches waiting to be sent, and may retry them. Fifth, the backend often buffers incoming data before writing it to storage.

A Brief History That Explains the Design

Large companies hit these problems long before OpenTelemetry existed. Google’s Dapper paper, published in 2010, described tracing a huge fleet by sampling only a small fraction of requests and collecting the data out of band, so that the monitored applications were never slowed down. Uber built Jaeger for the same reason and later gave it to the Cloud Native Computing Foundation. In 2019 two competing projects, OpenTracing and OpenCensus, merged to form OpenTelemetry. The lessons from those years live on in its design: keep the hot path tiny, send in groups, and protect the application first.

The first and most important batching layer lives inside your own application process.

Simple Versus Batch

OpenTelemetry offers two processors for spans. The simple processor sends each span the moment it ends, and it waits for the send to finish. That is fine on a laptop while you are debugging, but in production it would put a network call on the path of every request. The batch processor is the production choice. It accepts a finished span almost instantly and does the sending later on a background worker.

The Life of One Span

Here is what happens when a span ends. If a sampler already decided to discard it, it never reaches the processor, so sampling is your first and cheapest way to reduce load. Otherwise the processor tries to place the span into a bounded queue, which is a waiting line with a fixed maximum length. If the line is full, the span is dropped on the spot and a counter or warning notes the loss. If there is room, the span joins the line and the application moves on, usually within microseconds.

A background worker watches the queue. It wakes up in two situations: when the queue holds at least one full batch, or when the schedule timer fires. It then removes up to one batch worth of spans, hands them to the exporter, and waits for the exporter to finish or for a time limit to pass. Then it checks the queue again.

flowchart TD
  S["Span ends"] --> Q{"Is the queue full?"}
  Q -->|"Yes"| X["Drop the span and record the loss"]
  Q -->|"No"| W["Add span to the queue"]
  W --> T{"Full batch waiting
or timer fired?"}
  T -->|"No"| K["Keep waiting"]
  T -->|"Yes"| B["Remove up to one batch"]
  B --> E["Exporter sends the batch
waits for reply or timeout"]
  E --> T
    

The Hidden Speed Limit

Most SDKs send one batch at a time. The specification says the processor must not call the exporter concurrently, so a new batch cannot leave until the previous one has been answered. That creates a speed limit that surprises many teams. The maximum steady throughput of one processor is roughly the batch size divided by the time one export takes.

Take the default batch size of 512 spans and a network round trip to the Collector of 200 milliseconds. One export per 0.2 seconds gives about five exports per second, or roughly 2,560 spans per second. A service that creates 5,000 spans per second will fill its queue faster than it can drain it, no matter how large the timer settings are. The queue fills, spans start dropping, and nothing looks wrong on the application side.

Flushing on Shutdown

When an application stops, the spans still sitting in the queue would vanish unless someone sends them. A well-behaved SDK offers a flush operation that forces an immediate export and a shutdown operation that flushes and then stops the worker. Long-running servers call shutdown when they receive a stop signal. Short-lived programs, such as batch jobs and serverless functions, must call flush before they finish, or their final spans are lost.

Almost every batching decision in the SDK comes down to four numbers.

Names and Defaults

These settings can be changed in code or, more conveniently, through environment variables that every standard SDK understands. Using environment variables means operators can retune a service without rebuilding it.

How the Knobs Interact

The knobs are not independent, and most tuning mistakes come from changing one while forgetting the others.

Batch size must not exceed queue size. A batch is taken out of the queue, so a queue smaller than a batch could never produce a full one. In practice the queue should hold many batches, so that new items can keep arriving while one batch is out being sent. A common rule of thumb is a queue several times larger than the batch.

Delay matters mostly when traffic is low. At high traffic, the queue reaches a full batch long before the timer fires, so batches leave because they are full. At low traffic, the timer is what releases batches, and they leave partly empty. This is why a service that sends perfect 512-span batches at noon may send batches of 12 at 3 a.m.

Timeouts form a chain. The OTLP exporter has its own request timeout, 10 seconds by default, separate from the processor’s 30-second export timeout. If the exporter timeout is longer than the processor timeout, the processor gives up while the request is still running. Keep the inner timeout shorter than the outer one so that failures are reported by the layer that understands them.

Spans arrive one at a time and wait in a queue. The other two signals batch in their own ways, and each has its own scaling trap.

Metrics: Summaries Instead of Receipts

A metric is not stored as a long list of individual events. When your code records that a request took 120 milliseconds, the SDK does not keep that single event. It folds the value into a running summary in memory, for example a count, a sum and a set of histogram buckets. Think of a shop that does not keep every receipt on the counter but updates a tally sheet instead.

The part that exports these summaries is the periodic exporting metric reader. Every time its interval ends, it collects the current summaries of all metrics, packs them into one request, and sends them. The interval defaults to 60 seconds and can be changed with OTEL_METRIC_EXPORT_INTERVAL. The time allowed for one export defaults to 30 seconds and is set with OTEL_METRIC_EXPORT_TIMEOUT.

The Real Size Driver Is Cardinality

Cardinality means the number of unique combinations of attribute values for a metric. A request counter with the attribute “HTTP method” has a handful of combinations. Add “user identifier” and it has one combination for every user, which could be millions. Each combination becomes its own time series and its own data point in every export.

Here is the arithmetic that surprises people. Suppose a fleet has 5,000 pods and each pod reports 500 series. That is 2.5 million data points per interval. With a 60-second interval, the pipeline carries about 41,700 points per second. Cut the interval to 10 seconds for fresher dashboards and the same fleet now sends 250,000 points per second, six times more, without a single extra user.

The Synchronized Burst Problem

Metric readers export on a fixed rhythm that starts when the process starts. When a whole fleet restarts at the same moment, for example during a fast deployment, thousands of readers share the same rhythm. Every minute the Collector sees a sudden wall of data followed by silence. Rolling out changes gradually, or adding a small random delay at startup, spreads the load into a smooth stream.

Logs: Spans With a Temper

Log records use a batch log record processor that works like the span processor: a bounded queue, a batch size, a delay and a timeout. Its default delay is shorter, one second, because people expect log lines to appear quickly. The settings use the OTEL_BLRP_ prefix.

The danger with logs is that they are bursty in exactly the wrong way. When a dependency fails, every request starts writing error logs, so volume can jump tenfold at the same instant the system is struggling and the backend may be slow. A queue sized for a normal day overflows during the incident, and the lines you most wanted vanish first. Sizing the log queue for the bad day, and filtering noisy low-severity lines at the source, protects the records that matter.

The SDK groups data from one process. The Collector regroups data from thousands of processes into batches that suit the backend.

Why Batch a Second Time?

The senders in front of a Collector all make their own choices. One sends 12 spans because it is quiet, another sends 512 because it is busy, and a third sends a single span because someone configured a simple processor. If the Collector forwarded each request as it arrived, the backend would see the same messy mix.

The Collector’s batch processor fixes this by pouring everything into one container and handing the exporter evenly sized batches. It does not care who sent the data. It cares only about how many items it holds and how long the oldest item has been waiting.

The Settings That Matter

Size Is a Trigger, Not a Cap

This detail causes real incidents. The send_batch_size setting says “send when you reach at least this many”. If a sender delivers one request holding 20,000 spans, the processor adds all 20,000 and sends a batch that big. The setting did not stop it. Only send_batch_max_size splits batches into guaranteed maximum sizes. When your backend or next hop has a strict payload limit, set the maximum explicitly.

The next hop also has a limit. The Collector’s OTLP gRPC receiver, for example, accepts messages up to 4 MiB by default. A batch of thousands of spans with large attributes can cross that line, and the sender then receives a rejection. Estimate your average item size, multiply by the batch, and leave a safe margin.

Two Very Different Behaviors From One Setting

With the default 8,192 and 200 milliseconds, a busy and a quiet Collector behave like two different machines.

• Busy case. At 100,000 spans per second, 8,192 spans arrive in about 82 milliseconds. The size trigger fires long before the 200-millisecond timer, so batches are full and the processor sends about 12 per second.
• Quiet case. At 500 spans per second, filling 8,192 would take more than 16 seconds. The timer fires every 200 milliseconds and releases batches of only about 100 spans. Efficiency is far lower than the configuration suggests.

The lesson is that in quiet periods the timeout decides batch size, and in busy periods the size setting does. Tune both, and watch which trigger is actually firing, a topic covered later under monitoring.

Where It Sits in the Pipeline

Order matters. A healthy trace pipeline usually places components in this order: first the memory limiter, then any processors that add attributes, filter or sample data, and the batch processor last before the exporter. Batching late means you never waste effort grouping items that an earlier step would have thrown away. Putting the memory limiter first means it can refuse new data before the Collector’s memory fills.

Batching Per Tenant

A software-as-a-service platform may send data for many customers through one Collector. If the batch processor mixed everyone together, one batch could carry data for ten customers, and the exporter could not attach the right credentials or route to the right destination. Setting metadata_keys to a header such as a tenant identifier makes the processor keep a separate batch per value. The receiver must be told to keep request metadata, and the number of groups must stay bounded, because each group holds its own batch in memory.

After a batch is ready, it still has to reach the backend, and the backend is never perfectly reliable.

The Sending Queue

Most Collector exporters come with a built-in sending queue. Think of it as a loading dock. Finished batches are placed on the dock, and a group of workers called consumers pick them up and carry them to the backend. Three settings define the dock: whether the queue is enabled, how many consumers work in parallel (default 10), and how many items the dock can hold (the size).

Unlike the SDK, which sends one batch at a time, the Collector’s exporter sends in parallel. Suppose each request to the backend takes 250 milliseconds and each batch holds 8,192 spans. One consumer sends four batches per second, about 32,800 spans. Ten consumers raise the ceiling to roughly 327,000 spans per second. If a gateway receives more than that, the queue starts to fill.

flowchart LR
  A["Batch processor
produces batches"] --> B{"Is the sending
queue full?"}
  B -->|"Yes"| C["Reject the batch
error goes back upstream"]
  B -->|"No"| D["Sending queue"]
  D --> E["Consumers send in parallel"]
  E --> F{"Did the backend
accept it?"}
  F -->|"Yes"| G["Done"]
  F -->|"Temporary error"| H["Wait with growing delay
then retry"]
  H --> E
  F -->|"Permanent error"| I["Drop the batch and count it"]
    

Do the Memory Math Before You Raise the Size

A larger queue absorbs longer backend outages, which feels like a pure benefit. The cost is memory. In many versions the queue size counts batches, not individual items. With a size of 1,000 batches of 8,192 spans each, a completely full queue holds more than eight million spans. Even at one kilobyte per span in memory, that is about eight gigabytes. A Collector sized for two gigabytes will be killed long before the queue reaches that point.

So the real design question is not “how big can the queue be” but “how many seconds of backend outage do I want to survive, and how much memory does that need”. Multiply your steady ingest rate by the outage duration, convert to memory using a measured item size, and add headroom.

Retries With Growing Gaps

When the backend answers with a temporary error, such as “unavailable” or “too many requests”, the exporter does not give up. It waits, tries again, and waits longer each time. This pattern is called exponential backoff. Typical defaults start at five seconds, grow toward a maximum interval of 30 seconds, and stop retrying after a total of about five minutes. A random jitter is added so that hundreds of Collectors do not all retry at the same instant.

Permanent errors, for example data the backend considers invalid, are not retried because retrying cannot change the answer. Those batches are dropped and counted.

Persistent Queues: Surviving a Restart

An in-memory queue disappears when the Collector restarts, whether the restart is planned or the result of a crash. A persistent queue stores waiting batches on disk through a storage extension, so they survive. The cost is disk speed and some extra delay for each batch. Persistence is worth it when the data cannot be regenerated and restarts are common, for example gateways running on frequently replaced machines. It is not worth it for data that is cheap and plentiful, such as high-volume debug logs.

When a pipeline cannot keep up, something must give. The goal is to choose what gives, instead of letting the process choose for you by crashing.

What Backpressure Means

Backpressure is the signal a slow consumer sends to its producers to say “slow down”. Water in a pipe shows the idea: if the outlet is blocked, pressure builds and travels backward through the pipe. In a telemetry pipeline, the backend slows down, exporter queues fill, the batch processor cannot hand over batches, the receiver cannot accept new data, and finally the sending SDK finds its requests rejected or delayed.

Backpressure is healthy when it is controlled. It becomes dangerous when nothing stops the buffers in the middle from growing without limit.

The Memory Limiter Processor

The Collector provides a memory limiter processor as a safety valve. It checks the process’s memory on a regular schedule, set by check_interval, commonly one second. You give it a limit, either as a fixed number of mebibytes or as a percentage of available memory, plus a spike limit that estimates how much memory can be added between two checks.

The limiter has two thresholds. The soft limit is the main limit minus the spike limit. When usage rises above it, the processor starts refusing new data, returning an error to the receiver so the sender knows to retry later. If usage keeps climbing past the hard limit, the processor also forces the runtime to free memory as aggressively as it can.

What the Sender Sees

When the limiter refuses data, the sending SDK’s exporter receives an error. Standard OTLP exporters treat certain errors as retryable and try again with backoff. While one export is being retried, the processor’s single worker is occupied. The SDK queue fills behind it and eventually drops the newest items.

This sounds bad, but it is the intended design. Losing a few spans at the edge, where they are cheap and the application is unharmed, is far better than losing a whole Collector and every sender’s in-flight data. Pressure is pushed out to the edges, where it can do the least damage.

Scaling the Gateway Correctly

If the limiter is refusing data regularly, the gateway needs more capacity, and adding Collectors is the natural answer. Choose the signal for autoscaling carefully. Memory is a poor trigger because the limiter holds it near the ceiling by design, so a healthy but busy Collector already looks full. CPU usage and queue utilization react earlier and describe the real bottleneck.

Good tuning is arithmetic with a few measured inputs. Once you know four numbers, most settings follow.

The Four Inputs

• Arrival rate. How many items per second the process or Collector receives, at peak and on average.
• Export latency. How long one send takes from start to acknowledgment, including network time and backend processing.
• Stall tolerance. How many seconds of backend trouble you want to survive without dropping data.
• Item size. The average memory an item uses while waiting, and the average bytes it takes on the wire. Measure it instead of guessing.

Little’s Law in Plain Words

A simple rule from queueing theory, known as Little’s Law, says that the average number of items inside a system equals the arrival rate multiplied by the average time each item spends there. If 100 customers arrive per hour and each stays 30 minutes, about 50 customers are in the shop at any moment.

Applied to telemetry, the rule gives a direct way to size things. Items waiting in the queue equal the arrival rate times the seconds they wait. When the backend stalls, the wait grows by the stall length, so the queue must hold the arrival rate times the stall time, plus normal waiting.

Worked Example One: A Busy Service Instance

One instance of an order service creates 4,000 spans per second at peak. The agent Collector on its machine answers in about 100 milliseconds, and the team wants to survive a five-second Collector hiccup.

Worked Example Two: A Gateway Collector

A fleet produces 200,000 spans per second in total, spread over 10 gateway Collectors, so each handles 20,000 per second. The team uses the defaults: a size trigger of 8,192 and a timeout of 200 milliseconds.

Filling 8,192 spans at 20,000 per second takes about 410 milliseconds, which is longer than the 200-millisecond timer. The timer wins every time, and each batch holds only about 4,000 spans, half of what the setting suggests. The gateway sends five batches per second where it could have sent two or three.

There are two fixes. The team can raise the timeout to 500 milliseconds so the size trigger wins and batches are full, at the cost of up to 300 extra milliseconds of delay. Or it can run fewer, larger gateways so each receives more traffic. Neither is automatically right; the choice depends on how much freshness the business needs.

Every setting moves you along three axes. You cannot improve all three at once, so you choose which one matters most.

The Latency Budget

Data takes time to appear in a dashboard, and the delay is the sum of several waits. Understanding where the time goes tells you which knob is worth turning.

Adding the worst cases, a quiet service can see a span take 5 plus 0.2 plus 0.2 plus perhaps 10 seconds, so around 15 seconds. The biggest and cheapest piece to shorten is usually the SDK delay for low-traffic services. For high-traffic services the delay seldom fires, so shortening it changes little, and the backend wait dominates.

Three Starting Profiles

The table below shows illustrative starting points, not laws. Treat them as hypotheses to test against your own traffic.

Different Signals, Different Profiles

You do not have to use one profile for everything. Metrics that drive alerts can use a short interval while detailed traces use larger batches. Debug logs can be delayed and compressed heavily, while error logs go through a separate low-latency pipeline. The Collector allows several pipelines in one process, each with its own batch settings, so freshness is spent only where it pays off.

At some size one Collector is not enough, and the layout of Collectors becomes part of your batching design.

Why Use Two Tiers?

An agent Collector runs next to the applications, often one per server or as a helper inside each pod. Applications send to it over a very short network hop, so SDK exports finish quickly and the SDK queue stays nearly empty. The agent adds host details, performs the first batching and retry, and shields applications from backend trouble.

A gateway pool sits between all agents and the backend. It is the place for expensive shared work: tail sampling, heavy filtering, routing by customer, and holding credentials for the backend so that applications never see them.

flowchart TD
  A1["Service pods on node 1"] --> G1["Agent Collector node 1"]
  A2["Service pods on node 2"] --> G2["Agent Collector node 2"]
  A3["Service pods on node N"] --> G3["Agent Collector node N"]
  G1 --> LB["Layer 7 load balancer
or client-side balancing"]
  G2 --> LB
  G3 --> LB
  LB --> GW1["Gateway Collector 1"]
  LB --> GW2["Gateway Collector 2"]
  LB --> GW3["Gateway Collector M"]
  GW1 --> BE["Backend"]
  GW2 --> BE
  GW3 --> BE
    

How Agent Settings Shape Gateway Load

Agents multiply. A cluster with 2,000 nodes has 2,000 agents, and each agent sends requests to the gateway pool. If traffic per node is modest, the agent’s 200-millisecond timer releases a batch five times per second, which means 10,000 requests per second arrive at the gateways even though each request is small. Raising the agent timeout to one second cuts that to 2,000 requests per second, and each request carries more data. The gateways do less per-request work, and the price is up to 800 more milliseconds of delay.

The Long-Lived Connection Trap

gRPC keeps one connection open and reuses it for many requests. That is efficient, but it interacts badly with simple load balancers that spread connections instead of requests. Each agent connects once, gets assigned to one gateway, and stays there for hours. When you add gateways during a traffic spike, the new ones receive almost nothing because every existing connection is already attached to an old one. Some gateways run hot while others sit idle.

There are three common remedies. Use a load balancer that understands HTTP/2 and spreads individual requests. Let clients resolve several gateway addresses and rotate among them. Or set a maximum connection age on the gateway’s receiver, so it politely closes connections after a while and clients reconnect to whichever gateway is available. Setting a maximum age also helps rolling deployments finish smoothly.

Sidecars and Direct Export

Two other layouts exist. A sidecar runs a tiny Collector inside every pod, giving each application its own private batching layer at the price of many small Collectors. Direct export skips Collectors and sends from the SDK straight to the backend. That is simplest for small systems, but every retry, credential and backend outage then lands inside the application process, so most large teams avoid it.

Some Collector components keep data for a while before releasing it. Each one adds to the batching story.

Head Sampling and Tail Sampling

Sampling means keeping only a portion of the traces. Head sampling decides when a trace begins, for example “keep one in twenty”. It is cheap and happens before data reaches any queue, which makes it the first line of defence against volume. Its weakness is that it decides blindly, and an interesting slow request may be discarded by chance.

Tail sampling decides after the trace is complete, so it can say “keep every trace that contains an error or took longer than two seconds”. To do this the Collector’s tail sampling processor must hold every span of every in-progress trace until a decision wait, 30 seconds by default, has passed.

The Memory Cost of Waiting

Holding data for a decision is just a very long wait inside a queue, so Little’s Law applies again. A gateway receiving 20,000 spans per second and waiting 30 seconds holds about 600,000 spans at any moment. At a kilobyte each, that is roughly 600 megabytes before counting batches, exporter queues and runtime overhead. The processor also limits the number of traces it tracks, so traffic bursts can evict traces before a decision is made.

All Spans of a Trace Must Meet

Tail sampling only works if every span of a trace reaches the same Collector. Spans from one trace come from many services and machines, so a plain round-robin gateway pool would scatter them. The standard answer is a first tier whose exporter uses a load-balancing exporter. It looks at the trace identifier of each span and sends all spans with the same identifier to the same second-tier Collector. The second tier runs the tail sampling.

This has a batching side effect. The load-balancing exporter splits each incoming batch by destination. A batch of 8,192 spans spread over 10 second-tier Collectors becomes ten requests of about 800 spans. The more destinations you add, the smaller the pieces become, so first-tier batch sizes should be larger than you would otherwise choose, or you accept smaller requests between the tiers, where the network is usually cheap.

Components That Create Their Own Rhythm

Other components also hold data. A span-to-metrics connector turns traces into counts and durations and flushes them on its own interval, which means it adds its own freshness delay and its own memory use. Components that aggregate, deduplicate or group telemetry follow the same pattern. Each time you add one, ask three questions: how long does it hold data, how much memory does that hold cost, and where should the batch processor sit relative to it.

A batch is not only a count of items. It is also a number of bytes, and bytes are what networks, receivers and bills care about.

Why Bigger Batches Compress Better

Compression works by finding repeated patterns and replacing them with short references. Telemetry is very repetitive. Span names, attribute keys, service names and host details appear again and again. OTLP also groups spans under shared resource and scope descriptions, so a batch of 1,000 spans from one service stores the long list of resource attributes once, not 1,000 times. A batch of one span cannot share anything.

Choosing a Protocol and a Compressor

Compression choices carry a trade-off between CPU and bandwidth. Many SDKs send uncompressed data unless told otherwise, and an environment variable switches on a compressor. The Collector supports several, including gzip and zstd. Zstd usually reaches a similar or better ratio than gzip for less CPU, but both sides must support it. In cloud environments, traffic that crosses availability zones or leaves a network is often billed, so compression between agents and gateways can pay for itself in money as well as bandwidth.

Compression also changes where the bottleneck sits. At very high rates, compressing on the sender uses CPU on the application host, which may be the resource you can least afford to spend. Teams often keep the application-to-agent hop uncompressed, because it is local and cheap, and compress on the longer hops between agents, gateways and the backend.

Staying Under Payload Limits

Receivers and backends refuse messages above a certain size. Suppose the average span is 600 bytes. A batch of 8,192 spans is then about 4.9 million bytes, which exceeds a 4 MiB limit of roughly 4.2 million bytes. Larger attributes, such as long stack traces or database statements, make the problem worse. The result is a rejected request that cannot be fixed by retrying, because the batch is simply too big.

Defensive setups use three layers. They estimate average and worst-case item size. They set a maximum batch size on the Collector so oversized batches are split automatically. And they limit attribute length at the SDK, so a single misbehaving log line cannot create a monster span. To be safe, treat message size limits as applying to uncompressed data unless you have verified otherwise.

You cannot tune what you cannot see. The batching layers expose signals that tell you what to change next.

The Pipeline Needs Its Own Dashboard

The Collector reports about itself, using the same kind of metrics it handles for others. Traditionally these are served in the Prometheus text format on port 8888, and newer versions let you configure the destination in the Collector’s own settings. Metric names carry a prefix and can differ slightly between versions, so check the names exposed by your own build before you write alerts.

The Signals Worth Watching

The most useful single number is queue utilization: the current queue length divided by its capacity. A queue that stays below a modest fraction of capacity has plenty of room. A queue that sits above roughly 70 percent for several minutes is already in trouble, because a small additional stall will overflow it.

Equally useful is the ratio of timeout-triggered sends to size-triggered sends. It reveals which rule is controlling your batch sizes, and so tells you whether tuning the timeout or the size will help.

The SDK Side Is Quieter

Application SDKs historically say little about their own queues. Many only write a warning line when they drop items, and some do not even do that by default. Newer SDK releases are adding built-in self-metrics for queue size and processed counts, but support varies by language and version. Until your SDKs report them, two cheap tricks help. Watch the Collector’s received-item rate per service, which falls if the SDK is dropping. And compare the number of requests an application handled with the number of root spans that reached the backend.

Measure Freshness With a Canary

Internal counters tell you whether items move. They do not tell you how long the whole journey takes. A canary is a small synthetic producer that emits a tagged span or log every few seconds. A checker queries the backend for it and measures the gap between creation and visibility. That gap is your real end-to-end freshness, including every wait described in this guide, and it is the number users actually feel.

Tuning is tested during outages. Walk through what each layer does when the backend disappears.

A Backend Outage, Layer by Layer

sequenceDiagram
  participant App as Application SDK
  participant Agent as Agent Collector
  participant GW as Gateway Collector
  participant BE as Backend
  App->>Agent: Batch of spans
  Agent->>GW: Batch of spans
  GW->>BE: Export request
  BE-->>GW: Error, service unavailable
  GW->>GW: Queue batch and retry with backoff
  Note over GW: Queue fills during a long outage
  GW-->>Agent: Refuse new data
  Agent->>Agent: Queue batch and retry
  Agent-->>App: Slow answers or refusals
  App->>App: SDK queue fills and new spans are dropped
    

In the first seconds the gateway’s retries and queue hide the problem. Nothing is lost and nobody notices. As the outage continues, the gateway queue fills and the memory limiter starts refusing data. The agents now see errors, store batches in their own queues and retry. If the outage lasts longer still, agent queues fill and the SDKs start receiving slow answers or refusals. Finally the SDK queues fill, and the newest spans are dropped.

Each layer is a shock absorber, and the total outage you can survive is roughly the sum of the buffer times of all layers. This is why each buffer was sized from stall math earlier: you decide in advance how long you can ride out a failure and where the first drops should occur.

Common Failure Cases

The Recovery Surge

When the backend returns, every queue tries to drain at once. Gateways have minutes of backlog and ten consumers each, so the backend may receive many times its normal traffic and fall over again. This is a retry storm. Backoff with random jitter spreads the first wave. Limiting consumers, using a backend that can absorb bursts, or draining queues from the oldest data first at a controlled pace all help. Some teams deliberately accept dropping old data after an outage, because fresh data is more valuable than stale data and recovery stays calm.

Delivery Is Best Effort

A telemetry pipeline is designed to degrade gracefully, not to promise perfect delivery. Retries can cause duplicates, because a request may succeed on the backend even though the reply was lost. OTLP also allows partial success, where the backend accepts some items and reports the rest as rejected. Backends should tolerate duplicates, and teams should decide in advance which signals are allowed to lose data and which need stronger guarantees such as persistent queues.

Teams that run telemetry at scale tend to arrive at the same habits. Here they are, along with the mistakes that teach them.

Habits That Work

Mistakes That Teach

Telemetry often contains more sensitive information than people expect, and batching changes where that information sits and for how long.

What Is Inside Your Batches

A span or log line is built from attributes that developers add. Over time those attributes can pick up customer email addresses, account numbers, request bodies, database statements and internal host names. Because batching holds data in memory and sometimes on disk before it is sent, every waiting room is also a place where sensitive data can be read by anyone who gets access to the machine, a memory dump or a leftover file.

Protect Data Before It Waits

The safest data is data that was never stored. Filtering and redaction processors that remove or mask sensitive attributes should run before the batch processor and before any persistent queue. In the Collector this means placing them early in the processor chain, right after the memory limiter. Redacting later means the unprotected values have already been written into queues and temporary files.

Many teams also limit attribute lengths and block known-sensitive keys at the SDK. That keeps risky values out of the pipeline entirely and reduces payload size at the same time, which is a rare case where the safer choice is also the cheaper one.

Encrypt the Hops and Guard the Disk

Every network hop in Figure 1 should use encryption in transit, usually TLS, and should prove who is calling, either with client certificates or with tokens passed in request headers. Batching makes this cheaper than it sounds. A TLS handshake is costly, but a long-lived connection carrying large batches pays that cost once and shares it across thousands of items.

A persistent queue writes batches to disk. Those files contain the data exactly as it would be sent, so they need the same care as a database. Use encrypted volumes, restrict file permissions to the Collector’s own account, and clear or expire old queue files when a machine is retired.

Credentials Belong at the Gateway

The backend usually requires an API key or token. If every application holds it, a leak from any one service exposes the key, and rotating it means redeploying everything. A gateway Collector can hold the credential and attach it when it exports, so applications only need permission to talk to the nearby agent. This also lets you apply different keys per tenant when batches are grouped by customer.

Defending the Pipeline Itself

A pipeline that accepts data from many senders can be overwhelmed by accident or on purpose. A buggy release might create millions of unique metric series, or a hostile client could send enormous requests. The defences are the ones already covered in this guide, now seen as security tools: maximum request size on receivers, a bounded batch size, a memory limiter that refuses data when full, per-tenant grouping limits, and limits on attribute length and series count. Each setting protects not only performance but also the availability of the whole monitoring system.

At scale, tuning is also budgeting. Every setting moves money between CPU, network, memory and backend charges.

Where the Money Goes

A telemetry system has four main cost areas. Compute pays for the CPU that encodes, compresses, batches and forwards data, both inside applications and in Collectors. Network pays for bytes moved, and in many clouds, traffic between zones or out of a network is billed per gigabyte. Memory and disk pay for queues and buffers. Backend ingestion and storage usually costs the most, and it is often priced by volume.

Batching mainly reduces compute and network, because it cuts per-request overhead and enables compression. It rarely changes backend volume, because the backend still stores every item it receives. That is why sampling and filtering, which reduce item counts, usually save more than any batch setting.

A Back-of-the-Envelope Example

Take a platform producing 100,000 spans per second with an average size of 600 bytes. That is 60 megabytes per second, which adds up to about 5.2 terabytes per day before compression. If compression shrinks the data to a fifth of its size, which is plausible for repetitive spans but must be measured on your own data, the network carries about one terabyte per day.

The takeaway is the order of magnitude. Compression gives a large saving on the wire, sampling gives a larger saving everywhere, and batch size tuning refines what remains. If your first optimization is fiddling with a timeout, you may be working on the smallest lever.

The Cost of Being Too Safe

Reliability settings have prices too. A queue sized for a one-hour outage needs memory or disk that sits idle almost all the time. A persistent queue needs fast storage. Over-provisioned gateways run at low utilization to leave room for bursts. These choices are justified when the data is valuable, but they should be made deliberately and written down, because they are the first items people question when budgets tighten.

The Cost of Being Too Lean

The opposite mistake is also expensive. Dropping telemetry during an incident means engineers debug blind, and every extra hour of an outage costs far more than a few gigabytes of buffer. A fair way to decide is to ask what a lost hour of data costs the business, then compare that with the monthly cost of the buffers that would have saved it.

The specification defines the same settings for every language, but the runtime underneath changes how those settings behave.

Where the Background Worker Lives

A batch processor needs a background worker to wake up on timers and send data. How that worker is built depends on the language. In Java it is usually a dedicated daemon thread. In Go it is a goroutine, a very light unit of concurrent work. In Python it is a normal thread. In Node.js there are no extra threads for this job, so work is scheduled on the single event loop that also serves your requests. In each case the same four knobs apply, but the cost of the worker differs.

Node.js: Watch the Event Loop

In Node.js, turning thousands of spans into bytes happens on the same thread that handles incoming requests. A very large batch can block the event loop for a noticeable time while it is encoded, which shows up as latency spikes in the application itself. Moderate batch sizes, sent more often, keep each pause short. If you see periodic latency spikes that line up with the export schedule, large batches are a prime suspect.

Python and Pre-Fork Servers

Many Python web servers start a parent process and then copy it into several worker processes, a technique called forking. Threads do not survive a fork. If the batch processor was started in the parent before forking, the copied workers inherit a queue with no background worker behind it. Spans pile up, nothing is sent, and the queue eventually fills and drops data silently. The usual remedy is to initialize telemetry inside each worker after the fork, using the server’s own hook for that moment.

Java and Go: Shutdown Hooks and Contexts

Long-running Java and Go services need the flush to run when the process receives a stop signal. In Java a shutdown hook can close the tracer provider, and in Go the program should call the provider’s shutdown function with a deadline before exiting. Without it, the last few seconds of data, often the most interesting moments before a crash or deployment, are discarded.

Serverless and Short-Lived Runtimes

Serverless platforms freeze or destroy a function’s environment once the handler returns. Timers do not run while frozen, so a five-second delay may never expire. Functions need an explicit flush before returning, and they benefit from a very close Collector or platform extension so that the flush completes in milliseconds.

Do Not Assume Identical Defaults

The specification describes standard defaults, and most SDKs follow them, but implementations differ in detail: how drops are reported, whether exports can run concurrently, and which environment variables are honored. Before relying on a number from this guide in a particular language, confirm it in that SDK’s documentation and in a short test that fills the queue deliberately.

Fixed numbers are chosen for an average day. Real traffic is not average, so mature teams add a few mechanisms that react while the system is running.

Static Versus Dynamic Settings

Everything covered so far is a static setting: you pick a number, deploy it, and it stays until someone changes it. That works when traffic is predictable. When traffic swings widely, a dynamic layer on top of the static settings gives you room to maneuver without redeploying anything. The dynamic layer never replaces the careful sizing from earlier chapters. It only decides what to sacrifice first when the system comes under pressure.

Autoscaling the Gateway Tier

The most common dynamic mechanism is adding or removing gateway Collectors as load changes. It works best for stateless pipelines, where any Collector can handle any request. Scale on signals that rise before trouble starts, such as CPU use and exporter queue utilization, and scale out quickly but scale in slowly, so a brief lull does not remove capacity just before the next burst. Stateful tiers, such as tail sampling, need much gentler scaling because changing the number of Collectors reshuffles which Collector owns which trace.

Priority Lanes

Not all telemetry is equally valuable. Error traces, payment logs and health metrics matter more than debug messages and low-value routes. A Collector can route data into separate pipelines by attribute, giving the important lane its own small batch timeout, its own queue and its own export path. The bulk lane gets larger batches and a longer timeout. When the backend slows, the bulk lane fills first and drops first, while the priority lane keeps flowing.

Load Shedding

Load shedding means deliberately discarding the least valuable data once pressure passes a threshold, so the valuable data survives. A pipeline might start dropping debug-level logs when queue utilization passes a set level, or lower its sampling rate for noisy routes. Some tools support changing sampling rules remotely, so a central controller can tell thousands of processes to keep fewer traces without a redeploy. Management protocols such as OpAMP, designed for controlling fleets of telemetry agents, aim at the same goal for configuration in general.

Guard Rails for Adaptive Systems

Systems that adjust themselves can also oscillate. A controller that drops data when the queue is high and restores it when the queue is low can flip back and forth every few seconds, creating unstable behaviour that is harder to debug than the original problem. A few simple rules prevent this.

• Use separate thresholds for entering and leaving a mode. For example, start shedding at 80 percent queue use but stop only when it falls below 50 percent.
• Set hard floors and ceilings. A sampling rate should never be automatically lowered below a minimum that still lets you see errors.
• Make every change visible. Each automatic adjustment should appear in a metric and a log line, so that a human can tell why data volume changed.
• Practise manual override. On-call engineers need a documented switch that turns adaptive behaviour off during unusual incidents.

The same ideas show up differently depending on the shape of your traffic. These illustrative scenarios show how teams adapt the settings.

When you are ready to tune a real system, work in this order. Each step removes uncertainty from the next.

Short answers to the questions that come up most often when teams start tuning.

A quick reference for the terms used throughout this guide.

Batching in OpenTelemetry is a chain of waiting rooms. Tuning is deciding how large each room is and how long people may wait in it.

You started with a simple idea: sending items in groups is cheaper than sending them one by one. You then followed the idea through the SDK queue, the Collector batch processor, the exporter queue and the backend, and saw that every layer has its own size, its own timer and its own failure behavior. The arithmetic that ties them together is short, and measured numbers make it reliable.