Before we scale something, we need to know exactly what it is and what its moving parts are.

OpenTelemetry (often shortened to OTel) is an open standard for creating and moving telemetry data. It defines how applications describe what they are doing. The OpenTelemetry Collector is a standalone program, a vendor-neutral piece of software, that receives telemetry from many sources, optionally changes it, and sends it to one or more destinations. Think of it as a smart pipe placed between your applications and your monitoring tools.

Why does it exist? Without a Collector, every application would need to know the address, login, and data format of every monitoring tool. If you changed tools, you would have to change and redeploy every application. With a Collector in the middle, applications only talk to the Collector. You change the destination in one place. The Collector also takes work away from applications: batching data, retrying failed sends, removing sensitive fields, and adding helpful labels.

The five building blocks

Every Collector is assembled from a few kinds of components. Learn these names well because the rest of this guide uses them constantly.

The three signals

OpenTelemetry carries three main kinds of data, called signals. Traces follow a single request as it travels through many services, like a package tracking record. Metrics are numbers measured over time, such as requests per second or CPU use. Logs are text records of events. Each signal has its own pipeline in the Collector, and each behaves differently when you scale, which is why scaling is not a one-size-fits-all topic.

flowchart LR
  A["Applications"] --> R["Receivers: OTLP, Prometheus, filelog"]
  R --> P1["Processor: memory limiter"]
  P1 --> P2["Processor: attributes and filters"]
  P2 --> P3["Processor: batch"]
  P3 --> Q["Sending queue"]
  Q --> E["Exporters: backend, Kafka, another Collector"]
    

Read the diagram from left to right. Data enters through a receiver, passes the memory limiter first so the Collector protects itself, gets labeled or filtered, is grouped into batches for efficiency, waits briefly in a queue, and finally leaves through an exporter. Every scaling pattern in this guide is really a clever way of arranging many copies of this same pipeline.

Scaling is not a luxury. It is what separates a monitoring system that works in a demo from one that works at 3 a.m. during an outage.

When a company has five services, one Collector copy handles everything with room to spare. But telemetry volume grows in three ways at once. First, more services appear. Second, each service becomes busier. Third, engineers keep adding more detail, such as extra labels and more spans per request. The volume multiplies rather than adding up.

The cruel part is timing. Telemetry spikes exactly when systems are under stress. When an incident happens, services throw more errors, write more logs, retry more calls, and create more traces. That is the moment your Collector is hit hardest and the moment you need it most. A Collector that collapses during an incident leaves you blind at the worst possible time.

What actually goes wrong without scaling

When a single Collector is overloaded, several bad things can happen. Its memory fills up and the operating system kills it, which loses all data held inside. Its queues overflow and new data is thrown away. Applications that send data directly may slow down while waiting for an answer. Backends that are slow or temporarily offline cause data to pile up inside the Collector until it breaks. And one noisy application can starve all the quiet ones, because they share the same pipe.

Throughout this guide, keep four goals in mind: throughput (how much data per second), reliability (how little data is lost), latency (how quickly data arrives), and cost (how much computing power and network traffic you pay for). Every pattern trades these against one another.

You cannot scale what you do not understand. Let us follow one tiny piece of data through the machine.

Suppose a shopping app finishes a checkout request and creates a span. A span is one timed step of work, like “charge the credit card took 120 milliseconds.” Here is the full journey of that span through a Collector.

Two ideas to remember: stateless and stateful

A stateless component treats every piece of data on its own and remembers nothing between items. Batching, filtering, and adding labels are mostly stateless. If you add ten more Collector copies doing only this, any copy can handle any data. Scaling is easy.

A stateful component must remember earlier data to do its job. Tail-based sampling (which waits to see a whole trace before deciding to keep it), metric aggregation, and deduplication are stateful. They need related pieces of data to land on the same Collector copy. Scaling these is the hard part of this entire topic, and a large share of the patterns below exist purely to solve it.

Every system has a weakest link. Finding it tells you which kind of scaling will actually help.

There are two basic ways to give a Collector more power. Vertical scaling means giving one Collector a bigger machine, with more CPU and memory. Horizontal scaling means running more copies of the Collector side by side and sharing the work among them.

Vertical scaling is simple but hits a ceiling, and a single large process is a single point of failure. Horizontal scaling has almost no ceiling and gives resilience, but it introduces the question this guide keeps returning to: how is data divided among the copies?

What actually runs out first

Notice that backend capacity is on the list. Beginners often scale the Collector and then wonder why nothing improves. If the database that stores your traces can only absorb a certain rate, adding more Collectors only delivers the overflow faster. Sometimes the right answer is to sample, meaning deliberately keep only a fraction of the data, rather than scaling out.

Put a small Collector right next to your workloads, so data takes a short, safe first hop.

In the agent pattern, a Collector runs on every machine (every node, in Kubernetes terms, usually through a DaemonSet, which is a rule that says “run exactly one copy on each machine”). Applications on that machine send their telemetry to the local agent over a very short, fast, local connection. The agent then forwards the data onward.

Why agents help scaling

The agent does work close to the source. It can batch many tiny sends into a few large ones, which reduces network chatter dramatically. It can add machine-level labels (which node, which pod, which cloud zone) cheaply because it knows the local environment. It can collect machine-level data on its own, such as CPU and disk statistics of the host, or read log files from local disk. And because every agent handles only the workloads on one machine, the load is naturally spread out. Add a machine, and you automatically add an agent.

What agents should not do

Agents are small and numerous, so they should be lean. Heavy work such as tail sampling does not belong here because one agent only sees a slice of each trace. Putting big buffers on every machine also wastes memory across the whole fleet. A good agent batches, labels, protects itself, and forwards.

Give each application its own private Collector for the strongest isolation.

A sidecar is a Collector placed inside the same group of containers (a pod, in Kubernetes) as one application. The application and its sidecar share a network, so the application sends data to “localhost” and the sidecar takes it from there. Each application instance has its own dedicated Collector.

The big advantage: isolation

With sidecars, a chatty application can only overload its own Collector. Different teams can use different processing rules, such as one team hiding customer identifiers while another keeps everything. You can even give a sensitive application a sidecar with special security settings without changing anyone else’s setup.

The big cost: overhead

Every sidecar needs its own memory and CPU. If you run ten thousand application pods, you run ten thousand extra Collector processes. Even if each is small, the total adds up quickly, and the number of things to monitor and upgrade grows with it. Sidecars also start and stop together with the application, so short-lived jobs may end before the sidecar finishes sending its last batch. That can cause missing data at the end of a job unless shutdown is handled carefully.

A central pool of Collectors that every source talks to, scaled by adding more identical copies.

A gateway is a group of Collector copies running as a shared service, usually behind a load balancer. A load balancer is a traffic director that spreads incoming connections across several identical servers. Applications or agents send their data to one stable address, and the load balancer hands each connection to one of the gateway copies.

Why gateways are popular

Gateways give you one controlled place for heavy or sensitive work. You can hold the credentials for the backend in one place instead of on every machine. You can apply company-wide rules, such as removing personal data or enforcing size limits, once. You can change the destination backend without touching any application or agent. And because gateway copies are stateless for most pipelines, you can scale them by simply adding more copies or by using automatic scaling based on CPU or memory.

The sizing mindset

Begin with a few copies spread across different failure zones (different data centers or availability zones), so the loss of one zone does not stop collection. Test with realistic data to learn how much one copy can handle. For illustration only: if a copy comfortably handles 40,000 items per second and you expect 120,000 at peak, three copies would be exactly at the limit. You would run five or six to leave headroom for spikes and for the moment one copy is restarting.

The most common production design combines the last two patterns, taking the best of each.

In the two-tier pattern, small agents run close to the applications and a scaled pool of gateway Collectors sits behind them. The agents do the cheap local work: collect, label, batch. The gateways do the shared work: apply company rules, handle credentials, and talk to the backends. Each tier scales on its own terms. Agents scale with the number of machines. Gateways scale with the total volume of data.

flowchart LR
  A1["App Pod A"] --> AG1["Agent Collector on Node 1"]
  A2["App Pod B"] --> AG1
  A3["App Pod C"] --> AG2["Agent Collector on Node 2"]
  AG1 --> LB["Load Balancer"]
  AG2 --> LB
  LB --> G1["Gateway Collector 1"]
  LB --> G2["Gateway Collector 2"]
  LB --> G3["Gateway Collector 3"]
  G1 --> BE["Observability Backend"]
  G2 --> BE
  G3 --> BE
    

Follow the arrows. Application pods send data to the agent on their own node. Agents forward to a load balancer, which spreads the batches across the gateway copies. The gateways deliver to the backend. If Gateway 2 crashes, the load balancer simply stops sending to it, and the agents keep working. If the backend slows down, the gateways absorb the delay in their queues, and agents continue to accept data from applications.

Why this design earns its popularity

First, it separates concerns. Application teams only know “send to the local agent.” Platform teams own the gateways and can change backends without anyone noticing. Second, it reduces connection counts. Instead of ten thousand applications each opening connections to the gateways, perhaps a few hundred agents do. Fewer connections mean less overhead and easier load balancing. Third, it creates a buffer in two places, which adds resilience.

The cost of two tiers

You now operate two kinds of Collector, with two sets of settings to keep consistent and two places to watch. Each extra hop adds a little delay and a little chance of failure. A change that must apply everywhere has to be thought through twice. Use this pattern when the scale justifies it. A small company with a dozen services may be perfectly happy with only a gateway.

Adding copies only helps if work actually reaches them. This is the most common hidden problem in real setups.

OpenTelemetry data is usually sent using gRPC, a fast communication method built on HTTP/2. A key feature of gRPC is that it opens one connection and keeps it open for a long time, sending many requests over it. This is efficient, but it surprises basic load balancers.

A Layer 4 load balancer decides where to send traffic when a connection is first opened, and then every request on that connection goes to the same server. It balances connections, not requests. So if one busy agent opens a single long connection and sends a huge amount of data through it, one gateway copy gets all that load while others sit idle. A Layer 7 load balancer understands the requests inside the connection and can spread them individually across servers.

How the imbalance shows up

You scale from three gateway copies to six. CPU on three copies stays high and the new ones are nearly idle. Dashboards show uneven refused data across copies. After a rolling restart, the first copies to come back receive all the reconnecting clients and stay overloaded. These are classic signs that connections, not requests, are being balanced.

Ways to fix it

One more detail: when a gateway copy is removed during a scale-down or an upgrade, clients on it must reconnect. A graceful shutdown that stops accepting new data, finishes sending what it holds, and then exits makes this smooth. A sudden kill loses whatever was in memory.

Some decisions need the whole story. Scaling them means keeping the whole story in one place.

Recall that a trace is made of many spans from many services. Sampling means keeping only some traces to save money. There are two kinds. Head sampling decides at the very start of a trace, usually by random chance, such as “keep one in a hundred.” It is cheap and stateless, but it is blind: it might throw away the one trace that contained an error. Tail sampling waits until most of a trace has arrived, then decides. It can say “keep every trace with an error, keep every slow trace, and keep a small random share of the rest.” That is far smarter, but it needs all spans of one trace in the same Collector at the same time.

The problem

Spans from one trace come from different services, flow through different agents, and arrive at a pool of gateway copies through a load balancer that has no idea which spans belong together. They scatter. A tail sampling Collector then sees fragments and makes bad decisions, such as discarding half of an important trace.

The solution: a routing tier

The fix is a two-stage design. The first stage is a pool of Collectors using a load-balancing exporter. For each span, it looks at the trace ID (the unique number shared by all spans of a trace), computes a consistent hash, and sends the span to a specific second-stage Collector. The same trace ID always maps to the same destination. The second stage is a pool of Collectors that run the tail sampling processor. Because every span of a given trace lands on the same one, each can decide with full information.

flowchart LR
  AG["Agents and Applications"] --> T1["Tier 1 Collectors with load-balancing exporter"]
  T1 -->|"trace ID hash"| T2A["Tier 2 Collector A with tail sampling"]
  T1 -->|"trace ID hash"| T2B["Tier 2 Collector B with tail sampling"]
  T1 -->|"trace ID hash"| T2C["Tier 2 Collector C with tail sampling"]
  T2A --> BE["Tracing Backend"]
  T2B --> BE
  T2C --> BE
    

The diagram shows tier one as a stateless router that can be scaled freely. Tier two holds the memory-heavy, stateful buffers, and it scales by adding more members to the hash ring. The load-balancing exporter can discover tier-two members in several ways: a fixed list, a DNS name that returns all addresses, or by watching the Kubernetes service for changes.

Details that bite in production

• Decision wait time. The sampler holds spans in memory for a configured period before deciding. Longer waits catch slow traces but use more memory. Shorter waits use less but may decide before late spans arrive.
• Memory sizing. Memory needed is roughly the span rate multiplied by the wait time and by the average span size. A fleet receiving a high rate with a long wait needs a lot of memory per copy. Plan for it and test with realistic load.
• Resizing disrupts traces. When a tier-two copy is added or removed, the hash assignments change, and traces in flight at that moment may be split between two copies. A small amount of imperfect sampling during scaling events is normal. Avoid frequent automatic resizing of this tier.
• Other routing keys. Besides trace ID, you can route by service name so that all data from one service reaches one Collector. This is useful for generating per-service metrics from spans.

When the backend cannot keep up, give the data a safe place to wait.

A Collector’s internal queue is small and lives in memory. It handles short hiccups, but it cannot survive a backend outage that lasts an hour or a Collector crash. For stronger protection, teams put a message queue, most often Apache Kafka, between two stages of Collectors. A message queue is a durable storage system where producers write data and consumers read it at their own pace. Because the data is written to disk and replicated, it stays safe even if consumers are down.

flowchart LR
  P["Producers: Agents and Applications"] --> C1["Ingest Collectors with Kafka exporter"]
  C1 --> K["Kafka topics for traces, metrics, logs"]
  K --> C2["Processing Collectors with Kafka receiver"]
  C2 --> BE["Backends"]
    

On the left, ingest Collectors accept data from the world as fast as it arrives and immediately write it to Kafka topics, which are named, ordered streams. On the right, processing Collectors read from Kafka at a rate the backend can handle. The two sides are independent. A spike on the left simply makes Kafka hold more data. An outage on the right simply makes Kafka hold data until the right side recovers.

What you gain

• Durability. Data survives Collector crashes and backend outages for as long as Kafka retains it.
• Replay. If you fix a bug in processing or add a new backend, you can re-read recent data from Kafka.
• Independent scaling. Ingest and processing tiers are sized separately. Kafka partitions let you add more consumers to share the load.
• Natural fan-out. Several consumers can read the same topic for different purposes, such as one feeding a trace store and another feeding an analytics system.

What you pay

Kafka is another large system to run, secure, monitor, and pay for. It adds delay, so data takes longer to appear in dashboards. Partition counts and message sizes need thoughtful choices, and a poorly chosen setup can create its own hot spots. Do not add it casually. It earns its place when you cannot afford to lose data during backend outages, when traffic is extremely bursty, or when many consumers need the same stream.

Each signal has its own personality. Metrics and logs bring special challenges that traces do not.

Split pipelines by signal

A simple and powerful pattern is to run separate Collector pools for separate signals. Traces are bursty and memory-hungry when sampled. Metrics are steady but can have enormous numbers of distinct series. Logs are high-volume and mostly just need to be moved. If they all share one pool, a log flood can starve trace processing. Separate pools give each signal its own scaling rule, its own limits, and its own failure boundary.

Scaling Prometheus-style scraping

Prometheus is a popular system where metrics are pulled: a scraper visits each target, such as a service endpoint, every few seconds and reads its numbers. The Collector can act as that scraper. But if a single Collector scrapes thousands of targets, it becomes a bottleneck. If you simply run several Collectors with the same list, each target is scraped several times and you get duplicate data.

The solution in Kubernetes is the Target Allocator, a helper offered with the OpenTelemetry Operator. It discovers all scrape targets and divides them among Collector replicas, giving each replica its own share. If you add a replica, targets are redistributed. Several strategies exist, such as spreading by consistent hashing, by balancing the number of targets, or by assigning each node’s targets to the replica on that node.

flowchart TB
  SD["Service discovery: pods and services"] --> TA["Target Allocator"]
  TA -->|"assigns targets"| R1["Collector Replica 1"]
  TA -->|"assigns targets"| R2["Collector Replica 2"]
  TA -->|"assigns targets"| R3["Collector Replica 3"]
  R1 --> M["Metrics Backend"]
  R2 --> M
  R3 --> M
    

Stateful metrics work

Some metrics processing needs memory of the past. Converting running totals (cumulative values) into per-interval changes (delta values) requires remembering the previous reading of each series. Combining readings from many sources into one number requires seeing all of them in one place. And generating metrics from spans, using the span-metrics connector, keeps counts and timers inside each Collector. If several copies each produce part of a metric with identical labels, backends receive conflicting or overlapping values. Two remedies work well: route all data for one series or one service to one Collector using the load-balancing exporter, or add a label that identifies the producing Collector so that each copy writes its own distinct series that you can sum later.

Scaling logs

Logs are usually read from files on each machine, so the natural pattern is an agent on every node reading local log files. The agent remembers how far it has read in each file through a checkpoint, so it can continue after a restart without repeating or skipping lines. Logs are mostly stateless to process, so the gateway tier for logs scales like any stateless pool. The main risks are sudden floods, such as an application stuck in an error loop, which can produce huge volumes in minutes. Put limits and filters early, drop clearly useless lines at the agent, and use a message queue if you must never lose logs during backend problems.

Scaling out is not enough. Each copy must also survive overload without crashing and losing everything.

Collectors should degrade gracefully. When too much arrives, they should slow down or refuse some data deliberately instead of falling over. Four built-in tools make this possible.

Backpressure: pushing the problem upstream

Backpressure is the idea that when a downstream part is full, it signals upstream parts to slow down, instead of silently losing data. When the memory limiter refuses data, a well-behaved sender, such as an agent or an application SDK, keeps the data in its own buffer and retries. That works as long as the upstream has room to wait. In a multi-tier design, backpressure can ripple back from the backend to the gateways to the agents to the applications. SDK buffers are small, so ultimately data may be dropped at the edge. Decide consciously where you would rather lose data: at the edge, where it costs little to drop, or deep in the pipeline, where much work has already been spent.

Order of processors matters

A widely recommended order is: memory limiter first, then enrichment and filtering, then sampling, and batching near the end. Putting the batch step too early means that later steps handle fragments rather than efficient groups, and putting the memory limiter late means data piles up before the guard can act.

Keep copies slim

The standard distributions of the Collector include hundreds of components. For production, you can build a custom Collector containing only the components you use, using the official builder tool. A smaller build starts faster, uses less memory, and exposes less surface for security problems.

Letting the platform add and remove copies automatically sounds perfect. It works well for some layers and badly for others.

Autoscaling means the platform watches a signal, such as CPU use, and adds or removes Collector copies to match demand. In Kubernetes, the Horizontal Pod Autoscaler does this based on metrics like CPU or memory. Tools such as KEDA can scale based on external signals, such as how far behind a Kafka consumer group is.

Where autoscaling works well

• Stateless gateways. Any copy can handle any data, so adding or removing copies is harmless. Scale on CPU use, which tracks throughput well for most pipelines.
• Kafka consumers. Scale on consumer lag, which directly measures how far behind processing is.
• The first routing tier. It is stateless, so it scales like a gateway.

Where autoscaling is risky

• Tail sampling tier. Every resize changes the hash assignments and splits in-flight traces. Prefer a fixed size with headroom, adjusted deliberately.
• Metric scrapers. Resizing causes targets to move between replicas, which may produce gaps or duplicates around the change.
• Memory-based triggers. Collector memory depends on queue and batch settings and on Go’s garbage collection rhythm, so it can rise and fall in ways that do not match real load. CPU or queue depth are steadier signals.

Tuning to avoid flapping

Flapping is when the system keeps adding and removing copies in rapid cycles. Each removal can drop in-memory data and force reconnections. Avoid it with a generous scale-down delay, a gentle scale-down rate, and a minimum number of copies that already covers normal peaks. Scale up quickly, scale down slowly. Also ensure new copies start fast and pass readiness checks before receiving traffic.

More copies and more hops mean more doors. Each one needs a lock.

Telemetry often contains sensitive details: user identifiers, internal addresses, error messages with fragments of customer data, and sometimes secrets that were logged by accident. A scaled setup moves this data across many networks, so security must be designed in from the start.

Protecting data on the move

Use TLS, the same encryption that protects web browsing, on every link between applications, agents, gateways, queues, and backends. For links between your own components, use mutual TLS, where both sides prove their identity with certificates, so only trusted senders can talk to a gateway. Automate certificate renewal, because expired certificates are a classic cause of sudden monitoring outages.

Controlling who can send

A gateway that accepts data from anyone is an open door. Add authentication, using the Collector’s authentication extensions, such as tokens or identity-provider checks. In multi-team or multi-customer platforms, attach a tenant identity to each sender and use it to route data and enforce limits, so that one tenant cannot flood or read another’s data. Rate limits at the edge protect the whole fleet from one misbehaving sender. Also limit network exposure: bind receivers to the addresses they truly need, and use network policies so only expected callers can reach them.

Cleaning data before it travels onward

Processors can delete or mask sensitive attributes, hash identifiers, and drop fields completely. Do this as early as is practical, ideally at the agent or the first gateway, so sensitive data does not travel further than necessary. Treat these rules as shared policy managed in one place, and test them, because a rule that silently fails to match is worse than none.

Protecting secrets and the Collector itself

• Keep backend credentials in a secret store, not in plain text settings files.
• Run Collectors with the fewest privileges possible. An agent that reads log files needs read access to them, not full control of the machine.
• Use slim custom builds to reduce the number of components that could contain a flaw.
• Keep versions current and roll out upgrades gradually across the fleet.
• Restrict who can change Collector settings, since a changed setting can redirect all your telemetry.

Who watches the watchers? You must, or you will discover failures only when dashboards go quiet.

The Collector exposes its own metrics about its health and traffic, normally on a built-in port in a format monitoring systems can read. It can also produce its own logs and, if needed, traces. Treat the Collector fleet as a production service with its own dashboards and alerts. Exact metric names vary slightly between versions, but the ideas below are stable.

Alerts that matter

Alert on symptoms rather than raw numbers. Good examples: refused or dropped items above zero for several minutes; export queue above a high share of its capacity; a Collector restarted unexpectedly more than once recently; throughput on one copy much higher than its peers; backend export failure rate above a small threshold. Also add a heartbeat: a known test signal sent regularly through the whole path, so you notice if the entire pipeline stops, even when nothing inside reports an error.

Monitor the monitor separately

If the Collectors send their own metrics through the same pipeline that they are monitoring, a failure can hide itself. Send Collector health data through a separate, simple, independent path whenever possible.

There is no single best design. The right one depends on your size, your data, and your tolerance for risk and cost.

A simple growth path

You do not need everything on day one. A sensible journey looks like this. Start with applications sending to a small gateway of two or three copies in different zones. Add agents when you want host metrics, log collection, or fewer connections to the gateway. Fix connection balancing as soon as you see uneven load. Add the memory limiter, batching, and queue limits everywhere from the start. Introduce trace-aware routing only when you need tail sampling at volume. Introduce Kafka only when losing data during backend problems is truly unacceptable. Split pools by signal when one signal’s spikes begin to hurt another.

That last point deserves emphasis. Before building an elaborate fleet, ask whether you really need every item. Filtering health checks, trimming oversized attributes, lowering log levels, and sampling sensibly can cut volume by half or more, which is cheaper and simpler than scaling infrastructure to carry the waste.

Most scaling failures come from a short list of repeated mistakes. Learn them here instead of at 3 a.m.

More mistakes worth avoiding

• Huge queues “just in case.” They hide problems, consume memory, and lose more data when the copy dies. Size queues for a realistic outage window.
• Tiny batches. They create enormous numbers of small requests and waste CPU and network.
• Ignoring shutdown behavior. Killing copies without a graceful stop drops buffered data on every deployment.
• No headroom. Running at ninety percent of capacity means a single restart overloads the rest.
• Different settings per copy. Manual edits create drift. Manage settings as code, reviewed and rolled out gradually.
• Unbounded label values. Putting identifiers with endless variety into metric labels will exhaust memory somewhere.
• No test of failure. Teams rarely practice backend outages. Deliberately break the backend in a test environment and watch how queues, retries, and alerts behave.

Best practices in one place

Three typical situations, described as realistic illustrations, show how the patterns combine.

Multi-cluster and multi-region thinking

When you operate many clusters or regions, a common approach is a regional gateway in each one that gathers local data and forwards it onward, sometimes to a central backend. Benefits include shorter hops, regional failure isolation, and the ability to apply local compliance rules about where data may travel. The trade-off is more places to configure and more cross-region traffic to pay for if everything is eventually centralized. Sampling at the regional gateway, before data leaves the region, can reduce those transfer costs sharply.

A back-of-the-envelope sizing exercise

Suppose you expect, at peak, 300,000 spans per second. A load test shows one gateway copy comfortably handles 50,000 per second at about sixty percent CPU. Dividing, you need six copies for the peak. Adding headroom for one failed copy and a spike margin, you run nine. If instead you use tail sampling with a wait of thirty seconds and an average span of one kilobyte, each second of incoming data takes roughly a thousandth of a gigabyte per thousand spans, so buffering thirty seconds of 300,000 spans per second is on the order of nine gigabytes spread across the sampling tier. With four sampling copies, each needs a few gigabytes of working memory plus overhead. These numbers are only an illustration, but the method is real: measure one copy, divide by peak, add headroom, and calculate memory from rate, wait time, and size.

A pattern on a whiteboard becomes real only when it is turned into running software with safe rollouts.

Most large Collector fleets run on Kubernetes, a system that runs containers across many machines and restarts them when they fail. Kubernetes offers several ways to run a group of containers, and each way matches one of our patterns. Understanding the match saves a lot of confusion.

The OpenTelemetry Operator is a helper program for Kubernetes that understands these shapes. You describe a Collector in a single resource, pick its mode (daemon set, deployment, stateful set, or sidecar), and the Operator creates and updates the running pieces. It can also inject sidecars into pods that carry a special label, and it runs the Target Allocator when you ask for it. This means platform teams manage Collectors as one declarative thing rather than a pile of hand-built objects.

Giving the fleet a stable front door

Applications and agents need one stable address for the gateway. Kubernetes gives a group of copies a Service, a stable name that always leads to healthy copies. A normal Service spreads connections at Layer 4, which brings us back to the balancing puzzle from Chapter 9. A headless Service is a special kind that, instead of hiding the copies behind one address, lists all of their addresses when you ask. That is exactly what the load-balancing exporter and client-side balancing need, because they want to see every member in order to choose among them.

Rolling out changes safely

Collectors sit on the data path, so a bad rollout can blind you. Follow these habits.

• Roll slowly. Replace a small number of copies at a time so the rest keep serving traffic. Wait for each new copy to report ready before continuing.
• Protect minimum availability. A disruption budget tells Kubernetes the smallest number of copies that must stay running during maintenance, so a node upgrade cannot take down the whole gateway at once.
• Spread copies. Use rules that place copies on different nodes and in different zones, so one failure cannot remove most of your capacity.
• Set requests and limits wisely. A request tells the scheduler how much CPU and memory to reserve. A limit is the ceiling. Setting the memory request equal to the limit makes behavior predictable. Be careful with tight CPU limits, because throttling causes slow receives that look like network problems.
• Test config before shipping. Validate settings files in an automated step. A single typo in a pipeline name can cause a copy to refuse to start.
• Canary first. Send a small slice of traffic to the new version, compare its error and drop rates with the old version, and only then proceed.

Managing configuration as code

Store every Collector setting in version control, review changes like any other code, and apply them through an automated pipeline. This gives you history (“what changed before the outage?”), easy rollback, and consistency across copies. For fleets with many different Collectors, many teams generate settings from shared templates so the memory limiter, batching, and security settings are always the same and cannot be forgotten.

Outside Kubernetes

The same ideas apply on plain virtual machines, container services from cloud providers, or other schedulers. On virtual machines, an agent is a service installed on each machine, and a gateway is an auto-scaling group behind a load balancer. In cloud container services, an agent can run as a helper container beside each task, while a gateway runs as its own service. What changes is only the tooling that places and restarts copies. The patterns, risks, and sizing method stay the same.

A few small mechanisms decide whether a fleet is calm or chaotic. Let us look at each closely.

How batching really saves resources

Every network request has a fixed cost: opening the exchange, reading headers, checking authentication, compressing, and acknowledging. If you send one thousand items as one hundred requests of ten, you pay that fixed cost one hundred times. If you send them as one request of a thousand, you pay it once. Batching therefore trades a bit of delay for a lot of efficiency. A batch is sent when it reaches a target size or when a timer expires, whichever comes first. The timer guarantees that quiet periods do not leave data stuck forever.

There is a sweet spot. Very small batches waste CPU. Very large batches use more memory, create bigger spikes on the backend, and may exceed message size limits that the backend or a queue enforces. Watch the size of batches actually sent, not just the setting, because under light load the timer often fires first and batches are smaller than configured.

Queue arithmetic

The sending queue protects against short backend trouble. To size it, think about how long an outage you want to survive and how fast data arrives. If a copy ships ten batches per second and you want to ride out a thirty-second slowdown, the queue must hold about three hundred batches. Multiply by the typical batch size in bytes and you know the memory cost. If that cost is too big for the copy, you have three honest choices: accept dropping data after a shorter window, spill to disk with a persistent queue, or move buffering to a message queue.

Remember also that a full queue is an important signal. It does not mean “raise the queue size.” It means the exporter cannot send as fast as data arrives, which points to a slow backend, too few exporter workers, a network limit, or too much input. Enlarging the queue only postpones the problem and increases what you lose when the copy dies.

Retries and backoff

When a send fails, the exporter waits and tries again. Good retry logic uses exponential backoff: the wait grows after every failure, such as one second, then two, then four, up to a ceiling. It also adds a little randomness, called jitter, so that hundreds of copies that failed at the same moment do not all retry at the same instant and hammer a recovering backend. There is also a total time limit after which the data is dropped, because retrying forever fills the queue.

Compression

Telemetry is repetitive text-like data, so it compresses very well. Compressing between tiers can cut network traffic dramatically at the cost of some CPU. In cloud environments where traffic between zones or regions is billed, this saving can exceed the CPU cost many times over. Test your own data, since the balance depends on its shape.

Consistent hashing, explained simply

Our routing tier must map each trace ID to a specific second-stage Collector. The obvious idea is “take the trace ID as a number, divide by the count of copies, and use the remainder.” This works until the count changes. If you go from four copies to five, almost every trace ID suddenly maps somewhere new, and traces in flight are split everywhere.

Consistent hashing solves this. Imagine a clock face. Each Collector is placed at several points around the clock. Each trace ID is also placed on the clock and goes to the first Collector found moving clockwise. When a Collector is added, it takes over only the arcs just before its points, and everything else stays put. On average only about one fifth of traces move when you grow from four to five copies, instead of nearly all of them.

Even with consistent hashing, a small share of traces is disturbed when membership changes. That is why the guidance in earlier chapters keeps saying: resize stateful tiers rarely, and do it on purpose.

A design is only as good as its behavior on the worst day. Walk through the failures before they happen.

Principles for every playbook

• Fail small. Design so that every failure affects a small slice: one node, one tenant, one signal, one zone.
• Fail loudly. Dropped data should always raise a metric and an alert, never disappear silently.
• Decide where to lose data. When loss is unavoidable, prefer dropping low-value, high-volume items (debug logs, healthy traces) before errors and key metrics. Use priorities or separate pipelines to make this possible.
• Practice. Run game days where you break the backend or kill a zone in a test environment, and write down what you learned.

Once the basic patterns are in place, three bigger questions decide whether the system stays affordable and fair.

Combining head and tail sampling

Head sampling is cheap and stateless, tail sampling is smart and expensive. Many mature setups use both. A light head sample at the agent or SDK removes obvious waste, such as traces from health-check endpoints or a fixed random share of extremely high-volume, low-risk services. The surviving traces go to the tail sampling tier, which keeps all errors, all slow traces, and a small baseline of ordinary ones. This reduces the volume the stateful tier must hold in memory, which directly shrinks its size and cost. The caution is that a head decision cannot be undone, so only apply it where you are sure losing the trace costs nothing.

Generating metrics from traces before sampling

If you sample traces heavily, counts derived from the remaining traces understate real traffic. A better approach is to produce request counts and latency distributions from the full stream of spans before sampling, using a connector that turns spans into metrics, and then sample the traces afterward. This way your dashboards show accurate rates even though most individual traces are discarded. Because such a connector keeps running totals in memory, remember the earlier advice: route all data for one service to one copy, or label per-copy outputs so they can be combined safely.

Multi-tenancy and fairness

In shared platforms, many teams use the same fleet. Fairness needs three things. Identity: every sender is known, through certificates, tokens, or labels added at the edge. Limits: each tenant has a cap on rate and on cardinality, so one team cannot consume the whole fleet. Routing: data from different tenants can be sent to different destinations, or different Collector pools for very large tenants, using routing rules based on tenant labels. Visibility completes the picture: per-tenant dashboards of volume, drops, and cost, so teams can see their own impact. When people see the bill for noisy telemetry, they fix it.

Thinking about cost

The cost of an observability pipeline has four parts: compute for Collectors, network transfer between zones and regions, storage and indexing in the backend, and the human time to operate it all. Most of the time, the backend storage and the network transfer dwarf the Collector compute. So the biggest savings usually come from sending and storing less, not from shaving Collector size. Practical levers include dropping low-value data at the agent, sampling before data crosses regions, compressing between tiers, controlling label cardinality, shortening retention for low-value data, and keeping tiers right-sized. Review the volume by source regularly, because telemetry grows quietly.

Upgrading a large fleet

Collector versions move quickly, and component behavior can change between releases. Keep a documented upgrade routine: read release notes for the components you use, test the new version against recorded or synthetic traffic, canary it in one region or one tier, compare drop and error rates against the old version, and only then roll across the fleet. Upgrade the stateless tiers first, the stateful tiers last, and avoid upgrading several tiers in the same window so that any problem is easy to attribute.

A practical decision checklist

When you face a new scaling question, walk through these questions in order. They turn a vague worry into a concrete choice.

Teams that follow this order tend to avoid the two most expensive mistakes: building a complicated fleet to solve a problem that filtering would have removed, and scaling the Collector tier while the real limit sits in the storage backend. Revisit the checklist whenever traffic doubles, a new signal type is introduced, or a major new tenant joins, because each of those events can move the bottleneck to a different place.

The OpenTelemetry Collector grows from a single helpful pipe into a fleet through a small set of repeatable patterns. Agents collect near the source. Sidecars isolate individual applications. Gateways centralize shared work and scale horizontally. Two-tier designs combine them. Routing tiers keep related data together for tail sampling and certain metrics. Message queues absorb bursts and outages. Separate pools keep signals from harming each other. Throughout, the memory limiter, batching, bounded queues, health checks, and graceful shutdown keep each copy from collapsing, while monitoring and rehearsed failures keep you honest.