Before we design anything, we need to understand what the Collector does, and why its failure hurts more than most people expect.

The OpenTelemetry Collector is a standalone program that receives telemetry from applications, optionally processes it, and sends it onward to one or more destinations. Think of it as a sorting centre for observability data. It can accept data in many formats, add or remove information, group small messages into larger batches, drop things you do not want, and deliver the result to the storage or analysis system of your choice.

Because it sits between your applications and your backend, the Collector protects the applications from the backend’s moods. Applications hand data to a nearby, fast, friendly receiver and move on with their work. The Collector then takes on the hard job of delivering that data reliably over a network to a distant system that may be slow or temporarily unavailable. This is a very valuable service, but it also makes the Collector a point of concentration. Whatever the Collector loses, nobody else has a copy of.

What high availability means here

High availability simply means that the service keeps working when parts of it fail. For a Collector tier, that has two practical parts. First, the tier should keep accepting data when one instance dies, so applications never see long errors or stalls. Second, the tier should keep delivering data when the backend or the network misbehaves, so short outages become delays instead of permanent losses.

Each term below comes with what it is, why it exists, and a picture to hold in your head.

You cannot design against failures you have not listed, so let us walk through the realistic ones.

Failures in a Collector tier fall into a few families. Some are about the machine, some are about the network, some are about overload, and some are about people. Naming them makes later design choices feel less like magic and more like answers to specific questions.

High availability for the Collector rests on five simple habits, repeated at every layer.

Notice that none of these habits is about making any single Collector unbreakable. Individual programs crash, disks fill and networks wobble. High availability is the art of making sure those ordinary events stay ordinary, never growing into outages of your visibility. The rest of this tutorial turns each habit into concrete design choices.

Where you place Collectors decides which failures you can survive, so the pattern comes before the settings.

OpenTelemetry describes a handful of deployment patterns. They are not rivals; many real systems combine them. What matters for this tutorial is how each pattern changes the failure picture.

flowchart LR
    subgraph HOSTS["Application hosts"]
        A1["App A"] --> AG1["Agent collector"]
        A2["App B"] --> AG1
        A3["App C"] --> AG2["Agent collector"]
    end
    AG1 --> LB["Load balancer"]
    AG2 --> LB
    subgraph GW["Gateway tier: several replicas"]
        G1["Gateway 1"]
        G2["Gateway 2"]
        G3["Gateway 3"]
    end
    LB --> G1
    LB --> G2
    LB --> G3
    G1 --> BE["Observability backend"]
    G2 --> BE
    G3 --> BE
        

Agents are many and small, so their availability comes from simplicity and from a good path onward.

An agent usually runs one per machine, or as a sidecar next to a workload. Its job is intentionally light: receive local telemetry, add useful context such as the machine and cluster names, protect itself from overload, and forward the result onward. Because each agent serves only the applications on its own host, an agent failure affects only that host, which already limits the blast radius.

Keep the agent boring

The more work an agent does, the more ways it can fail and the more memory it needs. A reliable agent tier keeps processing minimal. Heavy tasks, such as sampling decisions that need whole traces or large aggregations, belong to the gateway tier where resources are shared and managed centrally.

Make restarts cheap

Agents get restarted for upgrades, configuration changes and node maintenance. If an agent holds large amounts of in-flight data in memory, every restart loses it. Keeping queues small and forwarding quickly makes a restart almost invisible. Where the agent runs on infrastructure that restarts often, applications’ own tracing libraries should also have sensible retry settings, so a few seconds of unavailability does not turn into lost spans.

The gateway is the busiest and most important tier, so it gets the strongest redundancy.

A gateway tier is a group of identical Collector replicas sitting behind one address. Because they are identical and hold no memory of earlier requests, any replica can handle any incoming batch, and if one disappears the others simply pick up its share. This is called a stateless design, and it is the easiest kind of design to make highly available.

How many replicas

The OpenTelemetry scaling guidance recommends keeping at least three replicas for resilience. The reasoning is simple arithmetic. With two replicas, losing one halves your capacity at the exact moment your remaining replica must absorb double the traffic. With three, losing one leaves two replicas sharing the load with a much smaller jump per survivor. Three is therefore a sensible minimum, and many teams add more to leave headroom for deployments and maintenance.

Several replicas are only useful if traffic actually reaches all of them, and telemetry traffic has a few quirks.

Applications and agents usually send telemetry over long-lived connections that carry many requests. A basic network-level load balancer, which spreads connections rather than requests, can therefore leave one replica overloaded while another sits idle, because each long connection sticks to the replica it first reached. The scaling guidance recommends load balancers that understand the application-level protocol, so they can spread individual requests across replicas.

flowchart TD
    S["Many senders with long-lived connections"] --> Q{"Balancer type"}
    Q -->|"Connection level only"| U["Uneven load: some replicas busy, some idle"]
    Q -->|"Request aware"| E["Even load: each request goes to a healthy replica"]
    U --> F["Fix: recycle connections or switch balancer"]
        

Health checks decide who gets traffic

A balancer should send traffic only to replicas that are ready. The Collector provides a health check extension that exposes a simple endpoint for this purpose. Orchestration systems and balancers can poll that endpoint to decide whether a replica should receive traffic or be restarted. A replica that is alive but overloaded or still starting should be marked not ready, so it is spared until it recovers.

Before spreading work over many replicas, make each replica hard to kill with its own memory and its own pace.

The most common cause of a Collector crash is simply running out of memory during a spike. When a process exceeds the memory it is allowed to use, the operating system or the orchestrator ends it abruptly, and everything held in memory at that moment is lost. Worse, when one replica dies, its share of traffic moves to the others, which can push them over their limits too. This chain reaction is how a small spike turns into a full outage.

The memory limiter

The Collector offers a memory limiter processor to prevent this. It watches memory use and, when usage approaches a configured limit, starts refusing new incoming data and asks the runtime to release memory, instead of continuing until the process is killed. Refusing data sounds bad, but it is far better than crashing: the senders are told to retry later, and the data already accepted is preserved. The documented recommendation is to place this processor first in every pipeline, so that it can protect everything that follows.

Batching for efficiency

The batch processor groups many small items into larger bundles before they are sent onward. Sending a thousand small requests is far more expensive than sending a few large ones, so batching lowers processor use and network overhead, which also helps resilience because an efficient Collector has more spare capacity. The usual order is memory limiter first, then other processing, then batching near the end.

The exporter is where short outages are turned into short delays, as long as the waiting line is sized with care.

When an exporter cannot deliver data, for example because the backend is restarting, it does not have to give up right away. Exporters can hold data in a sending queue and retry delivery with exponential backoff, which means waiting a little longer between each attempt so the struggling destination is not hammered. By default the retry effort continues for about five minutes before the data is given up on.

Two limits define how much protection a queue gives. The first is time: once the retry window passes, the data it covers is dropped. The second is size: once the queue is full, new incoming data is discarded because there is nowhere to put it. A queue therefore protects against outages that are shorter than both limits, and understanding these two numbers is the heart of capacity planning for resilience.

sequenceDiagram
    participant P as Pipeline
    participant Q as Sending queue
    participant X as Exporter
    participant B as Backend
    P->>Q: Hand over a batch
    Q->>X: Next batch
    X->>B: Send attempt 1
    B--xX: Unavailable
    X->>X: Wait, then retry with longer delay
    X->>B: Send attempt 2
    B-->>X: Accepted
    X->>Q: Done, remove batch
        

When the queue cannot save you

An in-memory queue vanishes when the process dies; a persistent queue writes to disk so work survives.

The Collector can store its queued data on disk using a storage extension. This acts like a write-ahead log: data is written to disk as it is queued, and when the Collector restarts it picks up where it left off instead of starting empty. The resiliency guidance suggests this approach for critical components such as gateways, where the loss of queued data after a crash would be expensive.

The disk must outlive the process, and ideally the machine

A persistent queue only helps if the storage survives the failure you are guarding against. A disk that is local to a machine protects against process crashes but not against losing the whole machine. In container platforms, this means attaching durable volumes to the Collector, and using workload types that keep each replica bound to its own volume across restarts. Where losing a machine must not mean losing data, the volume should be network-attached or replicated.

When a hop must tolerate long outages, put a durable middleman in the middle.

For the strongest protection, a durable message queue such as Kafka can sit between Collector tiers. Agents act as producers that publish telemetry to named topics, and gateway Collectors act as consumers that read from those topics at their own pace. Because the message queue stores data durably and independently, either side can fail or be restarted without immediately affecting the other.

flowchart LR
    AG["Agent collectors: producers"] --> K["Durable message queue: topics"]
    K --> G1["Gateway consumer 1"]
    K --> G2["Gateway consumer 2"]
    K --> G3["Gateway consumer 3"]
    G1 --> BE["Backend"]
    G2 --> BE
    G3 --> BE
        

Plain copying works for stateless tiers but breaks when a component needs to see all related data together.

Some Collector features must remember things between messages. The clearest example is tail sampling, which waits until it has seen enough of a trace to decide whether to keep it, for example keeping traces that contain errors. To decide, one processor instance must receive all the spans of a given trace. If the spans of one trace are spread across several replicas, none of them sees the whole story, and the decision is made on fragments. Span-to-metrics processing has a similar need: all data for the same service must reach the same place to produce accurate totals.

This means that a stateful tier cannot sit behind an ordinary balancer that spreads requests randomly. The scaling guidance describes the solution: put a first tier of stateless Collectors in front, equipped with a load-balancing exporter that picks the destination replica based on the trace identifier or the service name. All spans of one trace then travel to the same backend replica, which holds the complete trace.

flowchart LR
    APP["Agents and applications"] --> LB1["Tier 1: stateless collectors with trace-aware routing"]
    LB1 -->|"Trace A"| T1["Tier 2: tail sampler 1"]
    LB1 -->|"Trace B"| T2["Tier 2: tail sampler 2"]
    LB1 -->|"Trace C"| T3["Tier 2: tail sampler 3"]
    T1 --> BE["Backend"]
    T2 --> BE
    T3 --> BE
        

This small component is the glue of the two-tier design, so it is worth understanding how it chooses and what happens when the group changes.

The load-balancing exporter takes each item of telemetry, looks at a routing key, and sends the item to the backend Collector responsible for that key. Common routing keys are the trace identifier, which keeps whole traces together, and the service name, which keeps all data from one service together for span-to-metrics work. The exporter needs to know which backend Collectors exist, and it can learn that from a fixed list, from DNS records, or from a container platform’s service discovery.

What happens when the group changes

When a replica joins or leaves, the mapping from keys to replicas changes for some keys. Traces that were in progress on one replica may suddenly see new spans arrive at a different one, so a small number of traces can be split during the change. This is unavoidable with any routing scheme, and the practical response is to keep membership changes rare and gradual. Rolling restarts one replica at a time, with a pause between them, limit the number of affected traces compared with restarting everything at once.

Not all data is pushed to the Collector; some of it is fetched, and fetched data brings its own availability puzzle.

Applications push their traces and metrics to a Collector, so adding replicas behind a balancer works naturally. But some receivers pull data, for example by scraping the metrics endpoints of many services at regular intervals. If you simply run three replicas with the same scraping job, each replica scrapes every target, and you receive the same metrics three times. Duplicates inflate totals and confuse dashboards.

The answer is to divide the targets between replicas so that each target is scraped by exactly one. The OpenTelemetry Operator offers a Target Allocator, which distributes the scrape targets across Collector replicas automatically and redistributes them when replicas change. This is also how pull-based scraping scales: when a single replica takes too long to finish one scrape cycle, the targets are split across more replicas.

It is worth being honest about what retries and queues can and cannot promise.

With retries, a sender that does not receive a clear acknowledgement tries again. That protects against loss, but it creates a different risk: the first attempt might actually have succeeded, with only the acknowledgement lost on the way back, so the retry delivers the same data twice. Systems built this way are called at-least-once: every item arrives one or more times, never silently zero times, but sometimes duplicates appear.

Telemetry pipelines in practice aim for at-least-once delivery with small, tolerable duplicate rates. Traces and logs usually tolerate an occasional duplicate better than a gap. Metrics need more care, since a duplicated counter increment would distort totals, which is another reason to avoid running several copies of the same scraper. Backends often handle duplicates by recognising identical identifiers or timestamps.

The honest design statement is therefore not that no data is ever lost. It is something like: for backend outages shorter than a stated window and for single replica failures, no data is lost, and for anything beyond that, loss is bounded, measured and alerted on. That is a promise you can actually keep and test.

High availability fails quietly when capacity runs out, so decide up front how you will know it is time to grow.

The scaling guidance names a few signals to watch. Memory pressure, shown by the memory limiter refusing data, says the tier has reached its limits. Export queue build-up, measured as queue size relative to queue capacity, says the Collector cannot deliver as fast as data arrives; the guidance suggests scaling up when the queue is around sixty to seventy percent full, and scaling down when it stays low for a long time. For scraping receivers, the signal is a scrape cycle that takes almost as long as the scrape interval.

When more replicas will not help

Adding Collectors does not fix a problem that lives downstream. If the persistent queue stays saturated even after adding capacity, or if the count of failed sends keeps rising, the real bottleneck is probably the backend or the network between you and it. More Collectors would only move the pile-up from one place to another, and might even make the backend’s situation worse.

Most Collector fleets run on a container platform, which offers several building blocks that map neatly onto the ideas in this tutorial.

Container platforms such as Kubernetes offer different ways to run workloads, and each suits a different Collector role. Choosing the right one removes a lot of manual work from your availability story.

The availability toolkit of a container platform

The bigger the failure you want to survive, the further apart your copies need to be.

Everything so far protects against a replica or a machine failing. Cloud providers, however, group machines into zones that can fail together, and zones into regions that can, rarely, fail together too. A design that puts all Collector replicas in one zone survives a crashed process but not a zone outage. Spreading replicas across at least three zones means losing one zone leaves two thirds of your capacity, usually enough to carry on.

flowchart TD
    subgraph R1["Region one"]
        subgraph Z1["Zone A"]
            G1["Gateway replica"]
        end
        subgraph Z2["Zone B"]
            G2["Gateway replica"]
        end
        subgraph Z3["Zone C"]
            G3["Gateway replica"]
        end
    end
    APPS["Applications and agents in the region"] --> G1
    APPS --> G2
    APPS --> G3
    G1 --> BE["Backend"]
    G2 --> BE
    G3 --> BE
        

Regional gateways

For multi-region systems, a common design gives each region its own gateway tier, so telemetry from a region stays inside that region until it reaches the backend. A regional outage then affects only that region’s telemetry, and cross-region network costs and delays are limited to the final hop.

Sending to more than one destination

A Collector can send the same data to several exporters at once. This allows designs such as writing to a primary backend and a secondary store, or to a long-term archive. It also protects against the loss of a single backend, at the cost of doubled network use and a more complicated picture of where the truth lives.

Let us follow one span through an agent plus gateway design and mark every place where something could fail.

sequenceDiagram
    participant A as Application
    participant G as Agent collector
    participant L as Load balancer
    participant W as Gateway replica
    participant B as Backend
    A->>G: Send spans
    G->>G: Memory limiter then batch
    G->>L: Export batch
    L->>W: Route to a healthy replica
    W->>W: Process and queue on disk
    W->>B: Send batch
    B-->>W: Acknowledge
    W-->>G: Acknowledge
        

The best redundancy in the world does not help if every replica receives the same broken instruction at once.

A Collector is steered entirely by its configuration, which describes receivers, processors, exporters and pipelines. A single mistake, such as a wrong destination address, a missing processor or an unsuitable memory setting, can silently drop data or crash every replica. Because all replicas usually share one configuration, this is the classic common-cause failure: redundancy does not protect you because every copy is wrong in the same way.

An insecure Collector is an unreliable Collector, because anyone who can reach it can overload it or feed it garbage.

The Collector accepts network connections, holds credentials for the backend, and sees every piece of telemetry, which may contain sensitive details. Security and availability meet in several places, and ignoring them can turn a small incident into a large outage.

A telemetry pipeline that nobody watches will fail silently, so give it its own small dashboard and its own alerts.

The Collector reports internal metrics about its own behaviour. These are the earliest and most reliable warnings of trouble, because they show queues filling, data being refused and sends failing before any user notices missing telemetry.

Monitor from outside the pipeline

If the Collector reports its own health through itself, a failing pipeline cannot tell you it is failing. Send the Collector’s own metrics through a separate, simpler path, or scrape them with an independent system, so that you are told about the problem even when the main path is down. A small synthetic check that sends a test span regularly and verifies that it reaches the backend gives an end-to-end signal that no internal metric can match.

Patterns capture what usually works and anti-patterns capture what usually hurts, and both are cheaper to learn from a page than from an incident.

Patterns that work

Anti-patterns to avoid

A short checklist is often more useful than a long argument, so here are the habits that separate smooth operations from painful ones.

Examples help the ideas settle, so here are four situations that teams commonly meet.

Every safeguard buys protection and charges a price, so it helps to see them side by side.

The sensible approach is to climb the ladder only as far as your needs require. Many teams reach their goal with replicas, a memory limiter and sending queues alone. Persistent queues come next for critical gateways. Message queues and multi-region fan-out are for the few systems where the cost of losing telemetry is very high.

Let us size a gateway tier with real arithmetic, using illustrative numbers you can replace with your own.

Suppose your system produces thirty thousand spans per second at peak, and you decide that the telemetry pipeline must ride out a ten-minute backend outage without losing data, even while one gateway replica is down. These numbers are illustrative, and the average size of one serialised span is assumed to be about one kilobyte.

Remember that this arithmetic assumes the backend recovers and then drains the queue faster than new data arrives. If recovery is slow, the queue keeps growing, so the sizing must also consider how quickly the backend can catch up. Treat the result as a starting point to be checked with a real test, not as a promise.

A resilience feature you have never tested is a hope, not a design.

Failures are best met for the first time on a quiet afternoon, with the team watching, not at night during a real incident. A game day is a planned exercise in which you deliberately cause a failure and confirm that the system behaves as designed. Start in a test environment, then repeat carefully in production once you trust the results.

When something looks wrong, a symptom-first table gets you to the right question quickly.

A calm way to debug

The same Collector carries three kinds of data, and each kind has its own tolerance for loss, delay and duplication.

It is tempting to design one resilience policy for everything, but the three signal types behave differently when something goes wrong. Understanding those differences lets you spend your protection budget where it matters most and relax it where it does not.

You cannot design for a goal you have never written down, so give your telemetry pipeline its own simple targets.

Engineering teams routinely set targets for the services they run, such as how fast a page should load. The telemetry pipeline deserves the same treatment, because everyone depends on it during incidents. A small set of targets turns vague wishes like “very reliable” into numbers that drive design and alerting.

Resilience is not free, and a good design makes the price visible so that the team can decide what to buy.

Every layer of protection carries a cost. Some costs are obvious, such as more machines. Others are quieter, such as network charges for crossing zones, disk space for queues, or the time engineers spend maintaining more moving parts. Looking at the bill by category helps you spot where a small design change can save a lot.

Ways to spend less without losing safety

When the pipeline itself is in trouble, a short, rehearsed checklist beats clever improvisation.

A runbook is a short document that tells an on-call engineer what to do first, second and third. Writing one for the Collector tier before it is needed turns a stressful, confusing night into a series of small, calm steps. The best runbooks are short, specific to your setup, and tested during the failure drills described earlier.

A tutorial should be honest about the changing parts, so here is how to stay safe on moving ground.

The Collector is built from many components, and each one has its own stability level, which the project documents publicly. Some are considered stable, while others are still maturing, and components can change names, options or defaults between releases. The core ideas in this tutorial, replicas, memory protection, queues, retries, routing and monitoring, are lasting. The exact option names and default values are not, so always verify them against the documentation for the version you run.

It is the practice of running redundant, well-buffered, well-monitored Collector tiers so that crashes, restarts, traffic spikes and backend outages do not turn into lost or unreachable telemetry.

The scaling guidance recommends at least three for resilience. Size the group so the remaining replicas can carry peak load with one missing.

Small systems can work well with agents only. A gateway becomes valuable when you need central configuration, shared credentials, heavier processing, or a resilience boundary between applications and the backend.

It refuses new data when memory is nearly exhausted, instead of letting the process be killed. Senders are told to retry, and data already accepted is protected.

If losing queued data on a crash or restart would be costly, yes, especially at gateways. For small agents holding little data, in-memory queues are often enough. Make sure the disk itself survives the failures you care about.

When outages of the next tier may exceed what local queues can cover, when bursts are extreme, or when several consumers need the same stream. For many systems it is more complexity than needed.

Because each replica must see all spans of a trace to decide about it. A normal balancer spreads spans randomly, so each replica sees fragments. Route by trace identifier instead.

Most likely several replicas scrape the same targets. Divide the targets among replicas with a target allocator, or run singleton sources as a single replica or with leader election.

Not absolutely. You can promise no loss for defined failures, such as one replica dying or a backend outage shorter than your buffer window, and bounded, alerted loss beyond that.

When the bottleneck is downstream, for example a slow backend or a saturated network. Rising failed sends and queues that stay full despite more capacity are the clues.

Watch queue size against capacity, items refused, failed sends, drops, restarts, disk usage and ready replica counts, and add an end-to-end synthetic check that travels through the real path.

Two causes dominate in practice: running out of memory during a spike, and a faulty change rolled out to every replica at once. A memory limiter and staged rollouts address both.

We began with a city that depended on a single post office and saw how the same fragility can sit inside any observability setup that depends on one Collector. High availability design replaces that fragility with a few repeatable habits. Run several replicas across failure domains behind a balancer that spreads real requests. Protect each Collector’s memory so spikes cause polite refusals, not crashes. Put queues at every risky boundary, move them to disk where crashes matter, and add a message queue only where the outage you must survive is long. Route stateful work, such as tail sampling, by trace or service key, and shard pulling receivers so nothing is scraped twice. Spread copies across zones, roll changes out gradually, secure every hop, and watch the Collector with monitoring that does not depend on the Collector. Finally, test each promise with deliberate failure drills, because the only reliable design is the one you have already seen survive.