Before we learn the solution, we need to feel the pain it removes.

Modern software is rarely one program running on one computer. A single online shopping order might touch a website, a payment service, a stock service, a shipping service and a database. When something goes wrong, engineers need evidence about what each piece was doing. That evidence is called telemetry, which simply means data that software sends out about its own behavior so that humans can understand it later.

Telemetry is useful only if it lands somewhere people can look at it. The place where it lands is called a backend. A backend might be a tool that draws graphs, a search engine for logs, a long-term storage bucket, or a security system that hunts for suspicious activity. The trouble is that real organizations almost never use just one backend.

Why teams end up with several backends

Different teams have different needs. The platform team may love one tool for graphs, the security team may require another for investigating attacks, and the finance team may only care about a cheap archive that satisfies an audit rule. Companies also merge, and each side arrives with its own tools. Nobody gets to choose a single winner without a long fight.

There is also the matter of change. Contracts end, prices rise, and better tools appear. If every application is hard-wired to talk to one specific vendor, then changing vendors means editing and redeploying hundreds of services. That is slow, risky, and exactly the kind of work engineers dread.

Fan-out is the answer to this mess. The word comes from electronics and software, where it describes one output feeding many inputs. Picture the ribs of a hand fan opening from a single point. In telemetry, the single point is where data is collected, and the ribs are the many backends that receive it.

Fan-out only makes sense once you know the pieces it connects.

OpenTelemetry, often shortened to OTel, is an open standard and a family of tools for producing and moving telemetry. It is vendor neutral, which means it does not belong to any single monitoring company. It was created so that applications could describe their behavior in one common language, and then send that description anywhere.

The three main kinds of signal

OpenTelemetry calls each type of telemetry a signal. Understanding the three main ones is important because fan-out treats each of them separately.

The two big building blocks

OpenTelemetry has two pieces that matter for this post. The first is the SDK, a software development kit, which is a library living inside your application. It creates the telemetry and hands it off. The second is the Collector, a separate program that receives telemetry, processes it, and forwards it. Fan-out is mostly a Collector story, although the SDK can do a limited version of it.

Applications talk to the Collector using a shared protocol called OTLP, the OpenTelemetry Protocol. Because everybody speaks OTLP, an application does not need to know what lies behind the Collector. It sends data to one friendly address and trusts the Collector to take care of the rest.

flowchart LR
  A["Application with OTel SDK"] -->|"OTLP"| C["OpenTelemetry Collector"]
  C --> B1["Backend 1"]
  C --> B2["Backend 2"]
  C --> B3["Backend 3"]
        

Notice how the application only knows about the Collector. That single design choice is what buys you freedom. Adding a fourth backend becomes a change to the Collector, not to every application.

The Collector is built from small, replaceable parts that you snap together.

If you understand five words, you understand the Collector. Think of it as a factory assembly line. Raw material comes in at one end, gets cleaned and shaped in the middle, and leaves in boxes at the other end.

Where fan-out and fan-in naturally appear

A pipeline is allowed to have more than one receiver and more than one exporter. When several receivers feed one pipeline, their data is merged together, which is called fan-in. When one pipeline has several exporters, every item is copied to each of them, which is fan-out. You do not need a special feature to get fan-out. It is simply what happens when you list more than one exporter.

That is why the Collector is such a natural home for this architecture. The concept is built into its shape. What takes skill is not turning fan-out on, but designing it so that one slow or broken backend does not hurt the others.

A precise definition, plus the three promises a good design makes.

A multi-backend fan-out architecture is a design in which telemetry produced once is delivered to two or more independent backends, with the copying and routing handled by a shared layer rather than by each application. The word independent matters. Each backend should be able to slow down, change, fail or be replaced without forcing a change in the producers and without damaging delivery to the others.

The three promises

Fan-out is not the same as load balancing

These two ideas are often confused. In load balancing, each item goes to just one of several identical receivers so that the work is shared. In fan-out, each item goes to every destination, and the destinations are usually different from each other. Fan-out multiplies copies. Load balancing divides work. Real systems often use both, as you will see in the topology chapter.

Notice the row for routing. A mature fan-out design is rarely a blind copy to everything. It is usually a mix: some data goes everywhere, some data goes only to specific places, and some data is trimmed before it leaves. We will build up to that mix step by step.

Understanding the internals explains almost every production surprise.

When a pipeline has several exporters, the Collector places a small internal piece in front of them that is commonly called a fan-out consumer. Its job is to take one batch of data and hand it to each destination in turn. You never configure this piece directly, but knowing it exists helps you predict behavior.

Copies, and when they are made

Data inside the Collector is held in memory. If two consumers were handed the very same memory and one of them changed it, for example by deleting a password field, the other would see the damage. To prevent that, the Collector asks each component whether it modifies data. If a component does modify data, it receives its own private copy. If a component only reads data, the Collector can safely share the original and skip the expensive copy.

This is why adding a processor that edits data, such as one that removes sensitive labels, can increase memory use when it sits in a fan-out position. It is also why read-only components are cheaper. The Collector tries hard to avoid needless copying, but it cannot copy for free.

One after another, not all at once

An important detail is that the fan-out step hands data to its consumers one after another in the same flow of work. It does not wake up a separate worker for each backend. If one consumer takes a long time to accept the data, the next consumer waits. This sounds worrying, but it is exactly why exporters have queues, which the reliability chapter explains. A queue lets the exporter accept data instantly and do the slow network work later in the background.

What happens when one destination says no

If one consumer reports an error, the fan-out step still tries the others and then reports a combined result back upstream. The receiver at the front door may then tell the sending application that the data was not fully accepted, and many senders will retry. The consequence is subtle. A failure in one backend can make the sender resend data that other backends already received, producing duplicates. Good designs prevent this by giving each backend its own queue so that its failures are absorbed locally instead of travelling backwards.

flowchart TD
  R["Receiver"] --> P1["Shared processors"]
  P1 --> F["Fan-out step"]
  F --> E1["Exporter A with queue"]
  F --> E2["Exporter B with queue"]
  F --> E3["Exporter C with queue"]
  E1 --> B1["Backend A"]
  E2 --> B2["Backend B"]
  E3 --> B3["Backend C"]
        

You can copy data inside the application or outside it. The choice has big consequences.

The OpenTelemetry SDK inside an application allows you to register more than one exporter. So technically you can do fan-out right inside the program that creates the data. Many beginners start there because it seems simple: add another destination and you are done. However, this approach pushes the cost and the risk into every application.

Fan-out inside the application

With SDK-level fan-out, each application must know the addresses, passwords and protocols of every backend. Every extra exporter uses processor time, memory and network bandwidth of the application itself. If a backend is slow, the application’s background threads can fill up, and at worst the business program suffers. Changing the list of backends needs a new release of every service.

Fan-out in the Collector

With Collector-level fan-out, the application sends one stream and is finished. The Collector, which is purpose-built for moving data, handles copying, retrying, queuing and secrets. Changing a destination is a configuration change on a handful of Collector instances. The application cost stays flat no matter how many backends you add.

There is a trade-off worth admitting. The Collector adds one more moving part that someone must run, watch and upgrade. For a single small service with a single backend, that effort may not be justified. For anything shared by several teams or several backends, the effort pays for itself quickly.

The simplest form of fan-out, and the right place to start learning.

In this pattern there is one pipeline for a signal, a shared set of processors, and a list of two or more exporters at the end. Every item that survives the processors is copied to every exporter. It is easy to understand, easy to configure, and often exactly what you need during a vendor migration.

Strengths of this pattern

The shared-processor limitation is the main reason teams outgrow this pattern. The moment two backends need different treatment, such as full detail for one and a trimmed version for another, a single pipeline is no longer enough. That leads us to the next pattern.

Give each destination its own assembly line while sharing one front door.

In this pattern, one receiver feeds several pipelines at the same time. Each pipeline has its own processors and its own exporter. Because the pipelines are separate, each backend gets a personalized version of the data. Because they share a receiver, applications still send only once.

flowchart LR
  R["OTLP Receiver"] --> PA["Pipeline A: full detail"]
  R --> PB["Pipeline B: sensitive data removed"]
  R --> PC["Pipeline C: sampled for cost"]
  PA --> EA["Exporter A"] --> BA["Debugging backend"]
  PB --> EB["Exporter B"] --> BB["Security backend"]
  PC --> EC["Exporter C"] --> BC["Long-term archive"]
        

Why this is the workhorse of real deployments

Each pipeline can contain a different mix of processors. The debugging pipeline might keep everything. The security pipeline might hide personal details and keep only certain events. The archive pipeline might keep one item out of every ten to save money. None of these choices affects the others.

Isolation also improves. Each pipeline has its own instance of every processor and its own queue to its exporter. Memory limits and batching therefore apply separately, and a backlog in the archive path does not block the debugging path. This is the closest the Collector gets to true independence between destinations.

Choosing between pattern one and pattern two

Connectors let pipelines talk to each other, which unlocks smarter fan-out.

So far every copy went to every pipeline. Often that is wasteful. A connector is a bridge between pipelines. It is the exporter at the end of one pipeline and the receiver at the start of another, so data can hop from one assembly line to the next without leaving the Collector.

Selective fan-out with routing

Imagine one company running services for two customers, and each customer insists that its telemetry go only to its own backend. A routing connector can read a customer label on every item and steer it to the correct pipeline. Nobody receives data that does not belong to them. This is fan-out with brains: still one entry point, but a decision about where each item goes.

flowchart LR
  R["Receiver"] --> RT{"Routing connector"}
  RT -->|"customer one"| P1["Pipeline for customer one"]
  RT -->|"customer two"| P2["Pipeline for customer two"]
  RT -->|"everything else"| P3["Default pipeline"]
  P1 --> X1["Backend one"]
  P2 --> X2["Backend two"]
  P3 --> X3["Shared backend"]
        

Turning traces into metrics on the way out

Storing every trace for months is expensive, but the high-level numbers derived from them are cheap. With a span-to-metrics connector, the traces pipeline can feed a metrics pipeline. You then send the detailed traces to a short-retention tool and the compact metrics to a long-retention tool. This is a second flavor of fan-out in which one input becomes two different kinds of output.

At scale, fan-out becomes a layered topology instead of one Collector.

One Collector is fine for a demo. A company with thousands of application instances needs layers. The usual layout has two tiers. Small agent Collectors run next to the applications, for instance one per server or one per node in a cluster. A smaller number of larger gateway Collectors sit in the middle and perform the heavy work, including the final fan-out to backends.

flowchart LR
  A1["App group 1"] --> AG1["Agent Collector"]
  A2["App group 2"] --> AG2["Agent Collector"]
  A3["App group 3"] --> AG3["Agent Collector"]
  AG1 --> GW["Gateway Collector pool"]
  AG2 --> GW
  AG3 --> GW
  GW --> B1["Observability backend"]
  GW --> B2["Security backend"]
  GW --> B3["Archive storage"]
        

Why two tiers help

Keeping a whole trace together

Some processing, notably deciding which complete traces are worth keeping, requires that all pieces of one trace arrive at the same Collector. If the pieces are scattered across many gateway instances, the decision will be wrong. A load-balancing exporter solves this by choosing the destination from the trace identifier, so every span of one trace reaches the same gateway. Here load balancing and fan-out cooperate: first the data is spread by trace across a set of Collectors, and then each Collector fans out to the backends.

Follow a single span from birth to its final resting places.

It helps to walk through the full journey of one item. We will use a span, which is one step in a trace, from a checkout service in an online store. The same story applies, with small differences, to metrics and logs.

sequenceDiagram
  participant App as Application
  participant Agent as Agent Collector
  participant GW as Gateway Collector
  participant BA as Backend A
  participant BB as Backend B
  App->>Agent: Send batch of spans
  Agent->>Agent: Add host and region labels
  Agent->>GW: Forward batch
  GW->>GW: Check memory then copy to pipelines
  GW->>BA: Deliver full detail version
  GW->>BB: Deliver trimmed version
  BA-->>GW: Accepted
  BB-->>GW: Accepted
        

Every one of those steps is a place where something can be slow, full or broken. A good design assumes that each step will fail some day and decides ahead of time what should happen. The next chapter explains the tools you have for that.

The difference between a demo and a production system is what happens on a bad day.

Backends are not perfect. They get upgraded, overloaded, misconfigured and occasionally disappear. A fan-out design multiplies the number of things that can go wrong, because every extra backend is another possible failure. The goal is simple to say and hard to do: one backend’s bad day must not become everyone’s bad day.

The sending queue

A sending queue is a waiting line inside an exporter. Data enters the line immediately and a set of background workers take items off the line and deliver them. This is the single most important isolation tool. With a queue, a slow backend only makes its own line longer. Without one, the slow backend holds up the whole hand-off, and every other destination waits behind it.

Retries

When a delivery fails, the exporter can try again after a pause that gets longer each time. This is called retry with backoff. It rescues data from short outages such as a backend restarting. It must have a time limit, because retrying forever fills the queue with old data and hides a real problem.

Memory protection

A memory-limiting processor watches how much memory the Collector uses. When usage gets high, it starts refusing new data so that the Collector never runs out of memory and crashes. Refusing data is unpleasant, but a controlled slowdown is far better than a crash that loses everything in flight. It should be placed first in each pipeline so that it protects everything after it.

Batching

A batching processor groups many small items into larger packages. Larger packages mean fewer network calls, better compression and lower cost at the backend. It is usually placed near the end of the processor list, after filtering has already reduced the volume.

Disk-backed queues

By default, queues live in memory, so a Collector restart loses whatever was waiting. A persistent queue stores waiting data on disk using a storage extension. After a restart the data is still there. The price is slower writes and the need to manage disk space, so teams usually enable this only for the backends where losing data is unacceptable.

Backpressure, explained simply

Backpressure is the way a full system tells its upstream neighbors to slow down. When queues fill and memory protection triggers, the Collector starts rejecting new data, and senders respond by waiting or buffering. In a fan-out design you must decide which destinations are allowed to cause backpressure. Usually only the critical ones should. A low-value archive should drop data rather than slow down the primary observability path.

Sending data to many places means many places where it can leak.

Telemetry often contains more than engineers realize. A log line may hold an email address. A trace label may carry a customer number. Once you copy that data to several backends owned by several teams or companies, you have also copied the risk. Fan-out makes security design a first-class concern rather than an afterthought.

Remove what a destination does not need

The principle of least data says each backend should receive only what it needs to do its job. A graphing tool rarely needs customer emails. A security tool might need them but should not share them with outsiders. Separate pipelines allow you to strip, hash or mask sensitive fields on the path to each destination individually.

Protect the connections and the keys

Every link in the chain should be encrypted, from application to agent, agent to gateway, and gateway to each backend. Credentials for backends should live only in the gateway layer and be loaded from a secrets manager, never pasted into files that many people can read. Authentication on the receiving side prevents strangers from pushing fake data into your Collector.

Data residency and legal boundaries

Some laws require that data about people in a region stay in that region. A fan-out design can respect this by using routing to keep each region’s telemetry inside regional backends, while only anonymous summary numbers travel to a global dashboard. This is much easier to enforce in one central routing layer than inside hundreds of applications.

Fan-out multiplies copies, and copies cost money, memory and bandwidth.

The most common surprise after enabling fan-out is the bill. Backends usually charge by the amount of data they ingest or store. If you send the same stream to three paid backends, you pay roughly three times, even though the application produced it once. Smart fan-out is as much about deciding what not to send as it is about what to send.

Where the multiplication hits

Ways to keep costs under control

Sizing the Collector

Collector capacity depends on data volume, the number of pipelines, how much processing each does, and how many exporters send. Start by measuring real traffic rather than guessing. Run a representative load, watch processor time and memory, and add a safety margin. Because the gateway tier is stateless in most designs, you can scale it by adding instances, which is easier than rebuilding a single giant server.

The system that observes everything must itself be observed.

It is an uncomfortable truth that the Collector can fail silently. If it drops data, your dashboards simply look quieter, and people may mistake silence for health. The Collector therefore publishes its own telemetry about how it is doing, and a fan-out setup must pay close attention to it, because every destination has its own queue and its own failure story.

The questions to keep asking

Send the Collector’s own signals somewhere independent

If the Collector reports its health only through itself, then a failure of the Collector also hides the failure report. Many teams therefore send the Collector’s own metrics to a separate, simple destination, so that the warning system does not depend on the thing it is warning about.

Compare backends to each other

One of the delightful side effects of fan-out is that you have natural reference points. If backend A shows ten thousand requests per minute and backend B shows nine thousand, someone has dropped data. Periodic comparison between destinations is a cheap way to detect silent loss, as long as each backend is meant to receive the same stream.

Six situations where fan-out solves a genuine business problem.

Notice the shape of these stories. In every one, the business wanted more than one audience for the same facts, and they wanted to change audiences without disturbing the source. That is the heart of the architecture.

Every architecture is a bargain. Here is what you gain and what you pay.

The trade-offs in plain terms

Fan-out trades simplicity at the edge for responsibility at the center. You remove complexity from hundreds of applications and concentrate it in a small, expert-managed layer. This is usually a good trade, because it is much easier to keep a few Collectors healthy than to keep every application correctly configured. But it only works if that central layer is treated as a real product with owners, tests, version control and on-call support.

That row about blast radius deserves emphasis. Centralization makes changes easy, and it makes mistakes easy too. A typo in a gateway setting can break delivery to every backend simultaneously. This is why mature teams test configuration changes in a staging environment, roll them out gradually to a few Collectors first, and keep a quick way to roll back.

Habits worth copying, and habits worth running away from.

Patterns worth adopting

Anti-patterns to avoid

A practical checklist you can use before going live.

Best practices

Common mistakes

Match the architecture to the size and needs of your organization.

There is no single correct fan-out design. A startup with ten services and a company with ten thousand have different needs. The guide below helps you choose a sensible starting point and shows when to evolve.

flowchart TD
  S["Start: do you need more than one backend?"] -->|"No"| N["Single exporter, keep it simple"]
  S -->|"Yes"| Q1{"Do all backends need identical data?"}
  Q1 -->|"Yes"| P1["One pipeline with several exporters"]
  Q1 -->|"No"| Q2{"Is data owned by team, customer or region?"}
  Q2 -->|"Yes"| P3["Routing connector to separate pipelines"]
  Q2 -->|"No"| P2["Separate pipelines with tailored processors"]
  P1 --> Q3{"Large scale or many applications?"}
  P2 --> Q3
  P3 --> Q3
  Q3 -->|"Yes"| T["Add agent and gateway tiers"]
  Q3 -->|"No"| D["Single Collector tier is enough"]
        

Evolve in small steps. Every additional layer adds capability and operational duty. Add a layer only when you can name the problem it solves, and remove complexity when the problem it solved has gone away.

Short, direct answers to the questions people ask most.

Not when the copying happens in the Collector. The application sends one stream to a nearby Collector and returns to its work. The cost of extra backends is paid by the Collector, not by the business program.

Only if you use one shared pipeline with several exporters. With separate pipelines, each backend can receive a different version, with different filtering, sampling or redaction.

The exporter for that backend retries for a limited time and holds data in its queue. If the outage lasts longer than the queue and retry limits allow, data for that backend is dropped. Healthy backends should continue to receive data as long as each has its own queue.

Yes. Pipelines are defined per signal, so every signal has its own list of exporters. It is very common to mix different tools for different signals.

No. Fan-out sends copies to different destinations. High availability keeps a service running when parts fail. They can support each other, for example by running several gateway instances, but they solve different problems.

It can. If a sender resends data after a partial failure, a backend that had already accepted the data may receive it twice. Per-destination queues and careful retry limits reduce the chance, and many backends can tolerate or remove duplicates.

A single Collector tier is enough for small and medium systems. Add the two-tier layout when you have many application instances, need central control of policy and secrets, or require trace-aware routing at scale.

The fan-out concept is identical, but practical concerns differ. Metrics are small but sensitive to label explosion. Logs are large and often contain personal information. Traces need special care for sampling. Treat each signal on its own terms.

Watch the Collector’s own counters for refused and failed items, watch queue fullness, and compare ingestion counts between backends that are meant to receive the same stream. A regular comparison catches silent loss early.

It can be, provided you remove or mask sensitive fields on the path to that vendor, encrypt the connection, restrict credentials, and confirm that the vendor’s data handling meets your legal obligations.

Some systems do work that way, especially for metrics. But many telemetry flows are push based, and pulling from every application by every backend would multiply connections and complicate security. A central push layer gives control that a crowd of pullers cannot.

Remove it when nobody uses it, when it duplicates another tool without extra value, or when its cost exceeds its benefit. Every destination should have a named owner and a stated purpose, and destinations without both are candidates for removal.

A multi-backend fan-out architecture lets you produce telemetry once and deliver it to many destinations without burdening your applications. The OpenTelemetry Collector provides the natural home for this idea, because pipelines can have several exporters, several pipelines can share a receiver, and connectors can route or transform data between pipelines. The design challenge is not turning fan-out on. It is keeping destinations independent, controlling cost, protecting sensitive data, and watching the system that watches everything else.

Start with the simplest shape that works, move to separate pipelines when different destinations need different treatment, and add routing and tiers only as scale and policy demand. Give every backend its own queue, a clear failure rule, an owner and a reason to exist. If you do those things, adding the next backend, or retiring an old one, will feel almost boring, and that is the best compliment a piece of infrastructure can earn.