Before we understand the solution, we need to feel the pain it removes.

What is telemetry?

Telemetry is data that software sends out to describe what it is doing. The word comes from two Greek roots meaning “far” and “measure”. Think of the dashboard in a car. You cannot see the engine temperature or the fuel tank by looking at them directly, so the car measures them and shows the numbers to you. Software works the same way. A running program is invisible, so we ask it to report on itself.

Modern telemetry is usually split into three kinds of signals. Each signal answers a different question about your system.

The messy world before a central collector

Years ago, every monitoring product shipped its own special library and its own special agent. If you wanted Product A to show your traces, you installed Product A’s library inside every application. If you later wanted to try Product B, you had to change the code in every application again. Different teams used different tools, so a single company could end up with five or six agents running on every server, each fighting for memory and processor time.

This created three big headaches. First, vendor lock-in: changing tools meant rewriting code. Second, inconsistency: each tool named things differently, so comparing data across teams was painful. Third, operational weight: every extra agent was one more thing to install, upgrade and secure.

The Collector is one member of a larger family, so let us meet the family first.

OpenTelemetry, often shortened to OTel, is an open-source project that defines a common way to create and ship telemetry. It lives under the Cloud Native Computing Foundation, the same neutral home that hosts Kubernetes. It was formed when two earlier projects, OpenTracing and OpenCensus, merged so the industry could agree on one standard instead of two competing ones.

Think of OpenTelemetry as a shared language. Before it, every tool spoke its own dialect. After it, applications can describe their behaviour in one common vocabulary, and any tool that understands that vocabulary can read it.

The main pieces of the project

Notice something important. The SDK lives inside your application and creates the data. The Collector lives outside your application as a separate program and handles the data after it leaves. The two work together, but you can use the SDK without a Collector and, in some cases, a Collector without the SDK.

A plain-English definition, then the reasons behind each word.

The OpenTelemetry Collector is a vendor-neutral program that receives telemetry data, processes it, and exports it to one or more destinations. It is written in the Go programming language and ships as a single small executable. You run it as a separate process next to your applications, or as a shared service in your network.

Let us unpack the important words in that definition one by one.

A fair question: if applications can send data directly, why add another moving part?

It is true that many applications can ship data straight to a monitoring tool without any Collector. For a tiny project with one service, that is often fine and even recommended, because it is the simplest setup. But as systems grow, the direct approach starts to hurt. The Collector earns its place by solving several real problems.

Reason 1: Applications should stay light

When an application handles sending, retrying, batching and compressing telemetry itself, it spends its own memory and processor time on that work. Handing the job to a Collector means the application can send data quickly to something nearby and get back to serving customers. Think of a waiter who drops order slips in a box at the kitchen door instead of walking to every cook personally.

Reason 2: Change destinations without touching code

If every application knows the address and password of your monitoring tool, switching tools means changing and redeploying every application. With a Collector, applications only know the Collector’s address. When you change tools, you change one configuration in one place.

Reason 3: Central control over the data

A shared Collector is a natural checkpoint. Here you can remove customer emails, drop noisy health-check traces, attach the name of the team that owns each service, and enforce rules for the whole company in one place. Without it, you must trust every team to do these things correctly inside their own code.

Reason 4: Protection against hiccups

Monitoring tools sometimes go down or slow down. If applications send directly, they must decide what to do when the destination is unreachable. A Collector can hold data in a queue, retry politely, and shield your applications from outside trouble.

Reason 5: Handling many formats

Real companies have a mix of old and new systems. Some speak OpenTelemetry, some speak older tracing formats, and some expose metrics in other popular styles. A Collector can accept all of them and translate everything into the format your chosen tool expects.

Everything inside the Collector is built from a small set of Lego-like parts.

The Collector is made of components. You pick the components you need, connect them, and describe the whole arrangement in a single configuration file. There are five kinds of components. Three of them appear in almost every setup, and two are extras that add special abilities.

Receivers in more detail

Receivers work in two styles. In the push style, an application sends data to the Collector, and the receiver simply waits and listens. In the pull style, the receiver actively visits a target on a schedule and scrapes its numbers, much like a meter reader walking from house to house. The most common receiver understands the OpenTelemetry Protocol and listens on two well-known ports, one for a fast binary style and one for a web-friendly style. Other receivers understand older tracing formats, scrape metrics from servers, read log files from disk, or read messages from a queue.

Processors in more detail

Processors are optional but powerful. Because they run in the order you list them, order matters a great deal. Some processors protect the Collector itself, some reshape data, and some decide what to keep. We will explore the most useful ones in a later chapter.

Exporters in more detail

An exporter knows how to talk to one kind of destination. It handles the details of the network connection, compression, security and retrying. You can attach several exporters to one pipeline, and each receives its own copy of the data. This is how you send the same traces to two different tools at once, which is handy during a migration.

Connectors in more detail

Sometimes you want to turn one kind of signal into another. A popular example is counting spans from your traces and producing request-rate and error-rate metrics out of them. A connector does exactly that. It looks like an exporter from the point of view of the first pipeline and like a receiver from the point of view of the second.

Extensions in more detail

Extensions do not touch your telemetry directly. They add supporting features around the Collector. A health-check extension gives load balancers a simple page to ask “are you alive?”. Authentication extensions help receivers and exporters prove who they are. Diagnostic extensions expose internal pages that help engineers debug a misbehaving Collector.

A pipeline is the road that data travels on, from the front door to the back door.

Defining components is not enough. A component does nothing until you place it inside a pipeline. A pipeline is a named route for exactly one signal type. You can have a traces pipeline, a metrics pipeline and a logs pipeline, and you can even have several pipelines for the same signal if you want different handling for different data.

Each pipeline has three parts in a strict order: one or more receivers on the left, a chain of processors in the middle, and one or more exporters on the right. The Collector configuration file has a section where you list your pipelines, and only components named there are actually switched on. A component that is described but never used in a pipeline simply sits idle.

flowchart LR
  A["Apps and Services"] --> R1["Receiver: OTLP"]
  B["Servers and Hosts"] --> R2["Receiver: Host Metrics"]
  C["Log Files"] --> R3["Receiver: File Logs"]
  R1 --> P1["Processor: Memory Limiter"]
  R2 --> P1
  R3 --> P1
  P1 --> P2["Processor: Add Attributes"]
  P2 --> P3["Processor: Batch"]
  P3 --> E1["Exporter: Tracing Tool"]
  P3 --> E2["Exporter: Metrics Store"]
  P3 --> E3["Exporter: Log Platform"]
        

The diagram shows three different sources feeding the same chain. In real life, each signal usually has its own pipeline, but the picture captures the key idea: data enters on the left, gets cleaned and prepared in the middle, and fans out on the right.

Fan-in and fan-out

When two receivers feed one pipeline, we call it fan-in. When one pipeline feeds two exporters, we call it fan-out. In fan-out, every exporter gets its own copy of the data. This is useful, but it also means the Collector does extra work and uses extra bandwidth, so use it deliberately.

Let us follow one piece of telemetry from birth to destination.

To make the journey concrete, imagine an online shop. A customer clicks “Buy”, and the shop’s checkout service records a trace describing that click. Here is what happens next.

sequenceDiagram
  participant App as Checkout Service
  participant Rcv as Receiver
  participant Prc as Processors
  participant Exp as Exporter
  participant Bck as Monitoring Backend
  App->>Rcv: Send trace data
  Rcv->>Prc: Pass internal data
  Prc->>Prc: Protect, enrich, filter, batch
  Prc->>Exp: Hand over finished batch
  Exp->>Bck: Deliver data
  Bck-->>Exp: Acknowledge receipt
        

Why a common internal format matters

Because every receiver converts incoming data into one internal shape, you only need one set of processors for all your sources. A filter that removes health-check traces works whether the trace arrived in the OpenTelemetry format or an older format. This is the quiet superpower behind the Collector: it is a universal translator in the middle.

The same program can play different roles depending on where you place it.

One of the nicest things about the Collector is its flexibility. It is the same executable in every case. What changes is the location and the job you give it. There are three popular patterns.

Pattern A: No Collector (direct export)

Applications send straight to the backend. This is the simplest option and is perfectly reasonable for learning, small projects and quick experiments. Its weakness is that all the concerns we discussed earlier fall back onto the applications.

Pattern B: Agent mode

In this pattern, a Collector runs very close to each application, for example on the same server or as a helper container beside each application instance. Applications send to it over the local network, which is fast and reliable. The agent can add information about the host, do light processing, and forward data onward. It is like having a small mailbox at the end of every street.

Pattern C: Gateway mode

Here a pool of Collectors runs as a central service. All agents or applications send to a load-balanced address, and the gateway does the heavy work: strong filtering, sampling decisions, credential management and final delivery to backends. It is like the main sorting centre that all mailboxes eventually feed into.

The hybrid approach

Large organizations often combine B and C. Agents handle local concerns, and gateways handle company-wide concerns. This gives you the best of both worlds, at the cost of operating two layers.

flowchart TD
  subgraph NodeA["Server A"]
    A1["App 1"] --> AG1["Agent Collector"]
    A2["App 2"] --> AG1
  end
  subgraph NodeB["Server B"]
    B1["App 3"] --> AG2["Agent Collector"]
  end
  AG1 --> LB["Load Balancer"]
  AG2 --> LB
  LB --> GW1["Gateway Collector 1"]
  LB --> GW2["Gateway Collector 2"]
  GW1 --> BE1["Tracing Backend"]
  GW2 --> BE1
  GW1 --> BE2["Metrics Backend"]
  GW2 --> BE2
        

Running on Kubernetes

In Kubernetes, the agent pattern is often implemented as a DaemonSet, which means one Collector pod on every node. Another option is a sidecar, which means one Collector container inside each application pod. The gateway pattern is typically a Deployment with several replicas behind a service. There is also an official operator that can manage Collectors for you as Kubernetes resources, which reduces manual work.

Not every Collector you download contains the same set of components.

The Collector project has hundreds of components, written by the core team and by the wider community. Packing all of them into one program would make it huge and would increase the security surface. So the project publishes several ready-made versions, called distributions, and also lets you build your own.

Which one should you choose?

For learning and experiments, the contrib build is the friendliest because almost everything you read about will be available. For production, many teams move to a custom build. A smaller program has fewer parts that could contain bugs, fewer things to patch, and a smaller footprint. It also makes it easier to explain to a security team exactly what is running.

Processors are where the Collector turns raw data into useful data.

There are many processors, but a handful appear in nearly every real setup. Learn these and you will understand most configurations you meet.

Order matters

Processors run in the order you list them, and the order changes the result. A good general pattern is to put the memory limiter first, so it can protect the Collector before any work begins. Then place the processors that add context and clean data. Place the batch processor last, so batches are formed from the final, cleaned items. If you put batch first and filter later, you waste effort packaging data you then throw away.

flowchart LR
  IN["Incoming Data"] --> M["1. Memory Limiter"]
  M --> D["2. Resource Detection"]
  D --> K["3. Kubernetes Attributes"]
  K --> F["4. Filter and Redact"]
  F --> S["5. Sampling"]
  S --> B["6. Batch"]
  B --> OUT["Exporters"]
        

Understanding sampling in simple words

Tracing every single request in a busy system can produce an enormous amount of data and an equally enormous bill. Sampling means keeping only some of it. There are two main styles.

That last requirement surprises many beginners. If you run several gateway Collectors and the pieces of one trace are spread across them, no single Collector sees the full picture. The usual fix is a two-layer design. The first layer uses a load-balancing exporter that routes by trace identity so that every piece of the same trace lands on the same second-layer Collector, which then makes the sampling decision.

Telemetry often contains more private information than people expect.

Traces and logs can accidentally include user names, email addresses, session identifiers, internal server names and even pieces of database queries. Because the Collector sits in the middle of all this data, it is both a risk and a chance to improve safety. Treat it like any other sensitive service in your network.

Protect the connections

Data travelling between applications, Collectors and backends should be encrypted using standard transport security. Receivers can require secure connections, and exporters can present certificates to prove their identity. Mutual authentication, where both sides prove who they are, is a stronger option for sensitive environments.

Control who can talk to the Collector

An open receiver that accepts data from anyone is an invitation for abuse. Someone could flood it with fake data and drive up your costs or hide real problems. Authentication extensions let you require a token or a username and password before a receiver accepts anything. Network rules, such as firewalls and private networks, add a second layer of defence.

Bind to the right address

By default, careless configurations may listen on every network interface. If a Collector only needs to serve applications on the same machine, it should listen only on the local address. Exposing it more widely than necessary is one of the easiest ways to create an accidental hole.

Remove sensitive data early

Use the redaction, attributes and transform processors to remove or mask private values before data leaves your control. Doing this at the Collector means a single, consistent rule protects every application, including ones whose developers forgot to think about privacy.

Manage secrets carefully

Exporters need credentials to log into backends. Avoid typing these directly into configuration files that live in shared repositories. Instead, let the Collector read them from environment variables or from a proper secrets store provided by your platform.

Keep it updated

The Collector is software like any other. Follow the project’s release notes, upgrade regularly, and prefer custom builds with only the components you need. A smaller build has less code that could contain a weakness.

A watcher that nobody watches is a silent risk.

It is a little funny, but true: the tool that helps you observe everything else must itself be observed. If your Collector quietly stops sending data, your dashboards will look calm, and you may mistake silence for health. Fortunately, the Collector reports on itself.

What the Collector tells you about itself

The Collector produces its own internal metrics and logs. These describe how many items each receiver accepted, how many each processor dropped, how many each exporter sent successfully or failed to send, how full the exporter queues are, and how much memory and processor time the program is using. You can scrape these metrics or even send them through a dedicated pipeline to your monitoring tool.

Health checks and diagnostics

The health-check extension offers a simple web page that orchestration systems can poll to decide whether to restart a Collector or stop sending it traffic. Diagnostic extensions provide pages that show recent activity inside pipelines, which helps you see exactly where data stops flowing during an investigation. A debug exporter can print what the Collector sees to its own output, a very handy trick while building a new configuration.

What happens when traffic grows, or when something breaks?

Back-pressure: the polite “slow down”

When a Collector receives data faster than it can forward it, something has to give. Back-pressure is the idea of pushing the problem back toward the sender instead of letting it pile up forever. The memory limiter is the main tool here. When memory crosses a safe threshold, it asks receivers to refuse new data for a moment. Well-behaved senders will wait and retry rather than flood the Collector further.

Queues and retries

Exporters keep a small queue of items waiting to be sent. When a send fails, they retry using growing wait times, so a struggling backend is not hammered. By default, the queue lives in memory, which is fast but disappears on restart. For stronger safety, you can enable a persistent queue backed by disk storage so that waiting items survive a crash or restart, at the price of a little speed and extra disk space.

Scaling horizontally

Most Collector setups scale by adding more identical copies behind a load balancer. This works well for stateless work such as receiving, enriching, filtering and batching. Anything stateful, such as tail sampling, needs the careful two-layer arrangement we described earlier so related data lands together. Metrics scraped by pull-style receivers also need care: if two Collectors scrape the same target, you get duplicate numbers, so the scraping work must be divided among them.

Scaling vertically

Giving a single Collector more memory and processor power is the simplest fix, and it is often enough for moderate loads. Just remember that one big instance is a single point of failure, so most teams prefer several medium ones.

Habits that keep a Collector setup healthy for years.

Good habits

Useful design patterns

Mistakes to avoid

No tool is perfect, so let us weigh the Collector honestly.

Collector versus vendor agents

Picture how different teams use the same program in different ways.

You do not need to learn everything at once. Here is a sensible order.

No. Applications can send data directly to a backend that understands the OpenTelemetry Protocol. The Collector is recommended as your system grows because it adds flexibility, control and protection, but it is optional for small setups.

No. It does not store data or show charts. It is a pipeline that receives, prepares and forwards data. You still need a backend for storage and a dashboard tool for viewing.

The SDK lives inside your application and creates telemetry. The Collector is a separate program that receives that telemetry and processes and forwards it. One is the producer, the other is the post office.

Traces, metrics and logs are the main three. Support for additional signal types is still emerging, and the maturity of each signal and each component can differ, so check the project’s documentation for the current status of the pieces you plan to use.

No. It is vendor-neutral and runs anywhere you can run a small program: laptops, virtual machines, containers and Kubernetes clusters, in any cloud or on your own hardware.

Usually it does the opposite. Applications hand data to a nearby Collector quickly and move on, instead of doing batching, retries and long-distance delivery themselves. The Collector itself does need processor time and memory, so size it properly.

Senders will retry for a while, but they cannot wait forever, and data held only in memory may be lost. To reduce risk, run several replicas, use persistent queues where data is valuable, and set up alerts so you notice problems early.

Both are valid. One per server (agent) is good for host-level data and fast local hand-off. A shared pool (gateway) is good for central rules and sampling. Many organizations use both together.

Yes. Attach several exporters to a pipeline and each one receives its own copy. This is especially helpful when migrating from one tool to another.

The contrib distribution is the friendliest for learning because it includes the widest range of components. Later, for production, consider a custom build that contains only what you use.

Because each processor receives the output of the one before it. Putting the memory limiter first protects the Collector, and putting batch last packs the final cleaned data. Wrong ordering can waste effort or weaken protections.

Yes. Filtering out noisy data, sampling traces, and removing high-cardinality labels before data reaches a paid backend can reduce volume and cost dramatically, while keeping the information you truly need.

The OpenTelemetry Collector is the central depot of your observability story. Applications drop their traces, metrics and logs at its door and carry on with their real work. Inside, receivers accept data in many formats, processors protect, enrich, filter and pack it, and exporters deliver it to whichever destinations you choose. You can run it beside each application, as a shared gateway, or both, and you can shrink it to a custom build that contains only what you need. Used wisely, it gives you freedom from vendor lock-in, control over privacy and cost, and a calmer life for your application developers. Used carelessly, it becomes one more fragile service, so give it the same care you give any production system: secure it, monitor it, scale it, and keep it updated.