What Is Collector Agent Mode in OpenTelemetry?
A beginner-friendly guide to running a small OpenTelemetry Collector right next to your applications: what the agent does, where it lives, how it differs from a gateway, and how to run it safely and cheaply.
Imagine a large city where thousands of households produce parcels to be posted. If every household had to drive to the central sorting office, the roads would jam and many people would simply give up. So the city places a small post box at the end of every street. Residents drop their parcels in the box within seconds, and a postal worker collects from each box, adds the correct stamps, and carries everything onward to the central sorting office. The residents stay close to home, and the long journey is handled by someone else. In OpenTelemetry, that little post box at the end of the street is called the Collector in agent mode.
This guide starts from zero. You do not need to know what the Collector is, what a deployment mode means, or why anyone would run a small program beside every application. We will build each idea step by step, using everyday comparisons and simple pictures, until you can explain agent mode to a colleague with confidence.
1The Problem: Sending Telemetry From Everywhere to Everywhere
Before we meet the agent, let us feel the problem it removes.
Applications produce telemetry, and it must travel
Modern software describes its own behaviour through telemetry: traces that follow requests, metrics that measure health, and logs that record events. All of this data has to travel from the application to a monitoring tool somewhere else, often in another network or even another country.
The trouble with long journeys
If every application sends its telemetry straight to the monitoring tool, several headaches appear. Each application must know the address and password of the tool. Each must handle slow networks, retries and failures by itself. Each spends memory and processor time on work that is not its real job. And if the company changes tools, every application must be changed and redeployed.
There is also a quieter problem. Some telemetry is not produced by your application code at all. The server itself has processor and memory numbers. The files on disk contain logs from programs that know nothing about OpenTelemetry. Somebody must gather these as well, and they live on the machine, not inside the application.
Picture a factory where every worker, instead of putting finished goods on a conveyor belt beside their desk, must personally drive each product to the customer. Deliveries would be slow, workers would be exhausted, and any change of customer address would require retraining everybody. A local conveyor belt that carries goods away is the obvious fix. The agent is that conveyor belt.
Many beginners think the Collector is only a big central service. It can be, but it can also be a tiny helper that runs beside every application or on every server. That second role is agent mode.
2A Quick Refresher: What the Collector Is
Agent mode only makes sense once you remember what the Collector does.
The OpenTelemetry Collector is a vendor-neutral program that receives telemetry, processes it, and exports it to one or more destinations. It is written in the Go programming language and ships as a single executable. You describe what it should do in a configuration file that lists its components and connects them into pipelines.
Receivers
The front door. They listen for incoming data, or go out and fetch it, then hand it to the pipeline.
Processors
The workshop. They inspect and change data on the way through, one step after another, in a fixed order.
Exporters
The back door. They send finished data to a destination such as a monitoring tool or another Collector.
Extensions
Helpers beside the data path, such as health checks and authentication.
An important point for this guide: there is only one Collector program. There is no special “agent program” and no special “gateway program”. The words agent and gateway describe where you place it and what job you give it, not a different product.
Think of a van. The same van can be used as a local delivery van that visits every street, or as a long-distance truck between warehouses, depending on the route it is given. The vehicle is the same, only the role changes. The Collector is that van, and agent mode is the local delivery route.
3So, What Is Collector Agent Mode?
A plain definition first, then the reasons behind each word.
Agent mode is a way of deploying the OpenTelemetry Collector in which a Collector instance runs very close to the applications that produce telemetry, such as on the same server, in the same pod, or on the same node. Applications send their data to this nearby Collector, which does light processing and then forwards the data onward to a backend or to another Collector.
That definition has three important ideas inside it.
Close to the source
The agent lives beside the workload. The network hop between application and agent is short, fast and reliable, often just a connection to the same machine.
Light local work
The agent does the jobs that are best done locally: accepting data quickly, adding facts about the host, collecting machine-level numbers, reading local files, batching and forwarding.
Forwards onward
The agent is usually not the final stop. It passes data on to a gateway Collector or directly to a monitoring tool, which handles the heavier central tasks.
4The Three Deployment Patterns at a Glance
To understand the agent, it helps to see its neighbours.
There are three popular ways to arrange the path from application to monitoring tool. Agent mode is the middle one, and it is often combined with the third.
No Collector
Applications send straight to the backend. Simplest to start, but every concern falls back onto the applications.
Agent Mode
A Collector runs beside the applications and forwards onward. This is our topic.
Gateway Mode
A shared pool of Collectors runs as a central service that receives data from many agents or applications.
flowchart LR
subgraph NodeA["Server A"]
A1["App 1"] --> AG1["Agent Collector"]
A2["App 2"] --> AG1
end
subgraph NodeB["Server B"]
B1["App 3"] --> AG2["Agent Collector"]
end
AG1 --> GW["Gateway Collectors"]
AG2 --> GW
GW --> BE["Monitoring Backend"]
Think of a courier network. Street post boxes are the agents, collecting parcels close to where people live. The regional sorting office is the gateway, handling the heavy sorting and stamping. The destination is the monitoring tool. Each layer does the job it is best placed for.
5What an Agent Does: Its Main Jobs
An agent has a short list of jobs, and each one is best done close to the source.
Accept Data Quickly
Applications hand their telemetry to a nearby receiver over a fast local connection and immediately return to their real work.
Add Host and Environment Facts
The agent knows things the application may not, such as the machine name, the cloud region, or the Kubernetes pod and namespace, and attaches them to every record.
Collect Machine-Level Metrics
Processor, memory, disk and network figures belong to the host, not to any one application. Only a program running on that host can read them directly.
Read Local Log Files
Many programs write logs to files. An agent on the same machine can follow those files and turn the lines into log records.
Batch, Compress and Retry
The agent groups data for efficient sending, compresses it, and retries politely if the next hop is busy, shielding applications from network trouble.
Light Cleaning
Simple, stateless tasks such as removing a noisy label or dropping an unwanted record are cheap to do locally.
What an agent usually does not do
Some jobs are poor fits for an agent. Anything that needs to see data from many machines at once, such as deciding which complete traces to keep, belongs in a central place. Anything that should happen exactly once for the whole cluster, such as gathering cluster-wide numbers, should not be repeated by every agent. We will return to these points later.
Think of a hotel porter at each floor. The porter takes your bag quickly, writes the room number on the tag, and sends it to the lobby. The porter does not decide which guests get upgrades, because that decision needs information from the whole hotel. Local, quick, simple jobs go to the porter. Whole-hotel decisions go to the front desk.
6Why Run an Agent? The Benefits in Plain Words
An extra program has to earn its place, so let us look at what it gives you.
Reason 1: Applications stay light
When an application only has to hand data to a process on the same machine, it spends very little effort on telemetry. The heavy work of batching, compressing, securing and retrying happens in the agent, not inside the application’s own memory and threads.
Reason 2: A short and reliable first hop
The distance between application and agent is tiny. Network failures on that hop are rare, and delivery is fast. If the long-distance network is slow or the backend is down, the agent absorbs the trouble while the application carries on.
Reason 3: Change destinations without touching applications
Applications know only the local address of the agent. When you change monitoring tools, add a second destination, or alter credentials, you change the agent configuration. No application needs to be rebuilt or redeployed.
Reason 4: Host-level data collection
Machine metrics and local log files can only be gathered by something running on that machine. An agent is the natural collector for them, and it can send them down the same pipeline as application data.
Reason 5: Automatic context
An agent can attach host, container, cloud and Kubernetes details to every piece of telemetry automatically. Developers do not have to remember to add those details in their code, and the labels are consistent across all services.
Reason 6: Smaller blast radius
Each agent serves only the applications on its own host. If one agent misbehaves, only that host is affected, not the whole company’s telemetry.
With a Local Agent
- Applications offload batching, retries and security
- Fast, reliable first hop on the same machine
- Host metrics and local logs can be collected
- Host and environment labels added automatically
- Backend changes require no application changes
Without an Agent
- Every application carries the full delivery burden
- Long network hops straight from each application
- Host metrics and files go uncollected
- Developers must add environment labels themselves
- Changing tools means changing every application
7A Typical Agent Pipeline, Step by Step
Let us follow telemetry through a realistic agent.
Imagine a server running two applications. A Collector runs on the same server in agent mode. Here is the journey of a trace, plus the other kinds of data the agent gathers.
The application hands over data locally
The application’s SDK sends spans to the agent over the OpenTelemetry Protocol, usually to an address on the same machine.
A receiver accepts the data
The receiver decodes the message and converts it into the Collector’s internal format.
The memory limiter protects the agent
First in the chain, it watches memory use and refuses new data for a moment if memory gets too high, so the agent does not crash or hurt its neighbours.
Resource detection adds host facts
The agent discovers details such as the host name, cloud provider and region, and attaches them to the data.
Environment labels are added
In Kubernetes, a processor looks up the pod, namespace and deployment for each piece of data and labels it.
The batch processor groups the data
Many small items are packed into larger bundles, which are cheaper to send over the network.
An exporter forwards the batch
The exporter sends the batch to a gateway Collector or to the monitoring backend, using encrypted connections, with a queue and retries if needed.
flowchart LR
APP["Applications on this host"] --> R1["Receiver: OTLP, local only"]
HM["Host and process numbers"] --> R2["Receiver: Host Metrics"]
LF["Local log files"] --> R3["Receiver: File Logs"]
R1 --> M["Memory Limiter"]
R2 --> M
R3 --> M
M --> RD["Resource Detection"]
RD --> KA["Environment Labels"]
KA --> B["Batch"]
B --> E["Exporter: forward with retries"]
E --> NEXT["Gateway or Backend"]
A good general order for processors is: memory limiter first, then context-adding processors, then any simple filtering, and the batch processor last. This protects the agent first and packs only the final, cleaned data.
8Where Does the Agent Live? Placement Options
“Close to the application” can mean several different places.
Depending on how your software runs, an agent can be placed in a few ways. Each option trades isolation against cost and effort.
Service on the Host
On virtual machines or physical servers, the agent runs as a normal background service. One agent serves every application on that machine.
Node Agent in Kubernetes
One agent runs on every node, managed as a DaemonSet. All pods on that node send to the agent on their own node.
Sidecar Container
An agent runs as an extra container inside each application pod. It serves only that pod, giving strong isolation.
Embedded in a Process
In rare cases, the agent is bundled with a specific package or appliance and started together with it.
| Question | Host Service | Node Agent (DaemonSet) | Sidecar |
|---|---|---|---|
| How many agents? | One per machine | One per node | One per pod |
| Resource overhead | Low | Low | Higher, since it repeats for every pod |
| Isolation between applications | Shared | Shared by pods on a node | Strong, each pod has its own |
| Host metrics and node logs | Excellent | Excellent | Poor, it sees only the pod |
| Per-application settings | Harder | Harder | Easy |
| Operational effort | Moderate | Low once set up | Highest, many copies to manage |
A host service or node agent is like one post box per street, shared by all houses. A sidecar is like giving every house its own private post box. Private boxes give more separation, but the street fills up with boxes and each one needs emptying. Most streets choose a shared box unless a house has a special reason for its own.
Running on Kubernetes
In Kubernetes, the node agent is the most common choice. Because every pod on a node can reach the node’s agent, applications send data to a stable local address. The agent can read the node’s log files, gather node and pod resource numbers, and ask the Kubernetes system which pod produced which data. There is also an official operator that can create and manage Collectors for you as Kubernetes resources, which reduces manual work and helps keep configuration consistent.
Choosing sidecars for every application “for isolation” without counting the cost. In a large cluster, thousands of sidecars can use more memory in total than the applications themselves. Start with node agents, and use sidecars only where isolation truly matters.
9Agent Versus Gateway: Choosing the Right Job for the Right Place
Agents and gateways are partners, not rivals.
In a gateway, a pool of Collectors runs as a shared central service. Agents and applications send data to a load-balanced address, and the gateway performs the heavier tasks that need a wider view or central control. Understanding which job belongs where is the most important design skill for agent mode.
| Job | Best Done in the Agent | Best Done in the Gateway |
|---|---|---|
| Accepting data from applications | Yes, fast and local | Possible, but adds distance |
| Collecting host metrics | Yes, only possible locally | No |
| Reading local log files | Yes | No |
| Adding host and pod labels | Yes, it knows its own surroundings | Possible, with lookups |
| Simple dropping of noisy data | Yes, saves network cost early | Yes |
| Company-wide privacy rules | Possible, but hard to keep consistent | Yes, one central place |
| Keeping or dropping whole traces by outcome | No, it sees only part of the picture | Yes, with the right design |
| Managing backend credentials | Avoid, spreads secrets widely | Yes, fewer places to protect |
| Cluster-wide metrics collected once | No, would be repeated by every agent | Yes, from a single instance |
Why tail sampling does not belong in the agent
To keep every slow or failed trace and discard most boring ones, a Collector must see all the spans of a trace together. But one trace usually passes through many services on many machines, so each agent sees only a few of its spans. No single agent holds the whole picture. Decisions of this kind belong in a gateway layer, arranged so that all spans of one trace reach the same Collector.
flowchart TD
AG1["Agent on node 1"] --> LB["First layer: route by trace identity"]
AG2["Agent on node 2"] --> LB
AG3["Agent on node 3"] --> LB
LB --> S1["Sampling Collector 1"]
LB --> S2["Sampling Collector 2"]
S1 --> BE["Backend"]
S2 --> BE
A handy rule: if a job needs to see only one machine, do it in the agent. If a job needs to see many machines, or must be the same for the whole company, do it in the gateway.
10Host Metrics and Local Logs: Jobs Only an Agent Can Do
Some data lives on the machine, and only a resident can fetch it.
Host metrics
Processor load, memory use, disk space, disk activity and network traffic are facts about the machine. Applications rarely report them, and a central service cannot reach into every server to read them. An agent running on the host uses a host metrics receiver that reads these numbers directly from the operating system on a regular schedule and turns them into standard metrics.
Why it matters
When an application slows down, the first question is often “was the server itself overloaded?”. Host metrics answer that, and sitting next to application metrics and traces, they make the answer easy to find.
Local log files
Many programs, including databases, web servers and older applications, write logs to files and know nothing about OpenTelemetry. An agent uses a file log receiver to follow these files as they grow, split the text into separate entries, pull out fields such as time and severity, and convert each entry into a standard log record.
Why it matters
This lets you bring software you cannot change into the same system as everything else, without installing anything inside that software.
Remembering where it stopped
An agent that reads files must remember how far it has read. If it restarts, it should continue from where it stopped, not start again and send duplicates, or skip lines and lose them. A small storage extension lets the agent keep this bookmark on disk safely.
Think of a building caretaker who reads the electricity meter in the basement every hour. Nobody upstairs can do this, because the meter is in the basement. The caretaker also reads the notice board in the lobby and notes down new announcements, keeping a bookmark of the last one read. The agent is the caretaker of the host.
Running a file reader without any saved bookmark. After every restart, the agent re-reads old files and floods your backend with duplicate lines, or skips lines written while it was down. Use storage so progress survives restarts.
11Resource Detection and Enrichment
The agent knows where it lives, so it can label everything correctly.
A message such as “connection timed out” is nearly useless if you do not know which machine, container or region it came from. Agents are perfectly placed to add this information, because they live in the same environment as the applications. Two kinds of processors do most of this work.
Resource Detection
Discovers facts about the environment automatically, such as the host name, operating system, cloud provider, region and instance identifier.
Kubernetes Attributes
Looks up which pod, namespace, deployment and node a piece of data came from, and attaches labels such as team or application name.
Consistency across the company
Because the agent adds labels in the same way for every service on every host, dashboards and queries work the same everywhere. Teams do not need to remember to add them.
Correct cloud labels for free
The agent can ask its own cloud environment for its identity. Applications do not need cloud credentials or lookup code for this purpose.
Permissions to look things up
To find pod details, the agent needs permission to read certain information from the Kubernetes system. Grant only the read access it needs, and no more.
Agree on a small set of labels that every signal should carry, such as service name, environment, team and region, and let the agent add whatever the application did not. Consistent labels are what make shared dashboards possible.
12Reliability: Queues, Retries and Back-Pressure
What happens when the next stop is slow or down?
An agent sits between your applications and the rest of the telemetry path. If something downstream stumbles, the agent is the first to feel it. Understanding how it copes helps you avoid lost data and crashed servers.
Queues and retries
When the agent cannot deliver a batch, its exporter keeps the batch in a queue and retries with growing wait times, so a struggling destination is not hammered. By default, this queue lives in memory, which is fast, but its contents disappear if the agent restarts. For stronger safety, a persistent queue can store waiting data on disk so that it survives restarts, at the price of some speed and disk space.
Back-pressure
If the queue fills up, the agent must choose. It can drop new data, or it can tell senders to slow down. The memory limiter processor is the main protection: when memory use crosses a safe limit, it asks receivers to refuse new data for a moment. Well-behaved senders wait and retry instead of piling on.
Protecting the host
An agent shares a machine with your applications. It must never become the reason a server runs out of memory. Always give the agent a memory limit, both inside its configuration through the memory limiter and from the outside through the platform’s resource limits. Think of it as a polite guest that must stay within its own corner.
| Situation | What the Agent Does | What You Should Configure |
|---|---|---|
| Next hop is briefly unreachable | Holds data in its queue and retries | A sensible queue size and retry limits |
| Next hop is down for a long time | Queue fills and data may be dropped | A persistent queue and alerts on queue size |
| Application sends a sudden burst | Memory limiter may refuse some data briefly | A memory limit with headroom |
| Agent restarts | Memory queue is lost, disk queue survives | Persistent queue and saved file bookmarks |
| Agent crashes | Local applications must retry or drop | A supervisor that restarts it quickly and health checks |
Assuming the agent can hold unlimited data. It cannot. Every queue has a limit, and a long outage downstream will eventually cause drops. Alert on queue growth and failed sends, so you notice before data is lost.
13Security: A Small Program With Wide Access
Agents are close to everything, which makes them both useful and sensitive.
An agent runs on the same machine as your applications. It may read log files from disk, read operating system numbers, and look up details about other containers. That closeness is powerful, so treat the agent like any other trusted service on the host.
Listen only on the local address
If applications are on the same machine, the receiver should accept connections only from that machine. Listening on every network address by accident invites strangers to send data to your agent.
Encrypt the onward journey
Data leaving the agent for a gateway or backend should travel over encrypted connections, and the agent should verify the identity of the destination.
Keep credentials in few places
If every agent holds backend passwords, secrets spread across hundreds of machines. A common design is for agents to authenticate only to the gateway, and the gateway alone holds the backend credentials.
Grant least privilege
Give the agent only the file access and cluster permissions it truly needs. Reading logs from one folder is very different from reading everything on the disk.
Remove sensitive data early
Because the agent is the first stop, it is a good place for basic redaction of obvious secrets, before data travels the network.
Keep it updated
The agent is software like any other. Follow release notes, patch regularly, and prefer a custom build with only the components you use to reduce the code you must trust.
Problem
Agents run on every server with their receivers listening on all network addresses, and with the backend password stored in each agent’s configuration.
Why It’s Harmful
Anyone who can reach a server’s network can feed fake data to its agent, and anyone who gains access to one machine can steal the backend credentials used by the whole company.
Correct Approach
Bind receivers to the local address, use encrypted connections, let agents authenticate only to the gateway, and keep backend credentials in the gateway alone.
14Watching the Watcher: Monitoring the Agent Itself
A watcher that nobody watches is a silent risk.
If an agent quietly stops forwarding data, dashboards will look calm, and you may mistake silence for health. Fortunately, the Collector reports on itself, and there are a few simple things to watch.
Accepted Versus Refused
If receivers refuse data, the agent is overloaded or protecting itself, and applications may be losing telemetry.
Send Failures
Rising failures on an exporter mean the next hop is unreachable or rejecting data.
Queue Size
A queue that keeps growing means the agent produces data faster than it can send it.
Memory and Processor Use
Steady growth may indicate a leak, a heavy configuration or a flood of data.
Health checks and restarts
A health-check extension offers a simple page that the platform can ask, “are you alive?”. If the answer is no, the platform restarts the agent. Combined with a supervisor, such as the host’s service manager or Kubernetes, this ensures that a crashed agent returns quickly.
The fleet view
Because there may be hundreds or thousands of agents, look at them as a group. Chart the total refused data and total failed sends across all agents, and be able to pick out the one or two misbehaving hosts. A single agent with an odd pattern often points to a local problem, such as a full disk or a misconfigured application.
Send the agent’s own health metrics along a path that does not depend on the data it normally carries. If the main path breaks, you still want your alarm bell to ring.
15Managing Many Agents: Configuration at Scale
One agent is easy. Five thousand agents need a plan.
Agent mode multiplies the Collector across your whole estate. That brings a management challenge: how do you keep hundreds of copies configured, updated and healthy? A few practices make it manageable.
Keep the agent configuration small and the same everywhere
A short, standard configuration is easier to roll out and reason about. Heavy, special-case logic belongs in the gateway.
Store configuration in version control
Treat it as code. Review changes with teammates, and keep a history so you can roll back quickly.
Use automation to deploy
Use your platform’s tools, such as Kubernetes manifests, the Collector operator, or configuration management systems, to roll out and update agents.
Roll out changes gradually
Update a few agents first, watch their health, and then continue. A bad change to every agent at once would blind the entire company.
Pin versions
Use fixed versions so behaviour does not change unexpectedly, and read release notes before upgrading because some component settings change between versions.
Consider remote management where suitable
Some environments use a control protocol that lets a central server push configuration to agents and track their status, which can simplify large fleets.
Letting each team hand-edit its own agent configuration. Soon every host is slightly different, nobody knows why, and one bad edit can silence a service. Standardise, review and automate.
16Best Practices and Anti-patterns
Habits that keep agents small, safe and dependable.
Good habits
Keep agents thin
Use them for local acceptance, host data, simple labelling, batching and forwarding. Leave heavy, wide-view tasks to the gateway.
Always include a memory limiter first
Protect the host from a runaway agent, and give the agent resource limits at the platform level too.
Bind receivers locally
Accept connections only from the same machine or pod unless there is a clear reason to open up more.
Encrypt and authenticate onward connections
Protect the path to the gateway, and keep backend credentials out of the agents.
Use persistent queues where data matters
Let waiting data and file bookmarks survive restarts, and alert when queues grow.
Collect once what should be collected once
Use agents for per-host data, and use a single instance for cluster-wide data so that numbers are not duplicated.
Build a custom distribution for production
Include only the components your agents need. It is smaller, safer and easier to explain to a security team.
Mistakes to avoid
Problem
Tail sampling is configured inside every agent so that each agent decides which traces to keep.
Why It’s Harmful
Each agent sees only some spans of any trace, so decisions are inconsistent, important traces are dropped, and memory is wasted holding partial traces.
Correct Approach
Forward from agents to a routing layer and a gateway that sees whole traces, and make sampling decisions there.
Problem
Every agent is configured to collect cluster-wide metrics from the Kubernetes system.
Why It’s Harmful
The same numbers are collected hundreds of times, producing duplicates, wasting resources and putting needless load on the cluster control system.
Correct Approach
Collect cluster-wide data from one dedicated Collector instance, and keep agents focused on their own node.
Problem
The agent runs with no memory limit and no memory limiter processor.
Why It’s Harmful
A traffic burst or a slow destination can grow the agent’s memory until the host runs out, affecting the very applications the agent was meant to serve.
Correct Approach
Put a memory limiter first in every pipeline, and set platform-level resource limits on the agent as well.
Problem
Applications are pointed at a remote address for the agent, such as one on another machine, instead of a local one.
Why It’s Harmful
The short, reliable first hop is lost. Applications are again exposed to network delays and failures, and the main benefit of agent mode disappears.
Correct Approach
Send application data to a local address, and let the agent handle the longer journey onward.
17Advantages and Trade-offs of Agent Mode
Agent mode is powerful, but it adds something new to operate.
Advantages
- Applications stay lightweight, with delivery work moved out
- Short and reliable first hop from application to agent
- Only way to collect host metrics and local log files
- Host, cloud and Kubernetes labels added automatically and consistently
- Backend changes require no application changes
- Failures stay local to one host or node
Disadvantages / Trade-offs
- Many copies to deploy, update, secure and monitor
- Uses memory and processor time on every host
- Configuration drift risk if not standardised
- Cannot make decisions that need a view across machines
- Sidecars multiply overhead in large clusters
Choosing a pattern
| If You Have… | Consider | Reason |
|---|---|---|
| One or two services and one destination | No Collector | Simplest start, with the least to operate |
| Many hosts and a need for host metrics or local logs | Agent mode | Only a local program can collect those |
| Need for company-wide rules and sampling | Gateway mode | Central place sees everything |
| A large organisation with mixed needs | Agents feeding a gateway | Local work in agents, central work in the gateway |
“Agent mode and gateway mode are alternatives, so I must pick one.” In practice, they are usually combined. The agent handles what is local, the gateway handles what is central, and the two together cover everything.
18Real-World Scenarios
Picture how different teams use agents every day.
An online retailer on virtual machines
A retailer runs its services on a few hundred virtual machines. A Collector agent on each machine receives traces from local services, reads the web server’s log files and gathers machine metrics. All of it flows to a small gateway pool. When the team decides to move to a new monitoring vendor, they change only the gateway.
A streaming company on Kubernetes
A streaming platform runs thousands of pods. One agent runs on every node as a DaemonSet, adding pod, namespace and team labels, and reading container logs from the node. Developers simply point their services at the local node address. The platform team never has to ask developers to add labels.
A bank isolating a sensitive workload
A bank has one sensitive payments service that must send telemetry to a separate, restricted destination. The team gives that service a sidecar agent with its own configuration, while all other services use shared node agents. The sensitive data never mixes with the rest.
A factory with machines that cannot be changed
Factory controllers write logs to local files and cannot run new software. A small agent beside them reads the files, adds machine and plant labels, and forwards the records. Engineers can finally compare error patterns across plants.
A platform team stops an outage cascade
During a backend outage, applications that sent data directly began to slow down as their buffers filled. After the team introduced local agents with queues and memory limiters, the same kind of outage no longer touched application performance, because agents absorbed the trouble.
19Getting Started: A Gentle Path
You do not need to design everything at once.
Run one agent on a test machine
Use the contrib build to learn, with a receiver for local applications and a debug exporter that shows what arrives.
Point one application at it
Send the application’s telemetry to the agent’s local address, and confirm that data appears.
Add protection and polish
Add the memory limiter first and the batch processor last, then resource detection for host labels.
Connect a real destination
Forward to a backend or a gateway, using encrypted connections, and check that traces, metrics and logs arrive.
Add host metrics and local logs
Switch on the host metrics receiver and the file log receiver, with storage so progress survives restarts.
Roll out and monitor
Deploy agents using your platform’s automation, gradually, and watch refused data, send failures and queue size across the fleet.
Your first goal is not a perfect fleet. It is to see one application’s trace, one host metric and one local log line all arriving through a single agent. When that works, the rest is repetition.
20Frequently Asked Questions
It is a way of running the OpenTelemetry Collector close to your applications, such as on the same server, node or pod. Applications hand their telemetry to this nearby Collector, which adds useful labels, batches the data and forwards it onward.
No. It is the same Collector program. “Agent” and “gateway” describe where you place the Collector and what job you give it, not different products.
An agent runs next to the applications and does local work. A gateway is a shared central pool that receives data from many agents or applications and does heavier work such as company-wide rules and trace sampling. They are often used together.
Not necessarily. For a small system with one destination, sending directly or through a single Collector is often enough. Add agents when you need host metrics, local log collection, automatic host labels, or better protection for your applications.
A DaemonSet, with one agent per node, is the usual starting point because it is efficient and collects node-level data. Sidecars give stronger isolation per pod but multiply overhead, so use them only when you need per-application separation.
It is not a good place for it. An agent sees only part of each trace, but tail sampling needs the complete trace. Do it in a gateway layer arranged so that all spans of one trace reach the same Collector.
Data held only in memory queues is lost. A persistent queue on disk can preserve waiting data, and a storage extension lets file readers remember where they stopped, so they neither duplicate nor skip lines.
If it listens on every network address, anyone who can reach the machine may send data to it, which can inflate costs or hide real problems. Local binding limits access to applications on the same machine or pod.
Preferably in the gateway only. Agents authenticate to the gateway, and the gateway holds the backend credentials. This keeps secrets out of hundreds of machines and reduces the damage of one compromised host.
Yes, and this is one of its main strengths. A host metrics receiver reads processor, memory, disk and network numbers, and a file log receiver follows local log files, converting them into standard log records.
Place a memory limiter first in every pipeline, set resource limits on the agent through your platform, keep the configuration simple, and watch memory use across your fleet.
Keep one small standard configuration, store it in version control, deploy it with automation such as Kubernetes manifests or the Collector operator, roll out changes gradually, pin versions, and monitor the fleet as a group.
21Summary and Key Takeaways
Collector agent mode is a way of running the OpenTelemetry Collector right beside your applications, as a host service, a node agent or a sidecar. Applications hand their telemetry over a short, reliable local hop, and the agent adds host and environment labels, collects host metrics and local log files, batches and compresses the data, and forwards it with queues and retries to a gateway or a backend. It is the same Collector program as in any other role, only placed and configured differently. Keep agents thin and local, protect them with a memory limiter, bind receivers to the local address, encrypt the onward path, and keep backend credentials in the gateway. Leave wide-view jobs such as tail sampling and company-wide rules to a gateway layer, collect cluster-wide data once, and manage the whole fleet with version control, automation and gradual rollouts.
Key Takeaways
- Agent mode means close to the source — the same Collector program runs on the host, node or pod, accepting data over a fast local hop and forwarding it onward.
- Local jobs belong in the agent — accepting data, host and environment labels, host metrics, local log files, batching and retries are best done where the workload lives.
- Wide-view jobs belong in the gateway — trace sampling, company-wide privacy rules and backend credentials need a central place that sees everything.
- Choose placement with care — a host service or node agent is efficient, while sidecars give isolation at a higher cost in overhead and effort.
- Protect the host and the data — put a memory limiter first, bind receivers locally, encrypt onward connections, and grant the agent only the access it needs.
- Plan for failure — use queues, persistent storage and file bookmarks, and alert on refused data, send failures and growing queues.
- Manage the fleet like code — keep one small standard configuration, deploy by automation, roll out gradually, pin versions and monitor all agents together.