The same Collector software can behave very differently depending on where it runs.

The OpenTelemetry Collector is a program that receives telemetry, which is data that software sends out about its own behavior, such as traces, metrics and logs. It then processes that data and forwards it to backends, which are the tools where people look at the data. If you are new to this, you can think of the Collector as a post office for observability data.

Before the Collector can do any useful work, you must decide where to run it. That sounds like a small detail, but it shapes almost everything that follows: how much memory you spend, how safely teams are separated, whether you can see the health of the machine itself, and how painful upgrades will be. Two teams can use the identical Collector software and end up with completely different experiences purely because of placement.

The two answers this post compares

Notice that the gateway is mentioned here as a companion, not a rival. Sidecar and DaemonSet are both ways of placing a Collector close to the source of data. A gateway is a central Collector pool further away. Most mature systems combine a nearby Collector with a gateway, and the main question this post answers is which kind of nearby Collector fits which situation.

Six simple definitions that make the rest of the post easy to follow.

Both deployment styles live inside Kubernetes, so a short vocabulary lesson helps. Kubernetes is a system that runs and manages many programs across many computers. You tell it what you want running, and it keeps that true even when machines fail.

Why pods matter so much here

Containers in the same pod share a network space. This means they can talk to each other using the address “localhost”, exactly as two programs on one laptop can. That detail is the whole secret behind sidecar Collectors: the application sends data to localhost, and the sidecar is right there to catch it.

A DaemonSet Collector is in a different pod, so the application cannot reach it through localhost. The application must find the node’s Collector over the network instead. The ways to do that are an important part of this story and we cover them in chapter six.

A private Collector that travels inside every application pod.

In a sidecar deployment, the Collector runs as an extra container in the same pod as your application. If your application has twenty pods, there are twenty Collectors, one inside each. They start together, they run on the same node, and they stop together. The name comes from the motorcycle sidecar, a small extra seat bolted to the side of the main vehicle that always travels with it.

flowchart LR
  subgraph PodA["Pod A"]
    A1["Application container"] -->|"localhost"| S1["Sidecar Collector"]
  end
  subgraph PodB["Pod B"]
    A2["Application container"] -->|"localhost"| S2["Sidecar Collector"]
  end
  subgraph PodC["Pod C"]
    A3["Application container"] -->|"localhost"| S3["Sidecar Collector"]
  end
  S1 --> G["Gateway or backend"]
  S2 --> G
  S3 --> G
        

The core idea in one sentence

The sidecar is a personal assistant for exactly one application instance. It knows nothing about the neighbors, it serves no one else, and it can be tuned to the needs of that single workload.

What a sidecar is good at

What a sidecar costs

The price is multiplication. Every pod pays for its own Collector in memory, processor time and image storage. A cluster with thousands of small pods can spend more resources on Collectors than on some of the applications. Upgrades are also broader, because a new Collector version only takes effect when each pod is restarted.

From the annotation on a pod to data leaving the container.

You rarely add a sidecar by hand to every deployment. Most teams use the OpenTelemetry Operator, a program that watches Kubernetes and does the work for you. You describe a Collector configuration once and mark it as sidecar mode. Then you place a short label, called an annotation, on any workload that should receive one. When a new pod is about to be created, the Operator intercepts it and quietly adds the Collector container.

sequenceDiagram
  participant Dev as Engineer
  participant K as Kubernetes
  participant Op as Operator
  participant Pod as New Pod
  Dev->>K: Create workload with sidecar annotation
  K->>Op: Pod about to be created
  Op->>K: Add Collector container to the pod
  K->>Pod: Start application and Collector together
  Pod->>Pod: Application sends data to localhost
  Pod->>Pod: Sidecar processes and forwards data
        

Shared lifetime

Because the sidecar and the application are in one pod, they are scheduled together and deleted together. This has two practical effects. First, you never have a situation where an application runs without its Collector available, since both are placed on the same node at the same moment. Second, there is a subtle shutdown problem: when the pod is told to stop, the application may still be trying to flush its last batch of telemetry just as the sidecar is also shutting down.

Where the sidecar sends data

The sidecar usually does little heavy work. It receives data from the application, adds a few labels such as the pod name, protects itself from overload, batches data into efficient packages, and forwards it. The destination is normally a gateway Collector pool or sometimes a backend directly. Keeping the sidecar light matters, because whatever it consumes is taken from every single pod.

One shared Collector for every node, looking after everything that runs there.

A DaemonSet is a Kubernetes feature that guarantees one copy of a pod on every node. In a DaemonSet deployment of the Collector, each node runs exactly one Collector pod, and every application pod on that node shares it. When the cluster grows and a new node appears, Kubernetes automatically starts a Collector there. When a node disappears, its Collector disappears with it.

flowchart LR
  subgraph N1["Node 1"]
    P1["App pod"] --> D1["Node Collector"]
    P2["App pod"] --> D1
    P3["App pod"] --> D1
  end
  subgraph N2["Node 2"]
    P4["App pod"] --> D2["Node Collector"]
    P5["App pod"] --> D2
  end
  D1 --> G["Gateway or backend"]
  D2 --> G
        

The core idea in one sentence

The DaemonSet Collector is a shared neighborhood service. It is efficient because its cost grows with the number of machines, not the number of pods, and it can also see things about the machine that an individual pod cannot.

What a DaemonSet Collector is good at

What a DaemonSet Collector costs

The price is sharing. Many unrelated pods depend on one Collector, so a noisy neighbor can crowd out others, a Collector problem affects every pod on the node, and per-application customization becomes awkward. The Collector also needs wider permissions in order to read host files and ask the Kubernetes control plane about pods, which raises security questions that a sidecar avoids.

A sidecar is reached through localhost. A DaemonSet needs a different trick.

An application pod on a node must send its telemetry to the Collector pod on the same node, not to one on a different machine. Sending it to a random node’s Collector would add network travel and spoil the whole purpose of a node-local helper. Kubernetes offers several ways to arrange the delivery, and each has trade-offs.

Why this matters for the comparison

The sidecar has an almost magical advantage here: the address never changes. Every application always sends to localhost, and no node or service names need to be configured. With a DaemonSet you must choose one of the methods above, configure the applications, and keep that arrangement working through upgrades and cluster changes. This is one of the quiet operational costs of the DaemonSet approach.

The two styles side by side across the questions that actually decide the choice.

Reading the table, a pattern emerges. Sidecar wins on isolation, flexibility and simplicity of the connection, while DaemonSet wins on efficiency and on seeing the machine. Neither is simply better. Each is strong exactly where the other is weak, which is why many organizations use both.

Cost is the most common reason teams change their mind.

Let us do a simple imagined example with round numbers, only to show the shape of the cost. Suppose a cluster has twenty nodes and four hundred application pods. Suppose a lightweight Collector needs a modest amount of memory just to exist, before it handles any real traffic. The exact number depends on the configuration, so we will simply call it one unit.

The sidecar bill depends on pods, and pods are many. The DaemonSet bill depends on nodes, and nodes are few. A DaemonSet Collector must be sized larger because it serves many pods, so it will not be literally twenty times cheaper, but the baseline overhead of idle Collectors is dramatically lower. The savings are largest when pods are small and numerous, such as many tiny microservices or short batch jobs.

Where sidecars hurt the most

Where DaemonSets hurt the most

A DaemonSet Collector must be sized for the busiest node, or at least be able to grow, because one Collector has to absorb everything on that machine. If the nodes are very different in size, a single setting is either too small for the large nodes or wasteful on the small ones. Teams often solve this by running separate DaemonSets for different node groups.

What happens when one application misbehaves, or when the Collector itself fails.

Sharing is efficient, but sharing has a dark side. In a DaemonSet deployment, all pods on a node push telemetry into the same Collector. If one pod suddenly produces a flood of data, perhaps because of a bug that logs in an endless loop, the shared Collector becomes overloaded. Its memory protection starts refusing data, and the well-behaved neighbors on the same node lose their telemetry too, through no fault of their own.

Blast radius, explained simply

The blast radius is how much damage a single failure can cause. A sidecar failing damages one pod. A DaemonSet Collector failing, whether through a crash, a bad upgrade or an out-of-memory kill, damages every pod on that node at the same time. Across the cluster, a bad DaemonSet release can affect every node at once, which is why careful rollout matters.

Ways to reduce the DaemonSet’s risk

The shared design is not hopeless. Teams limit damage by setting sensible memory limits, using separate receivers or pipelines for very important workloads, limiting the maximum size of what any single sender may push, and rolling out new versions gradually to a few nodes first. Some teams also run a dedicated DaemonSet for a few critical namespaces, restoring partial isolation without giving every pod its own sidecar.

Who decides what the Collector does, and how easy is it to change?

Collector behavior is controlled by a configuration that describes its receivers, processors and exporters. In a sidecar world, the configuration can be different for every application. In a DaemonSet world, one configuration usually serves a large group of workloads. This difference shapes how teams work together.

Sidecar: freedom with fragmentation

Because each sidecar is configured separately, application teams can choose their own filtering, sampling and destinations. A payments team can strip card-related fields, while a search team keeps everything. The Operator allows several sidecar configurations to exist, and each workload picks one by name. This is wonderfully flexible.

The danger is fragmentation. After a year there may be dozens of slightly different configurations, each owned by a different team, each with its own quirks. Finding out which workloads still use an old setting becomes detective work, and rolling out a company-wide change means asking many teams to act.

DaemonSet: consistency with rigidity

A DaemonSet is managed by one platform team with one configuration. A company-wide rule, such as always adding the cluster name or always removing a sensitive field, is applied in one place. Everyone receives it automatically and nobody needs to remember anything.

The rigidity appears when a single team needs something different. Adding a special rule for one team to a shared configuration affects everyone, so each exception needs careful review. Over time the shared configuration can grow into a long list of exceptions that nobody dares to touch.

This is the area where the DaemonSet has a clear and almost unbeatable advantage.

Applications create traces, metrics and logs about themselves. But there is another kind of telemetry that nobody inside an application can produce: information about the machine and about all the containers on it. How much processor is the machine using? How much memory is each container consuming? What did every container write to its log files? A pod-local sidecar has very limited view of any of this.

Why logs favor the DaemonSet

Most applications on Kubernetes simply print to their standard output, and the platform stores that output in files on the node. A DaemonSet Collector mounts the node’s log folder and reads these files for every pod, with no change at all to any application. That is a powerful property: you get logs from all workloads, including old software nobody dares to modify.

A sidecar cannot see the node’s log folder by default. It can be arranged to read files that the application itself writes into a shared volume inside the pod, but this requires application changes or special pod design. For container logs at scale, the DaemonSet is the standard answer.

flowchart LR
  subgraph Node["One node"]
    A["App pod writes to standard output"] --> F["Log files on the node"]
    B["Another pod writes to standard output"] --> F
    H["Host and kubelet statistics"] --> D["DaemonSet Collector"]
    F --> D
  end
  D --> G["Gateway"]
  G --> L["Log backend"]
  G --> M["Metrics backend"]
        

Raw telemetry is far more useful when it says which pod, team and node it came from.

A trace without context is like a letter without a return address. Backends are most useful when every item carries labels such as the pod name, namespace, deployment and node. This process of adding labels is called enrichment, and the Collector has a processor dedicated to doing it for Kubernetes by asking the cluster about the pod that sent the data.

How the DaemonSet identifies the sender

A shared node Collector receives connections from many pods. To work out which pod sent a particular batch, it looks at the network address the batch came from and matches it against the cluster’s list of pods. This works well in normal conditions. It does require the Collector to have permission to watch pods in the cluster, and it can be fooled if traffic is translated by an intermediate layer so that the original address is lost.

How the sidecar identifies the sender

A sidecar sits inside the pod, so it can be told its own identity directly through environment settings that Kubernetes provides, such as the pod name and namespace. There is no guessing from network addresses. The sidecar can stamp the identity onto data with great confidence, and it needs less access to the cluster’s control plane, because it does not need to look up other pods.

Both styles can lose data. They lose it in different situations.

Every Collector keeps data briefly in memory while it batches and sends it onward. If the Collector disappears before sending, that data vanishes unless it was stored on disk. So the question is not whether data can be lost, but when, how much, and how likely.

Sidecar loss scenarios

A sidecar shares its pod’s fate. When a pod is killed suddenly, perhaps because it ran out of memory, the sidecar dies at the same instant and anything waiting in its memory is lost. During a normal shutdown there is the ordering problem described earlier. On the positive side, the application never needs to survive a network failure to reach its Collector, because the path is just localhost.

DaemonSet loss scenarios

The node Collector can be restarted independently of the applications, for example during an upgrade. While it is down, applications on that node have no local receiver. Well-behaved SDKs will retry and buffer for a while, but a long outage can overflow their buffers. If the Collector is killed for exceeding its memory limit, all data waiting on that node is lost at once, which can be a larger amount than a single sidecar would hold.

Tools that reduce loss

The two styles ask for very different levels of trust from your cluster.

Security teams care about one question above all: what can this component see and do if it is compromised or misconfigured? The answer differs sharply between sidecars and DaemonSets, and it is often the deciding factor in regulated companies.

The sidecar security picture

A sidecar lives inside an application’s pod and normally needs no special powers. It does not read host files, and it does not usually ask the cluster about other pods. If it is compromised, the attacker is already inside the same pod as the application, and gains little extra. The Collector’s credentials, if any, can be limited to that single workload’s needs.

The DaemonSet security picture

A DaemonSet Collector reading logs and node statistics needs to mount parts of the node’s file system, and may need extra permissions to talk to the kubelet and the cluster’s API. These powers are broader than a normal application should have. If this Collector is compromised, an attacker may reach log files from every pod on the node, which can contain sensitive information. It is therefore important to give it read-only access, restrict what it can mount, and keep its permissions as narrow as the task allows.

Shared practices for both

Some platforms make the choice for you.

These examples show that the right answer is often decided by the environment, not by preference. Check what your platform allows before designing, because a plan that depends on a feature your platform forbids will have to be thrown away.

Most mature platforms stop asking “which one” and start asking “which one where”.

Sidecars and DaemonSets are not mutually exclusive. A common production design uses a DaemonSet for node and log collection across the cluster, adds sidecars only for the few workloads that need isolation or special handling, and funnels everything into a gateway pool that performs the heavy and policy-related work.

flowchart LR
  subgraph Node1["Node 1"]
    A1["Regular app pods"] --> D1["DaemonSet Collector"]
    H1["Node and log data"] --> D1
    subgraph Special["Special pod"]
      A2["Isolated app"] --> S1["Sidecar Collector"]
    end
  end
  D1 --> GW["Gateway Collector pool"]
  S1 --> GW
  GW --> B1["Observability backend"]
  GW --> B2["Security backend"]
  GW --> B3["Archive storage"]
        

Who does what in the hybrid

A word about scraping metrics

Many Kubernetes applications expose metrics on a web address that a collector periodically visits and reads, which is called scraping. If every DaemonSet Collector tries to scrape every target in the cluster, the same numbers are collected many times. The OpenTelemetry Operator offers a component called the Target Allocator, which divides scrape targets fairly among Collectors so that each target is read exactly once. This is a separate deployment that works alongside the Collectors, and it is worth knowing about whenever metric scraping is part of your design.

Running these Collectors day after day is where the real costs show up.

Upgrading

Upgrading a DaemonSet is a rolling update: Kubernetes replaces the Collector pods node by node, usually a few at a time. Application pods are untouched. The platform team can complete an upgrade without asking anyone, and can stop the rollout if early nodes show trouble.

Upgrading sidecars is different. The Collector image is baked into each pod when it is created, so a new version reaches an application only when that application’s pods are restarted. A platform team that wants everyone on a new version must coordinate with every application team or force restarts. Meanwhile many versions of the Collector coexist in the cluster, each with its own behaviors and bugs.

Scaling

Sidecars scale automatically with the applications, because adding a pod adds its Collector. The sidecar’s resource limits, however, are fixed per pod and may need tuning per workload. A DaemonSet scales with nodes, but within a node its capacity is fixed. A very busy node may overload its single Collector, and the only remedies are raising its limits or splitting the work, for example through a second Collector dedicated to logs.

Monitoring the Collectors themselves

All Collectors publish their own health numbers, such as how much data they accepted, refused and failed to send, and how full their queues are. In a DaemonSet design you watch a manageable number of Collectors, one per node. In a sidecar design you may need to watch thousands, so you rely on aggregated views and alerts on patterns instead of looking at each one.

Six situations and the choice that usually fits each.

A balanced summary of what you gain and what you pay with each style.

Sidecar

DaemonSet

The trade-offs mirror each other. What one style makes easy, the other makes hard. That symmetry is why a thoughtful design starts by listing which of the strengths above are essential for the workloads in question, instead of looking for a universal winner.

Habits that work well, and habits that cause trouble later.

Patterns worth adopting

Anti-patterns to avoid

A checklist to run through before you commit to a design.

Best practices

Common mistakes

A simple path from your situation to a sensible starting design.

flowchart TD
  S["Start: where should the Collector run?"] --> Q1{"Can your platform run DaemonSets?"}
  Q1 -->|"No"| SC["Use sidecars, keep them thin"]
  Q1 -->|"Yes"| Q2{"Do you need node metrics or container logs?"}
  Q2 -->|"Yes"| DS["Use a DaemonSet as the base"]
  Q2 -->|"No"| Q3{"Does a workload need strict isolation or custom handling?"}
  Q3 -->|"Yes"| SC2["Sidecar for that workload"]
  Q3 -->|"No"| DS2["DaemonSet is usually enough"]
  DS --> Q4{"Any workload with special isolation needs?"}
  Q4 -->|"Yes"| HY["Hybrid: DaemonSet plus selective sidecars"]
  Q4 -->|"No"| DS3["DaemonSet alone"]
  SC --> GW["Add a gateway behind it"]
  HY --> GW
  DS3 --> GW
  DS2 --> GW
  SC2 --> GW
        

Treat this guide as a starting point and not a verdict. Your own measurements of cost, your own security rules and your own team structure will tip the balance, and it is perfectly reasonable to begin with one style and change as you learn.

Short, direct answers to the questions people ask most.

Neither is better in every case. A sidecar offers isolation, flexibility and a simple localhost connection. A DaemonSet offers lower overhead and a view of the machine and its logs. The right choice depends on your platform, your ownership model and your cost limits.

Yes, and many organizations do. A DaemonSet commonly covers general workloads, logs and node metrics, while sidecars serve the few workloads that need isolation. A gateway usually sits behind both.

The sidecar uses a slice of the same pod’s resources, so a poorly sized one can compete with the application for memory and processor time. With sensible limits and a light configuration the effect is small, but it should be measured.

Not in the general sense. It does not see the node’s log folder. It can read files that the application writes into a shared volume inside the pod, but that requires application or pod changes. A DaemonSet is the usual tool for cluster-wide log collection.

The DaemonSet Collector lives in its own pod, and localhost only reaches containers inside the same pod. Applications must reach the node Collector through the node’s address or a service arranged to deliver locally.

During the rolling update, each node’s Collector is replaced briefly. Applications on that node may need to retry for a short time. Well-configured SDK buffers and overlapping rollouts reduce the chance of loss.

Not strictly, but it is strongly recommended in larger systems. A gateway centralizes secrets, redaction, routing and whole-trace sampling, which lets the edge Collectors stay small and simple.

The OpenTelemetry Operator can inject sidecars automatically based on an annotation, manage Collectors in several modes including DaemonSet, and run the Target Allocator for metric scraping. It reduces manual work and keeps configurations consistent.

Usually when pods are many and small, but not always. A DaemonSet occupies part of every node, including nearly empty ones, and must be sized for busy nodes. Compare the total for your actual cluster before deciding.

Sidecars generally need fewer permissions and keep data within the pod boundary at the first hop. DaemonSets need broader access to read node files, so they must be tightly restricted. In both cases, redaction at a controlled gateway is a strong additional safeguard.

The biggest risks are cumulative overhead and sprawl. Many Collectors with many configurations and many versions are expensive to run and hard to keep consistent.

The biggest risks are shared fate and wide permissions. One overloaded or misconfigured node Collector affects every pod on that node, and the host access it needs deserves careful control.

Yes. Because applications speak the same protocol to either kind of Collector, switching placement mostly changes how the Collector is deployed and how applications locate it. Keep your configuration modular and expect to adjust as the cluster evolves.

A sidecar Collector runs inside every application pod, giving each workload a private, localhost-connected Collector with strong isolation and flexible configuration, at the price of overhead that multiplies with pod count and upgrades that require restarting applications. A DaemonSet Collector runs once on each node and is shared by every pod there, giving low overhead, easy fleet management and natural access to node metrics and container logs, at the price of noisy-neighbor risk, a wider blast radius and broader permissions.

Most real systems benefit from combining them: a DaemonSet as the default coverage, sidecars for the few workloads with genuine isolation needs, and a gateway behind both for policy, routing and whole-trace decisions. Choose by asking who owns the problem, what your platform allows, and what the real numbers say for your cluster, and keep the design modular so that it can change as you learn.