Before we meet the OpenTelemetry Operator, we need to understand the general idea of an operator, because everything else builds on it.

Kubernetes in One Paragraph

Kubernetes is a system that runs containers across many machines. You describe what you want, for example “keep three copies of this application running,” and Kubernetes keeps working until reality matches the description. Each kind of thing it manages, such as a Pod, a Deployment or a Service, is called a resource. You write the desired state in a file and give it to the cluster, and the cluster’s built-in controllers do the rest.

Custom Resources and Controllers

Kubernetes lets you invent new kinds of resources through something called a Custom Resource Definition, usually shortened to CRD. A CRD teaches the cluster a new word, such as “OpenTelemetryCollector.” Once the word exists, people can write short descriptions that use it. But a description alone does nothing, so a custom controller is also needed. The controller watches for these descriptions and takes real action, such as creating Pods, Services and configuration files that match them.

The Operator Idea

An operator is simply a custom controller together with the custom resources it understands, packaged to capture the knowledge of a human expert. The word comes from the idea of a human operator, someone who knows how to install, configure, upgrade and repair a complicated piece of software. The operator turns that expert’s checklist into code that runs inside the cluster all day and night.

The OpenTelemetry Operator is a Kubernetes operator from the OpenTelemetry project that manages two big jobs: running Collectors and injecting automatic instrumentation.

A Short Reminder of OpenTelemetry

OpenTelemetry is an open standard for producing and moving telemetry, which means traces, metrics and logs. Applications are instrumented, either by developers or by agents, so they emit this data. A component called the Collector receives the data, processes it and sends it to one or more storage and analysis tools. The standard is neutral, so you can change backends without rewriting your applications.

The Two Problems the Operator Solves

Running OpenTelemetry on Kubernetes by hand creates two repeating chores. The first is deploying and maintaining Collectors, which means writing Deployments, DaemonSets, ConfigMaps and Services, keeping them in sync, and updating them safely. The second is instrumenting applications, which usually means changing images, adding agents and setting many environment variables in every service. Both chores are repetitive and easy to get subtly wrong when multiplied across many teams.

Who Benefits Most

Platform teams benefit the most, because they can offer a standard telemetry service to many product teams without touching each application. Product teams benefit because they get telemetry with little effort. Security and operations teams benefit because configuration is visible, reviewable and consistent. Small teams with a single application may find the operator more machinery than they need, a point we return to when discussing trade-offs.

Where the Operator Lives and Who Maintains It

The operator is an open-source project maintained by the OpenTelemetry community, developed in its own repository and released under a permissive licence. Because it follows the same community process as the rest of OpenTelemetry, its releases, bug fixes and documentation evolve in the open. This also means features can change between versions, which is why reading release notes and pinning versions matter so much in practice.

What the Operator Is Not

It helps to be clear about the boundaries. The operator is not a place to store or look at telemetry, so you still need a backend such as a tracing store, a metrics database or a logging platform. It is not a replacement for the Prometheus Operator, although it can work alongside it and reuse the same monitor resources. It is also not a magic switch that makes every application perfectly observable, because automatic instrumentation captures common activity but cannot understand your business.

Operator versus a Plain Helm Chart

A common question is whether a Helm chart for the Collector is enough. A chart renders templates once and installs the result. It is excellent for simple, predictable installs. An operator keeps running after installation, watching for drift and reacting to changes, and it can take actions at runtime, such as modifying Pods through webhooks. The right choice depends on how much runtime behaviour you need.

Understanding the moving parts makes every later decision easier, especially when something goes wrong at three in the morning.

The Big Picture

The operator itself runs as a Deployment inside your cluster, usually in its own namespace. It contains a controller manager that runs several reconcilers, one for each custom resource type, and a webhook server that receives calls from the Kubernetes API server. It talks to the cluster API to read your descriptions and to create or update the objects needed to match them.

flowchart TD
  U["You apply custom resources"] --> API["Kubernetes API server"]
  API --> W["Operator webhook server"]
  API --> C["Operator controller manager"]
  C --> R1["Collector reconciler"]
  C --> R2["Instrumentation handling"]
  C --> R3["Target Allocator reconciler"]
  R1 --> O1["Deployments, DaemonSets, StatefulSets, ConfigMaps, Services"]
  R3 --> O2["Target Allocator workload"]
  W --> P["Pod mutation for sidecars and agents"]
        

The Custom Resources

The Webhooks

Admission webhooks are the secret behind the operator’s most magical features. When a new Pod is about to be created, the API server pauses and asks the operator’s mutating webhook if it wants to change the Pod. If the Pod carries an instrumentation or sidecar annotation, the webhook adds extra containers, volumes and environment variables on the spot. The Pod then starts already modified, and the application’s own manifest never has to change.

The operator also uses validating webhooks, which check that your descriptions make sense before they are accepted. For example, a collector resource with an invalid mode can be rejected immediately with a clear message, rather than failing quietly later.

Certificates and Trust

The API server only talks to webhooks over secure connections, so the operator needs valid certificates. In the usual setup a tool called cert-manager creates and renews them automatically. The operator can also generate its own certificates in some configurations, which is helpful in simple clusters but gives you less control over rotation and trust.

The Reconciliation Loop in Detail

Every reconciler follows the same rhythm. First, it is woken up because something changed, such as a new custom resource, an edited one, or a change in one of the objects it created. Second, it reads the current description and works out what the world should look like. Third, it looks at what actually exists. Fourth, it creates, updates or deletes objects to close the gap. Finally, it records the outcome and goes back to waiting. This loop is safe to repeat, because running it twice with no changes does nothing.

Ownership and Automatic Cleanup

Every object the operator creates carries an owner reference that points back to the collector resource. Kubernetes uses this link for garbage collection. If you delete the collector resource, all objects it owns are removed automatically, with no leftover ConfigMaps or Services. It also tells you at a glance which resource is responsible for any given object, which helps a great deal when debugging.

Status and Leader Election

The operator writes information back to the resource’s status, such as the version of the Collector in use and how many replicas are ready. This lets you check health with a simple query instead of searching through Pods. When you run several copies of the operator for availability, leader election ensures only one acts at a time, so two controllers never fight over the same objects.

Installing the operator is straightforward, but the surrounding choices decide how smooth your first month will be.

What You Need First

The operator needs a working Kubernetes cluster with permission to install custom resource definitions and cluster-wide roles. Most installations also need cert-manager, because of the webhook certificates described earlier. You should decide in advance which namespace will host the operator, which namespaces it will watch, and who in your organisation is allowed to create the custom resources it manages.

Ways to Install

Versions and Compatibility

The operator is released frequently, and each release states which Kubernetes versions and which Collector versions it supports. The operator can also pick a default Collector image that matches its own version, which keeps managed collectors aligned with what the operator was tested against. Always read the release notes before upgrading, because custom resource schemas can change, and some features are controlled by feature gates that start disabled and later become the default.

This is the most important resource in the operator, because it turns a short description into a running collector fleet.

What You Describe

A collector resource contains a handful of important fields. The mode says how the collector should run. The image says which Collector build to use. The configuration section holds the usual Collector pipeline description, with receivers, processors, exporters and the pipelines that connect them. Other fields cover the number of replicas, resource requests and limits, environment variables, volumes, scheduling rules and the service account to use.

What the Operator Builds From It

For each collector resource, the operator creates several Kubernetes objects. It generates a ConfigMap that holds the collector configuration, a workload object such as a Deployment, DaemonSet or StatefulSet, one or more Services so other Pods can reach the collector, and a service account if you did not provide one. If you enable features such as autoscaling or disruption budgets, it creates those objects too. All of these are owned by the collector resource, so deleting the resource cleans them up automatically.

The Value of Ports and Services Derived From Configuration

A nice feature is that the operator reads your collector configuration and works out which ports the receivers use, then exposes them automatically on the Service. This means you rarely need to maintain a separate list of ports. If you add a new receiver to the pipeline, the operator notices and updates the Service so traffic can reach it. This removes a common source of mismatch between configuration and networking.

A Tour of the Useful Fields

Beyond the mode, image and configuration, a collector resource exposes many Pod-level settings, which means you can tune a managed collector almost as finely as a hand-written workload. Knowing which fields exist saves you from editing generated objects, which the operator would overwrite.

Configuration Is Validated Early

The operator checks the structure of the collector configuration when the resource is created or changed. Some mistakes are caught immediately, for example a missing pipeline section or a wrong type. Other mistakes, such as naming a component that is not included in your chosen image, can only be discovered when the Collector starts. Use a test cluster and a small canary collector to catch these before they reach production.

Using Environment Values Inside the Configuration

The Collector configuration language lets you refer to environment variables, which is the correct way to keep secrets and per-environment values out of the written text. You declare the variable on the collector resource, supplied from a Secret or a plain value, and refer to it by name inside the configuration. The same description can then be reused across environments with only the variable values differing.

The mode is the single most important design choice you make for a collector, because it decides where the collector runs and what it is good at.

Mode One: Deployment

In deployment mode the operator creates a standard Kubernetes Deployment, which runs a chosen number of identical collector Pods anywhere in the cluster. This is the usual choice for a central gateway that many applications and agents send data to. It scales simply by changing the replica count or enabling autoscaling, and it suits tasks such as batching, filtering, sampling and exporting to backends.

Mode Two: DaemonSet

In daemonset mode the operator runs exactly one collector Pod on every node. This is the classic agent pattern. Applications send data to the collector on their own node, which keeps network hops short and makes it easy to collect node-level information such as host metrics and container logs. It is also the right mode when the collector needs access to files on the node.

Mode Three: StatefulSet

In statefulset mode each collector Pod gets a stable identity and can have its own persistent storage. This matters for components that need to remember something between restarts, such as a persistent queue, and for the Target Allocator pattern, where each collector replica is assigned a predictable share of Prometheus targets.

Mode Four: Sidecar

In sidecar mode the operator does not create a standalone workload at all. Instead it injects a collector container into the Pods of your application when they are created, triggered by an annotation. Each application Pod therefore gets its own private collector. This gives strong isolation and lets each team tune its own pipeline, at the price of more containers and more resource use.

Because the operator controls which Collector build runs and how it is configured, these choices affect security, size and available features.

Which Collector Image to Use

The OpenTelemetry project publishes several Collector builds. A core build contains a small set of well-supported components. A contrib build contains a very large collection of community components, such as many receivers, processors and exporters. A Kubernetes-focused build and vendor distributions also exist. The operator lets you set the image on each collector resource, and it provides a default if you do not.

Building a Custom Distribution

Many organisations build their own slim distribution that contains only the components they actually use, using the OpenTelemetry Collector builder tool. The result is a smaller image with fewer dependencies to patch, and you can point your collector resources at it through the image field. This takes some upfront effort, but it pays back in security reviews and faster Pod startup.

Keeping Configuration Safe and Readable

Collector configuration can become long and complicated. Keep it readable by using clear names for each pipeline, adding comments explaining why unusual rules exist, and splitting very large setups into several collectors by purpose. Store the resources in Git and review changes the same way you review code. Secrets, such as backend tokens, should come from Kubernetes Secrets mounted or injected as environment variables, never written into the configuration text itself.

Understanding Collector Components in Plain Words

A Collector pipeline is made of four kinds of parts, and the operator simply passes your description of them to the Collector. Receivers are the doors through which data enters, for example one that speaks the OpenTelemetry protocol or one that scrapes Prometheus endpoints. Processors are the workers in the middle that change the data, such as batching it, adding Kubernetes details, filtering unwanted items or limiting memory. Exporters are the doors out, sending data to a backend. Connectors link one pipeline to another, for instance turning spans into metrics.

Enriching Data With Kubernetes Details

A very common processor adds Kubernetes information, such as the Pod name, namespace, node and workload labels, to every piece of data that passes through. It works by recognising the sender, usually from its network address or an identifier on the data, and looking up the matching Pod from the cluster. This needs read access to Pod information, which the operator can help you grant through a service account. The payoff is large, because telemetry that knows where it came from is far easier to search and group.

Auto-instrumentation lets an application produce telemetry without its developers changing a single line of code.

Manual versus Automatic Instrumentation

Instrumentation means adding the code that records what an application is doing. Manual instrumentation means developers add that code themselves, choosing exactly what to record. Automatic instrumentation uses an agent that attaches to the running application and records common activity, such as incoming web requests, outgoing calls and database queries, based on knowledge of popular libraries. It gives broad coverage quickly, though it cannot know your business-specific events.

What the Operator Adds

Normally, setting up an agent means rebuilding images, downloading the right agent version, passing the right startup options and setting a dozen environment variables, for every service and every language. The operator does all of this at Pod creation time. You define the settings once, in an Instrumentation resource, and mark which workloads should receive them. The operator then prepares each new Pod so the agent is in place when the application starts.

Why This Matters at Scale

In a company with hundreds of services, asking every team to learn and apply instrumentation setup is slow and uneven. With the operator, a platform team can offer instrumentation as a service. Teams opt in with one annotation, and the platform team controls versions, endpoints and sampling in a single place. When a new agent version is released, the platform team updates one resource and new Pods start using it, a vast improvement over chasing hundreds of repositories.

How Auto-Instrumentation Compares With Other Approaches

The operator’s approach is one of several ways to get telemetry from an application. Each approach has strengths, and the best systems often combine them. The table compares the main options so you can see where agent injection through the operator fits.

The Limits of Automatic Agents

An agent can only record what it understands. It will see an incoming web request and an outgoing database call, but it will not know that a request was a premium customer’s checkout or that a retry loop was triggered by a business rule. It also depends on the libraries your application uses being supported. For unusual frameworks or custom protocols, you may see gaps in traces, such as a call that appears out of nowhere or a missing link between two services.

The injection flow is the heart of the operator’s most visible feature, so it is worth following one Pod through the whole journey.

The Journey of One Pod

The journey starts when a Deployment, or another controller, asks the cluster to create a new Pod. Before the Pod is stored, the API server calls the operator’s mutating webhook. The webhook checks the Pod’s annotations to see whether instrumentation is requested for a specific language. If it is, the webhook finds the matching Instrumentation resource, then modifies the Pod description before it is saved.

sequenceDiagram
  participant D as Deployment controller
  participant A as API server
  participant W as Operator webhook
  participant K as Kubelet
  D->>A: Create Pod
  A->>W: Ask to mutate Pod
  W->>W: Read annotation and find Instrumentation
  W-->>A: Return Pod with init container, volume and env vars
  A->>K: Schedule modified Pod
  K->>K: Run init container to copy agent
  K->>K: Start application with agent loaded
        

Why Only New Pods Are Affected

The webhook acts at Pod creation. Existing Pods are never touched, because Kubernetes does not let you reshape a running Pod in this way. After you add an annotation to a Deployment, its Pods must be recreated, usually by a rolling restart, before injection happens. This is also why a change to the Instrumentation resource only reaches Pods created afterwards.

Different runtimes load agents in different ways, so the operator treats each language separately and asks you to name the language you want.

The Language Families

The Annotation Pattern

The workload carries an annotation whose name includes the language, for example one meaning “inject Java instrumentation.” The value says which Instrumentation resource to use. A simple “true” means use the one in the same namespace. A name selects a specific resource, and a namespace-and-name pair can point to one in another namespace. Putting the annotation on a namespace instead of a workload applies it to every Pod created there, which is convenient but broad.

Multiple Containers and Languages

Some Pods have several containers, and they may even use different languages. The operator supports selecting which containers to instrument, so that sidecars, proxies and helper containers are left alone. This is important because injecting an agent into the wrong container wastes resources and can cause confusing failures.

Practical Notes for Each Runtime

Each language brings its own small surprises, and knowing them in advance saves hours of confusion. Java agents add some startup time and memory, so watch container limits after enabling them. Node.js and Python rely on startup hooks, so applications that launch through unusual wrapper scripts may bypass the hook, and the agent never loads. For .NET, the base image matters, because the agent must match the operating system library family inside the container. Go instrumentation relies on observing the running program from outside, so it needs privileges that other languages do not.

The Instrumentation resource is where the platform team decides how every instrumented application should behave.

The Main Settings

Kubernetes Details Added for Free

One of the nicest features is that the operator fills in useful resource attributes from the Pod itself, such as the Pod name, namespace, node and workload name. It does this by passing information into the container through standard Kubernetes mechanisms. These attributes make telemetry far more useful, since you can instantly filter traces and metrics by namespace or workload without asking developers to add anything.

Sampling Choices in Practice

Sampling at the agent is the cheapest place to reduce volume, because dropped data never leaves the process. A parent-based sampler is common, where each new trace is sampled at a fixed ratio, and downstream services follow the decision they receive. This keeps traces complete across services. For smarter decisions, such as always keeping errors, teams often combine agent-level head sampling with tail sampling in a collector.

How Service Names and Attributes Are Chosen

Every telemetry stream needs a service name, otherwise your tools show a pile of unnamed data. In many setups the operator derives the service name from the workload, using the name of the Deployment or a standard Kubernetes label. Recent versions can also use well-known labels, such as the application name and version labels, to fill in the service name and version. Whichever source you rely on, agree on one convention across the company and document it, because a consistent naming scheme is what makes dashboards and alerts reusable.

One Resource or Many

You can define several Instrumentation resources in different namespaces, each with its own endpoint, sampler or image set. This is useful when environments differ, for example a development namespace that samples everything and a production namespace that samples a small fraction. A workload can name a specific resource in its annotation, so teams with special needs can opt into a tuned setting without affecting others.

Agent Versions and Rollouts

Because agent images are defined in the Instrumentation resource, upgrading an agent is a one-line change that affects only newly created Pods. This gives a natural gradual rollout, since Pods pick up the new version as they restart through normal deployments. If a new agent version misbehaves, reverting the resource change and restarting affected workloads returns them to the previous version.

When you scrape Prometheus metrics with several collector replicas, something has to make sure each target is scraped by exactly one of them.

The Problem It Solves

Many Kubernetes applications expose metrics in the Prometheus format, and the Collector can scrape them with a Prometheus receiver. If you run a single collector, it scrapes everything, which becomes a bottleneck as the cluster grows. If you run several replicas with the same configuration, each one finds the same targets and scrapes all of them, so the data is duplicated and the load is multiplied instead of shared.

What the Target Allocator Does

The Target Allocator is a separate component, managed by the operator, that discovers scrape targets and divides them fairly among the collector replicas. Each replica asks the allocator which targets it should scrape, and the allocator answers with a distinct share. When Pods appear or disappear, the allocator updates the shares, so scraping rebalances automatically. It also understands the ServiceMonitor and PodMonitor resources that many teams already use with the Prometheus Operator, which makes migration easier.

flowchart LR
  SM["ServiceMonitor and PodMonitor resources"] --> TA["Target Allocator"]
  K8["Kubernetes service discovery"] --> TA
  TA --> C1["Collector replica 1"]
  TA --> C2["Collector replica 2"]
  TA --> C3["Collector replica 3"]
  C1 --> B["Backend"]
  C2 --> B
  C3 --> B
        

Allocation Strategies

The allocator can use different strategies to share targets. A simple strategy gives each replica the least-loaded share at each moment. Another strategy keeps targets attached to the same replica as long as possible, which reduces gaps and duplicates when the set of replicas changes. A third strategy groups targets by node, which pairs well with the daemonset mode, so each node-level collector scrapes only its own node. Pick the strategy that matches your mode and your tolerance for movement.

When Not to Use the Target Allocator

The allocator is valuable when several collector replicas must share a large number of scrape targets. If your cluster is small and a single collector can scrape everything comfortably, the extra component adds operational work without a clear benefit. Likewise, if your applications push metrics using the OpenTelemetry protocol instead of exposing Prometheus endpoints, there is nothing to scrape and no allocation needed. Choose it because a real scaling or duplication problem exists, not because it sounds more advanced.

The pieces described so far can be combined in several well-known ways, each with its own strengths.

Pattern One: Agent Only

Every node runs a daemonset collector that receives data from local applications and exports directly to the backend. It is simple and fast to set up, and works well for small to medium clusters. The downside is that every node-level collector needs backend credentials, and global decisions such as tail sampling are hard to make, because no single collector sees an entire trace.

Pattern Two: Agent and Gateway

Node-level collectors forward data to a central pool of deployment-mode collectors, which perform heavier work and export to the backend. This is the most common production pattern. It keeps credentials in one place, allows central policies such as filtering and sampling, and gives you a single point to protect from overload.

Pattern Three: Sidecar per Application

Each application Pod receives its own collector container. This gives strong isolation, so one team’s heavy pipeline cannot disturb another’s, and allows per-application customisation. The cost is higher resource use and more containers to manage, so it is best reserved for workloads that truly need it.

flowchart LR
  A1["App Pods"] --> D1["Node agent collectors: DaemonSet"]
  A2["Apps with auto-instrumentation"] --> D1
  D1 --> G["Gateway collectors: Deployment"]
  G --> T["Tail sampling and filtering"]
  T --> X["Backends"]
        

Choosing Between Them

A telemetry pipeline that falls over during a traffic surge fails exactly when you need it most, so scale and resilience deserve early attention.

Scaling the Gateway

For deployment-mode collectors the operator can create a Horizontal Pod Autoscaler, which adds or removes replicas based on measurements such as CPU or memory use. Set sensible minimum and maximum replica counts, and choose resource requests that match real usage. A gateway that is slightly over-provisioned is far cheaper than one that drops data during a sale or an incident.

Protecting Against Overload

Pair scaling with protective settings inside the collector. A memory limiter processor prevents a sudden flood from crashing the process, and a batch processor makes exports efficient. Sending queues and retries inside exporters smooth over short backend outages. The goal is for the system to bend under pressure and recover, rather than to break and lose everything.

Keeping Collectors Available During Changes

A Pod disruption budget limits how many collector Pods can be taken down at once during node maintenance or upgrades. Spreading replicas across nodes and zones with scheduling rules protects against the loss of a single machine or data centre. For the Target Allocator pattern, check how scrape ownership moves when replicas restart, so you understand any short gaps.

Capacity Planning With a Simple Example

Good sizing starts from measurement. Suppose your applications together send fifty thousand spans per second at peak, and the average span, once encoded, is about one kilobyte. That is roughly fifty megabytes per second of incoming data, so a gateway tier must comfortably handle that rate with headroom. If one collector replica can sustainably process ten thousand spans per second with its configured processors, you would plan for at least five replicas, and in practice seven or eight to survive the loss of a node and absorb bursts.

This kind of arithmetic is only a starting point, because real capacity depends on which processors you run. Tail sampling holds traces in memory while waiting, so it needs much more memory than simple batching. Always confirm with a load test that mimics real traffic, and revisit the numbers whenever traffic or processing rules change.

Backpressure and Queues

When a backend slows down, a collector has to decide what to do with data it cannot send yet. A sending queue holds it for a while, and retries try again after short delays. Queues are held in memory by default, which is fast but lost on restart. A persistent queue writes data to disk, which survives restarts but needs storage and careful sizing. In the operator, a statefulset collector with a volume is the natural home for persistent queues.

An operator holds broad powers inside the cluster, so its security needs the same attention as any other privileged component.

Permissions of the Operator Itself

To do its work, the operator needs permission to create and update Deployments, DaemonSets, StatefulSets, ConfigMaps, Services and more, and to watch the custom resources it manages. It also controls webhooks that touch Pod creation across the cluster. Treat its service account, its image and its namespace as sensitive, restrict who can change its configuration, and keep it patched.

Who May Create Custom Resources

Anyone who can create an OpenTelemetryCollector resource can cause Pods to run in your cluster, and anyone who can create an Instrumentation resource can influence which images are injected into other Pods. Use standard Kubernetes role-based access control so only trusted platform engineers can create these resources, or limit teams to their own namespaces. Without this, a well-meaning user could accidentally cause a large rollout, and a bad actor could inject unwanted code.

Securing the Telemetry Itself

Special Care for Privileged Instrumentation

Most language agents run entirely inside the application container with no special rights. The Go approach, however, relies on eBPF, which needs elevated privileges and access to parts of the node. Because this widens the attack surface, use it only for workloads that need it, run it in tightly controlled namespaces, and follow your security team’s policies for privileged containers.

A system that watches everything else must be watched itself, otherwise a silent failure will leave you blind without warning.

What to Watch on the Operator

The operator exposes its own health information, including whether its Pods are running, how often its reconciliation loops succeed or fail, and how long they take. Repeated reconciliation errors usually mean a custom resource is invalid or a permission is missing. Also watch webhook call duration and failures, since slow webhooks delay Pod creation across the cluster.

What to Watch on Managed Collectors

Collectors publish internal metrics about the data they receive, process and export. Useful signals include the number of data points accepted and refused, the number sent successfully and the number that failed to send. Queue size and memory use show when a collector is nearing overload, and restart counts show instability. Send these self-metrics to a place that is independent of the collector fleet being watched.

Alerts Worth Setting

Building Useful Dashboards

A good operations dashboard for this system answers four questions at a glance. Is the operator healthy? Are all expected collectors running and ready? Is data flowing in and out at normal rates? Is anything being refused or failing to send? Put these four panels at the top, and add detail below for memory use, queue size, restarts and the latency of exports. Keep the dashboard simple enough that an on-call engineer can read it in thirty seconds.

Using Logs and Events

The operator writes logs that describe what it decided to do and why, and Kubernetes events record notable moments, such as an object being created or a webhook failing. Collect these like any other workload’s logs. When you investigate an incident later, the sequence of events in the operator’s logs often reconstructs the story precisely, including the exact moment a resource was changed.

The operator makes upgrades easier, but only if you treat upgrades as a planned process rather than a surprise.

Two Things Get Upgraded

There are two separate upgrade tracks. The first is the operator itself, which brings new features, new custom resource versions and changed defaults. The second is each managed collector and each injected agent, which have their own versions. The operator can help by moving managed collector instances toward a version it was tested with, depending on your settings, but you remain responsible for deciding when and how.

Upgrade Order and Safety

Read the release notes first. Upgrade custom resource definitions carefully, because the cluster needs to understand any new schema before resources use it. Upgrade the operator in a test cluster, confirm that existing collectors and instrumentation still behave, then roll out to production in stages. Keep the previous version’s manifests available so you can roll back quickly if something unexpected appears.

GitOps Makes It Reviewable

Because everything the operator manages is described in plain resources, you can store them in Git and let a GitOps tool apply them. Every change then has an author, a reason, a review and a history. Rolling back becomes reverting a commit. This pairs naturally with the operator model, since both rely on the same idea: describe the desired state clearly, and let automation make it real.

Most problems with the operator fall into a few well-known patterns, and knowing them turns a long investigation into a short checklist.

Problem: Injection Does Not Happen

The Pod starts without an agent. The usual causes are an annotation placed in the wrong spot, a language name that does not match, an Instrumentation resource that does not exist in the expected namespace, or a Pod that was created before the annotation was added. Check the Pod’s final description to see whether an init container and environment variables are present, and look at the operator logs for webhook messages.

Problem: The Webhook Blocks or Delays Pods

New Pods hang or are rejected. Typical causes are an unavailable operator, expired or untrusted certificates, or network rules that stop the API server from reaching the webhook port. Check certificate validity, confirm that the webhook Service has healthy endpoints, and review the webhook’s failure policy to understand what happens when it is unreachable.

Problem: Collector Pods Crash or Restart

Look at the collector logs first. Configuration errors, such as a misspelled component name or a component missing from the chosen image, are common. Memory kills point to missing limits or missing memory limiter settings. If you use a slim custom image, confirm that every component named in the configuration is actually compiled into it.

Problem: Data Reaches the Collector but Not the Backend

Check the collector’s own metrics for failed sends, then verify credentials, endpoints and network policies. Look for refused data caused by memory pressure. Confirm that the pipeline definition connects the receiver to the exporter, since a component that is defined but not used in any pipeline does nothing.

A Calm Debugging Routine

When something is wrong, resist the urge to change things at random. Follow a fixed routine, which narrows the problem quickly and leaves a clear record of what you tried.

Temporary Debug Tools

A debug exporter that prints received data to the collector’s own logs is a handy way to see exactly what arrives and what leaves. Turn it on for a short time in a test collector, and never leave it enabled in a busy production pipeline, because printing every record is costly and may expose sensitive data in logs.

Naming common mistakes makes them easier to spot in reviews and design discussions.

Seeing the operator in a few realistic situations makes the earlier ideas concrete, and shows where it helps and where it adds burden.

A Gentle Adoption Path

Teams that succeed with the operator rarely adopt everything at once. They climb in small steps, gaining value and confidence at each one before taking the next. This path keeps risk low and lets everyone learn the pieces in a sensible order.

Measuring Success

Decide how you will know the operator is helping. Useful measures include the time it takes a new service to start producing useful telemetry, the share of services with consistent naming and sampling, the number of incidents caused by telemetry configuration mistakes, and the effort the platform team spends on collector upkeep. If these improve over a few months, the operator is earning its place.

Short, direct answers to the questions beginners and experienced engineers ask most often.

No. You can run collectors with ordinary Deployments or Helm charts and instrument applications by hand. The operator is a convenience that adds declarative management, code-free instrumentation and Target Allocator support. It pays off most when many teams or services share one cluster and consistency matters.

No. The Collector is the program that receives, processes and exports telemetry. The operator is a separate program that runs in the cluster and creates and manages Collectors, along with other things such as instrumentation injection. One manages the other.

The Kubernetes API server only calls webhooks over secure connections, so the operator’s webhook needs a valid certificate. Cert-manager is the common way to create and renew it automatically. The operator can also generate certificates itself in some setups, but then you manage rotation and trust more manually.

Injection happens when a Pod is created, not while it is running. After adding the annotation to the Pod template, restart the workload so new Pods are created. Also confirm the annotation is on the Pod template, not on the Deployment’s own metadata.

For a central gateway, use deployment mode. For node-level collection and logs, use daemonset mode. For Prometheus scraping with the Target Allocator, use statefulset mode. For strong isolation per application, use sidecar mode. Many teams combine a daemonset agent layer with a deployment gateway.

It supports several important ones, including Java, Node.js, Python and .NET, plus Go through an eBPF-based approach and web servers such as Apache HTTPD and Nginx. Coverage and maturity differ between languages, so check the current documentation for what is supported and what limits apply to your runtime.

It discovers Prometheus scrape targets and divides them among collector replicas so each target is scraped by exactly one replica. It also reads ServiceMonitor and PodMonitor resources, which helps teams reuse their existing Prometheus configuration.

Agents add some overhead, because they record and export telemetry and add small steps to startup. The size depends on the language, the agent and your sampling settings. Measure on a test environment, use sampling to control volume, and instrument only the containers that need it.

Existing collectors and already-injected Pods keep working, because they run independently. What stops is reconciliation and new injection. If the webhook is configured to fail closed for the Pods it covers, new Pod creation may be affected, so plan the operator’s availability and webhook failure behaviour deliberately.

Store collector and instrumentation resources in Git, apply them through a GitOps tool, pin image versions, and supply credentials through Kubernetes Secrets. Review every change like code, and test upgrades in a non-production cluster before rolling them out.

Yes. By default it can watch the whole cluster, or you can limit it to chosen namespaces. Each namespace can have its own collector and Instrumentation resources, which supports team ownership. Decide early whether you want a single shared operator or a more restricted setup, since this affects permissions and how you structure ownership across teams.

A sidecar collector runs inside each application Pod as an extra container, so every application has its own private collector. A daemonset collector runs once per node and serves all Pods on that node. Sidecars give isolation and per-application tuning at a higher resource cost, while daemonsets are lighter and better for node-level data.

Use separate namespaces with separate collector resources, add team identifiers as resource attributes, and apply role-based access control so teams manage only their own resources. At the backend, many platforms support separate tenants or projects, and collectors can route each team’s data to the right one.

The Collector handles logs as well, and the operator manages the collectors that process them, commonly with a daemonset that reads container log files on each node. Auto-instrumentation agents focus mainly on traces and metrics, so log collection usually comes from the node-level collector rather than from injected agents.

Data held only in memory, such as the contents of an in-memory queue, can be lost when a Pod restarts. To reduce this risk, use persistent queues backed by storage, run several replicas, and make sure applications retry sending for a short time. Perfect delivery is rarely free, so decide how much loss is acceptable for each signal.

It is fine for learning and for quick experiments, but many production teams prefer a slim custom distribution that contains only the components they use. A smaller image has fewer parts to patch and a smaller attack surface, and it starts faster. Whichever you choose, pin an exact version.

Begin with a few low-risk services in a test environment, measure startup time and resource use, and compare error rates before and after. Use workload-level annotations at first, set a conservative sampling rate, and widen the rollout in stages. Keep a clear way to remove the annotation and restart if a problem appears.

Learn the Collector first, since it is the core of the pipeline. Then install the operator in a test cluster, create one small collector, and try auto-instrumentation on a sample application. After you can see traces flowing, add the Target Allocator for metrics, and only then think about sidecars, tiers and advanced scaling.

The OpenTelemetry Operator is a Kubernetes operator that turns written descriptions of your telemetry setup into running reality. It manages Collectors in four modes, injects automatic instrumentation into workloads without code changes, distributes Prometheus scraping fairly through the Target Allocator, and keeps everything matching your desired state through continuous reconciliation. Used well, it lets a platform team offer a consistent telemetry service to many teams, with central control over versions, endpoints and sampling.

It is not magic, and it is not free. It is a privileged component with webhooks in the Pod creation path, so it deserves careful security, monitoring, version pinning and tested upgrades. For a very small setup it may be more than you need, but for a growing platform it replaces a large amount of repetitive, error-prone manual work with a clear, reviewable and repeatable process.

If you take one habit from this guide, let it be the discipline of describing the desired state clearly and keeping that description in version control. Everything else, from choosing a deployment mode to rolling out agents in stages, becomes easier when the setup is written down, reviewed by colleagues and applied by automation, because mistakes become visible, reversible and teachable instead of mysterious.