Collecting and storing telemetry gets most of the attention, but none of it exists until the application itself produces it.

The cold-start problem of observability

An observability platform is only as useful as the data that enters it. Before a trace can be stored, searched or sampled, some piece of code inside the application must notice that a request arrived, measure how long it took, and record which database or remote service it called. That act of adding measurement to software is called instrumentation.

In a company with a handful of services, developers can add instrumentation by hand. In a company with hundreds of services, written in several languages by many teams over many years, that approach stalls. Some teams are too busy. Some applications are old and nobody dares touch them. Some are built by outside vendors, and the source code is not even available. The result is patchy visibility, with exactly the blind spots that hurt most during an incident.

What teams usually face

To understand injection, first understand what is being injected and how it works inside a running program.

Manual instrumentation

In manual instrumentation, developers add OpenTelemetry API calls to their own code. They decide where a span begins and ends, what name it gets, and which business details are attached, such as an order type or a customer tier. This gives the richest and most meaningful data, but it costs developer time and must be maintained as code changes.

Automatic instrumentation

In automatic instrumentation, a prebuilt piece of software attaches to the application and adds measurement around well-known libraries and frameworks without any change to the application’s source. When the application handles a web request using a popular framework, the agent notices, starts a span, records standard details and ends the span. The same happens for outgoing web calls, database drivers, message queues and many other common building blocks.

How agents attach, language by language

The trick of automatic instrumentation is that it differs by language, because each runtime offers a different hook for attaching extra behavior.

The common challenge: getting the agent into the container

Whatever the runtime, the agent has to be present inside the container when the application starts, and the runtime has to be told to load it. Traditionally this meant editing the application’s container image to add the agent files and changing its startup command. That is exactly the kind of per-application work that does not scale. This is the gap that injection through the Operator fills: it performs both steps, delivering the agent files and switching on the runtime hook, automatically at pod creation.

flowchart LR
  subgraph Manual["Manual path"]
    M1["Edit application code"] --> M2["Rebuild image"]
    M2 --> M3["Redeploy"]
  end
  subgraph Injected["Injection path"]
    I1["Add one annotation"] --> I2["Operator fits agent at pod start"]
    I2 --> I3["Application runs instrumented"]
  end
        

Injection is built from ordinary Kubernetes features, so a short tour of those features makes everything later easy to follow.

Pods and containers

Kubernetes is a system that runs applications packaged as containers across a cluster of machines. A pod is the smallest unit it schedules. A pod holds one or more containers that share a network address and can share storage. Usually one container is the application, and others are helpers.

Init containers

An init container is a special container that runs to completion before the main containers start. Kubernetes runs init containers one after another, and the application only starts once all of them have finished successfully. They are used for preparation work, such as copying files or waiting for a dependency.

Shared volumes

A volume is a storage area that can be attached to containers in a pod. An empty directory volume starts empty when the pod starts and lives as long as the pod lives. Several containers in the same pod can mount it, so what one container writes, another can read. This is the pass-through shelf between the init container and the application.

Environment variables

An environment variable is a named setting handed to a process when it starts. Runtimes read certain variables to decide how to behave. For example, a Java runtime reads a variable that lists extra startup options, and a Node.js runtime reads one that lists modules to load first. By setting such variables, you can change a runtime’s behavior without touching the program.

Annotations and labels

Every Kubernetes object can carry labels and annotations, which are key and value pairs attached to its metadata. Labels are mainly used for selecting and grouping. Annotations are free-form notes that tools can read to decide what to do. Injection uses annotations as the opt-in signal.

Custom resources and operators

Kubernetes can be extended with new kinds of objects called custom resources. An operator is a program that runs in the cluster, watches certain custom resources and does the work of making reality match what they describe. It encodes the knowledge of a human operator, such as “when someone asks for this, create these things”.

Admission webhooks

Before Kubernetes stores a new object, it passes the request through admission control. A mutating admission webhook is a web service that Kubernetes calls during that step, which may modify the object before it is saved. This is the exact mechanism that lets the Operator change a pod’s definition on the fly, before the pod ever starts.

The Operator is the Kubernetes-native manager of OpenTelemetry components and the engine behind injection.

What it is

The OpenTelemetry Operator is an open-source Kubernetes operator maintained by the OpenTelemetry project. It installs into a cluster as a deployment and registers new custom resource types. It has two main responsibilities that matter to most users. The first is to manage OpenTelemetry Collectors, in modes such as sidecar, node-level agent, standard deployment and stateful set. The second, the subject of this tutorial, is to inject automatic instrumentation into application pods.

The resource types it provides

Inside the Operator

The Operator has a controller, which continuously reconciles the desired state described by custom resources with what actually exists. It also hosts webhooks. One validates and defaults the custom resources when users create them. Another, the pod mutating webhook, is called whenever any pod is about to be created in the cluster and decides whether to add instrumentation. Because that webhook must be reachable and trusted by the cluster’s control plane, the Operator also deals with certificates, typically obtained from a certificate manager or generated by itself, depending on how it is installed.

flowchart TB
  U["Platform engineer"] -->|"Creates"| IR["Instrumentation resource"]
  U -->|"Creates"| CR["Collector resource"]
  subgraph OP["OpenTelemetry Operator"]
    CTRL["Controller"]
    WH["Pod mutating webhook"]
  end
  CR --> CTRL
  CTRL --> COL["Collector pods"]
  IR --> WH
  DEV["Application team"] -->|"Deploys annotated workload"| API["Kubernetes API server"]
  API -->|"Pod creation request"| WH
  WH -->|"Modified pod"| API
        

Now the two halves can be joined: the idea of automatic agents plus the machinery of the Operator.

A precise definition

Auto-instrumentation injection via the Operator is the process by which the OpenTelemetry Operator, at the moment a pod is created, modifies the pod’s definition so that a language-specific instrumentation agent is delivered into the pod and activated, based on a shared configuration resource and an opt-in annotation on the workload. The application’s source code and container image are not changed. Only the running pod is.

Three words in that definition deserve attention. Shared configuration means the settings live in one Instrumentation resource that many workloads reuse. Opt-in annotation means nothing happens unless a workload asks. Moment a pod is created means injection is a start-time event: running pods are not touched, which has big consequences for rollouts later.

The Instrumentation resource, piece by piece

The Instrumentation resource is the house card. It is a namespaced Kubernetes object, and it describes how injection should behave for pods that refer to it. Its main parts are as follows.

One resource, many workloads

A key benefit is reuse. A platform team can create one Instrumentation resource per namespace, or one per environment, and every annotated workload that refers to it receives identical, vetted settings. Changing a sampling rate or an endpoint becomes a single edit rather than hundreds of code changes.

The annotation is the only thing an application team has to know, so it must be understood exactly.

One annotation per language

The opt-in signal is an annotation whose name includes the language to inject. The annotation names share a common prefix and end with the language, for example one for Java, one for Node.js, one for Python, one for .NET, one for Go, one for Apache HTTP Server and one for Nginx. There is also a variant that injects only the configuration for the SDK, with no agent, which is useful when an application already contains OpenTelemetry libraries and just needs consistent settings.

What the value can say

Where the annotation must be placed

This is the single most common beginner mistake. Pods are usually created by higher-level objects such as deployments, which hold a template for their pods. The annotation must be placed on the pod template inside the deployment, not on the deployment object itself. The webhook only ever sees the pod being created, so an annotation placed on the outer object is invisible to it.

Namespace-level opt-in

An annotation can also be placed on a namespace, in which case it applies to pods created in that namespace. This is attractive for platforms, because a team can say “everything in this namespace is a Java service, instrument it”. The trade-off is less precision, since every pod in the namespace is affected, including helpers that are not written in that language. Use it only where the namespace is genuinely uniform.

Pods with several containers

When a pod has more than one container, the Operator by default instruments the first container. A separate annotation lets you list which containers should be instrumented, and these can be of different languages if language-specific settings are provided accordingly. Sidecars such as service-mesh proxies are usually not meant to be instrumented, so naming the target container explicitly avoids mistakes.

Extra annotations for special cases

Following one pod from creation to first trace makes the whole mechanism concrete.

The journey of an annotated pod

sequenceDiagram
  participant D as Deployment controller
  participant A as API server
  participant W as Operator webhook
  participant N as Node
  participant I as Init container
  participant P as Application container
  participant C as Collector
  D->>A: Create pod with opt-in annotation
  A->>W: Admission review for the pod
  W->>W: Read annotation and Instrumentation resource
  W-->>A: Patch adds init container, volume and env vars
  A->>N: Schedule the modified pod
  N->>I: Run init container
  I->>I: Copy agent files to shared volume
  I-->>N: Exit successfully
  N->>P: Start application with agent settings
  P->>C: Send spans and metrics over OTLP
        

What changes inside the pod

It helps to compare the pod before and after. Before mutation it has the application container and maybe nothing else. After mutation it gains an init container, a shared volume with mounts, and extra environment variables. The application’s image, command and code are untouched, which is why the approach works for software you cannot rebuild.

flowchart LR
  subgraph Before["Pod as submitted"]
    B1["Application container"]
  end
  subgraph After["Pod after mutation"]
    A1["Init container with agent image"]
    A2["Shared empty directory volume"]
    A3["Application container with agent settings"]
    A1 -->|"Copies agent files"| A2
    A2 -->|"Mounted read by"| A3
  end
  Before -->|"Webhook rewrites"| After
        

The injected pieces look similar from outside, but each language switches its agent on in a different way.

Java: the most mature path

Java has the longest history and the richest coverage of frameworks and libraries. The injected agent is loaded by the Java runtime through its standard startup-options variable. The Operator adds its own option to whatever the container already specifies, so existing tuning is normally preserved. Teams should still measure startup time and memory, because bytecode rewriting is not free.

Node.js and Python: lightweight attachment

These runtimes load instrumentation as ordinary modules at startup. The init container supplies those modules and the environment variables arrange for them to run first. Compatibility with specific framework versions matters, so teams should check the published list of supported libraries rather than assuming everything is covered.

.NET: profiler-based

The .NET path uses a profiler component that the runtime loads when special settings are present. Those settings include paths into the shared volume. Operating-system flavor matters, as there are separate builds for common Linux variants, so match the agent to the base image your applications use.

Go: a different world

Because Go programs are compiled into a single executable with little room for runtime hooks, the Operator’s Go support uses an external observer based on kernel-level tracing technology. The Operator adds an extra container that shares the process space with the application and needs elevated permissions. That is a significant security consideration, and the feature is generally considered less mature than the others. Many organizations use manual instrumentation for Go services, or adopt this feature only in controlled environments after careful review.

Web servers: instrumenting the front door

The Apache and Nginx options place a module into the web server so that it records requests entering and leaving. They are valuable for gaining visibility at the edge of a system or in a gateway tier. Remember that they instrument the server itself and not the applications behind it.

Besides loading the agent, the Operator fills in the settings that normally cause the most mistakes.

Identity: who produced this data

The Operator derives a service name from pod metadata, using details such as the owning workload’s name, so that data is attributable without the application declaring it. It also adds Kubernetes context as resource attributes: the namespace, the pod name, the node, the container and information about the owning workload. These attributes are what let a trace be traced back to a specific deployment and pod during an incident.

Destination and protocol

The endpoint from the Instrumentation resource is injected as an environment variable that the agent reads. Each language’s agent uses either a binary streaming style or an HTTP style of the OpenTelemetry Protocol, and the default port and protocol can differ by language and by Operator version. Platform teams must make sure the chosen endpoint listens on the protocol and port that the injected agents will use.

Context propagation and sampling

Propagators decide how trace context is carried in request headers between services. Setting them centrally means every instrumented service speaks the same convention, so traces stitch together across teams. The sampler rule, usually a percentage that honors the upstream caller’s decision, is also applied uniformly, which gives predictable data volume.

Who wins when settings collide

Settings can come from several places: the Instrumentation resource, environment variables already present in the application container, pod-level annotations and the agent’s own defaults. In general, a variable explicitly defined in the application container tends to take precedence over one the Operator would add, and language-specific values override common ones. Because exact precedence rules can vary between versions and settings, test the combination you actually use before relying on an override.

Injection decides how data is produced. A Collector pipeline decides what happens to it next, and the two must fit together.

Where should the agents send data?

Injected agents need a destination that is stable, nearby and fast. Three common choices exist, each matching a deployment shape for Collectors.

flowchart LR
  subgraph Pod["Application pod"]
    APP["Application with injected agent"]
  end
  APP -->|"OTLP"| AGENT["Node-level Collector"]
  AGENT -->|"OTLP"| GW["Gateway Collector pool"]
  GW --> TR["Trace store"]
  GW --> ME["Metrics store"]
  GW --> LO["Log store"]
  IR["Instrumentation resource"] -.->|"Supplies endpoint setting"| APP
        

Keeping the address stable

The endpoint in the Instrumentation resource is effectively a contract. If a platform team re-shapes its Collector tiers, the address that applications use should not change, otherwise every pod must be recreated to pick up the new value. Using a stable service name, or a node-local address, protects application teams from internal changes.

Logs and metrics from injected workloads

Auto-instrumentation chiefly produces traces and, for many languages, runtime and framework metrics. Logs are usually collected separately, by reading container output through a node-level Collector. A well-designed platform makes the three line up by giving them the same Kubernetes resource attributes, so that an engineer can jump from a slow trace to the metrics and logs of the very same pod.

Injection depends on cluster-level plumbing, so a handful of prerequisites decide whether it works at all.

Webhook failure behavior

Every webhook has a policy for what Kubernetes should do if the webhook cannot be reached. If the policy blocks pod creation on failure, an unavailable Operator could stop applications from starting. If it ignores failure, pods start but are silently uninstrumented. Understand which behavior your installation uses and decide deliberately, because it trades safety of deployments against completeness of telemetry.

Because injection happens at pod creation, every change travels through restarts, and restarts need planning.

Turning it on for a workload

Adding the annotation to the pod template triggers the normal rolling update of the deployment, since a template change means new pods. As each new pod is created, it is instrumented. The old pods continue to run, uninstrumented, until they are replaced. No special action is needed beyond the usual rollout, but the change should still be treated as a deployment with its own risk.

Changing the shared settings

Changing an Instrumentation resource affects only pods created afterward. To apply a new endpoint or sampling rule across a fleet, workloads must be restarted gradually. Platform teams therefore schedule fleet-wide restarts in controlled waves rather than all at once, watching error rates and resource use as each wave completes.

Upgrading agent versions

Agent versions are chosen by the image reference in the Instrumentation resource. A new agent can change behavior: new spans, renamed attributes, different overhead. Treat agent upgrades like application releases. Test them in a staging environment with realistic traffic, then roll out by environment or by team, and keep the ability to return to the previous image.

flowchart LR
  A["Propose agent or setting change"] --> B["Test in staging with real traffic"]
  B --> C["Canary on a few workloads"]
  C --> D{"Healthy?"}
  D -->|"Yes"| E["Restart in waves"]
  D -->|"No"| F["Revert image or resource"]
  E --> G["Monitor drops, errors and overhead"]
  F --> A
        

Opting out and emergency stops

Provide a simple way to switch injection off for a workload that misbehaves: setting the annotation value to false, or removing it, then restarting. For emergencies, the platform team can also disable a language or scale down the Operator’s webhook, remembering the failure-policy consequences from the previous chapter. Document these steps in a runbook so nobody discovers them during an outage.

Injection becomes a true platform feature when onboarding is nearly effortless and guardrails are automatic.

The golden path

A golden path is the easiest, best-supported route to do something. For injection, the golden path might be a starter template for new services that already carries the right annotation, or a deployment package that adds it based on a single option such as the service language. When the easy route is also the correct route, adoption follows without pressure.

Automating the annotation

Policy engines for Kubernetes can add annotations to pods or workloads automatically, according to rules such as namespace, label or team. This lets a platform team enable injection for a whole class of workloads without asking each team to edit manifests. The usual caution applies: automatic enablement should be gradual, announced and reversible.

Cost awareness

Auto-instrumentation can dramatically increase data volume overnight, since every service starts producing spans for every request. Combine injection with sensible default sampling and with filtering in the Collector tier, and show each team its volume. Surprise bills are the fastest way to lose organizational support for a platform.

Injection modifies running workloads automatically, so it deserves the same scrutiny as any component with that much power.

The Operator is a privileged actor

The Operator’s webhook can alter any pod it is allowed to see. It therefore sits in a position of high trust. Limit who can modify the Operator’s deployment and its custom resources, restrict which namespaces it acts on where possible, and keep its version current. A compromised webhook could add harmful containers to every new pod in the cluster, so it deserves protection comparable to other cluster-wide controllers.

Supply chain of agent images

The init container runs an image, and that image’s contents execute inside your application’s pod. Treat agent images as software dependencies: mirror them to a registry you control, scan them for known vulnerabilities, pin them by exact version or digest, and update them on a schedule. Never let the Instrumentation resource point at an unreviewed image.

Permissions and pod security

For most languages, the injected init container needs only ordinary permissions and write access to a shared volume. Clusters with strict pod security rules should confirm that the injected additions, including the extra volume and container, comply with those rules. The Go option is the notable exception because it needs elevated permissions and shared process visibility, so it should be allowed only where those privileges are explicitly accepted.

What the agent may capture

Agents record details such as web addresses, headers and, for some database libraries, parts of queries. Some of that may contain sensitive values. Review the defaults for each language and agent version, disable capturing of sensitive headers or statements where needed, and rely on central masking in the Collector pipeline as a second line of defense. Compliance questions about what leaves a workload are easier to answer when settings are controlled in one shared resource.

Audit and change control

Keep Instrumentation resources, Operator configuration and policies in version control, require review for changes and record who changed what. When auditors ask how a service came to emit certain data, the answer should be a short walk through a repository history rather than a search through people’s memories.

Automatic instrumentation is free of developer effort, not free of cost, and the cost should be measured rather than assumed.

Where the overhead appears

Sizing the init container

The init container runs briefly, but Kubernetes still counts its resource requests when scheduling the pod. The Instrumentation resource lets you set resource requests and limits for the init container per language. Keep them modest, yet not so tight that copying fails or is slow on busy nodes.

Measure before and after

Compare latency percentiles, error rates, memory and CPU for the same workload before and after injection, at realistic load. Do this in a staging environment first, and again for a canary in production. Where overhead is too high, use sampling to reduce exported volume, disable instrumentation for noisy libraries where the agent supports that, or consider manual instrumentation of only the most important paths.

Injection is powerful but not magic. A clear view of both sides prevents disappointment.

Automatic versus manual, side by side

The best answer is usually both

Most mature organizations use injection to get broad, uniform baseline coverage quickly, then add manual instrumentation for the handful of business operations where generic spans are not meaningful enough. The two work together, since manual spans created through the OpenTelemetry API attach to the same traces that the agent creates.

The same lessons come up in almost every adoption; here they are in one place.

Patterns that work

Anti-patterns to avoid

When injection does not work, the cause is almost always one of a short list. Work from the outside in.

flowchart TD
  S["Pod has no agent or no telemetry"] --> Q1{"Init container and env vars present?"}
  Q1 -->|"No"| Q2{"Annotation on pod template?"}
  Q2 -->|"No"| F1["Move annotation to pod template and restart"]
  Q2 -->|"Yes"| Q3{"Language enabled and Instrumentation found?"}
  Q3 -->|"No"| F2["Enable language, fix name or namespace reference"]
  Q3 -->|"Yes"| F3["Check webhook reachability, certificates and logs"]
  Q1 -->|"Yes"| Q4{"Application starts normally?"}
  Q4 -->|"No"| F4["Check memory limits, conflicts and agent compatibility"]
  Q4 -->|"Yes"| Q5{"Collector receives data?"}
  Q5 -->|"No"| F5["Check endpoint, protocol, port and network policy"]
  Q5 -->|"Yes"| F6["Check sampling, filters and backend routing"]
        

Symptom and cause table

These are composite scenarios drawn from common patterns, shown to illustrate how the ideas combine in practice.

A staged path keeps each step useful, and a short list of habits keeps the whole effort calm.

flowchart LR
  A["Stage 1 Prove"] --> B["Stage 2 Wire to pipeline"]
  B --> C["Stage 3 Pilot"]
  C --> D["Stage 4 Harden"]
  D --> E["Stage 5 Scale"]
  E --> F["Stage 6 Refine"]
        

Quick answers to the questions that come up first.

No. The Operator modifies the pod definition at creation time, adding an init container, a shared volume and environment variables. The application’s image and source stay exactly as they were.

The most common reasons are that the annotation is on the deployment instead of the pod template, the language is not enabled in the Operator, the referenced Instrumentation resource cannot be found, or the webhook is unreachable. Existing pods also do not change until they are recreated.

No. Injection happens at pod creation only. Restart the workload, ideally through a controlled rolling restart, so that new pods pick up the new settings.

Yes. By default the first container is targeted, and a separate annotation lets you name the containers to instrument. This is useful for pods that include proxies or helper containers.

Agent injection delivers instrumentation files and activates them. SDK-only injection just sets the shared configuration values, such as endpoint and sampling, for applications that already include OpenTelemetry libraries in their own code.

It adds some startup time, memory and CPU overhead, usually modest but real. Measure it in staging and in a canary, adjust memory limits and sampling, and decide based on evidence rather than assumption.

No. Go uses an external, kernel-level approach that needs elevated permissions and a target executable path, and it is generally considered less mature. Many organizations use manual instrumentation for Go or limit the feature to controlled environments.

It depends on the webhook’s failure policy. Pod creation may be blocked, or pods may start without instrumentation. Know which your installation uses, and monitor the Operator so that a failure is noticed quickly.

Not safely on the same workload. Running two instrumentation sources can create duplicate data and conflicts. Choose one approach per workload, or use SDK-only settings if the application already contains OpenTelemetry libraries.

Injection produces telemetry inside workloads and sends it to an endpoint defined in the Instrumentation resource. A Collector pipeline then enriches, filters, samples and routes that data. They are complementary halves of an observability platform.

In the official OpenTelemetry Operator documentation and release notes. Names, defaults and maturity levels change between versions, so always confirm them for the version you run.

Auto-instrumentation injection via the Operator turns observability from a per-application project into a platform capability. A mutating webhook rewrites each annotated pod as it is created, an init container delivers the language agent, shared settings from one Instrumentation resource tell it where to send data, and the application starts producing traces without a single code change. Like the mechanic at the hangar door in the opening picture, the Operator fits the same standard recorder to every aircraft that rolls through, without disturbing the people who built them.

The approach is strongest when it is treated as a managed service: gradual rollouts, pinned images, sized resources, monitored webhooks and clear ownership. It does not replace thoughtful manual instrumentation, but it gives every team a broad, consistent baseline on day one and frees developers to spend their effort where business meaning lives.