What Is Auto-Instrumentation in OpenTelemetry?
A friendly, ground-up guide to getting traces, metrics and logs out of your application without rewriting it: what it is, how it works behind the scenes, what you get, what you do not get, and how to use it safely.
Imagine you move into a new house. You could install every smoke detector, door sensor and light switch by hand, drilling holes in every wall. That would take weeks. Or you could hire a company that arrives with a ready-made kit, fits sensors on every door and window in one afternoon, and connects them all to one control panel. You would not get a perfect setup tailored to your family, but you would be safe and informed by the evening.
Auto-instrumentation in OpenTelemetry is that ready-made kit. It attaches to your running application and starts producing telemetry, which is information your software sends out about itself, with little or no change to your code. In this guide, we will learn what it is, how it works, and when to use it, with simple words and everyday examples.
1The Problem: Adding Telemetry by Hand Is Slow
Before learning the tool, let us feel the problem it solves.
What does it mean to instrument?
Instrumentation means adding the pieces that make a program report on itself. For tracing, it means creating a record, called a span, every time something notable happens: a web request arrives, a database is queried, a message is sent. Each span records what happened, when it started and how long it took. Together, spans form a trace, which is the story of one request across your whole system.
The cost of doing it all by hand
Suppose your company has two hundred services. Each one uses a web framework, a database driver, an HTTP client and a message queue library. To trace them by hand, a developer must edit every service, find every place where these libraries are used, and wrap each one with tracing code. They must also remember to pass the trace identity along in every outgoing call. This is slow, easy to get wrong, and it never ends, because new code is written every day.
Worse, you cannot edit code you do not own. The web framework, the database driver and the cloud client libraries were written by other people. You cannot easily add tracing inside them, yet that is where much of the time is spent.
Think of a school wanting to know which students arrive late. One option is to ask every teacher to write down arrival times by hand. Another is to install an electronic gate at the main entrance that automatically logs every student who walks through. The gate does not know the students’ stories, but it records the key facts for everyone, with no effort from the teachers. Auto-instrumentation is the electronic gate.
2So, What Is Auto-Instrumentation?
The short answer first, and then the details.
Auto-instrumentation is a way of producing OpenTelemetry telemetry from an application by attaching a ready-made component to it, usually called an agent or an auto-instrumentation package, instead of writing tracing code yourself. The component watches for well-known libraries, such as web frameworks, database drivers and HTTP clients, and automatically creates spans, metrics and, in some cases, logs whenever those libraries are used.
The OpenTelemetry documentation often calls this zero-code instrumentation, because in the best case you do not touch your code at all. You change how the application is started, add a few settings, and the telemetry begins to flow.
What it automatically does for you
Creates Spans
For incoming web requests, outgoing calls, database queries, messages and more, with sensible names and details.
Passes the Trace Identity
Writes the trace headers into outgoing calls and reads them from incoming ones, so traces stay connected across services.
Produces Metrics
Request counts, durations and, in many languages, runtime facts such as memory and thread usage.
Adds Standard Attributes
Follows OpenTelemetry’s naming rules, called semantic conventions, so data from different services looks the same.
Bundles the SDK
Many agents include the SDK, the exporter and sensible defaults, so you do not need to assemble them yourself.
Connects Logs
Often links your existing logging library so that log lines carry the trace and span IDs.
3Three Levels: Zero-Code, Library-Based and Manual
Auto-instrumentation is one of three ways to instrument. Knowing the other two avoids confusion.
| Level | What You Do | Effort | Control |
|---|---|---|---|
| Zero-code (auto) | Attach an agent or wrapper when starting the application | Very low | Low to medium, mostly through settings |
| Library-based | Add instrumentation packages for specific libraries and switch them on in a small amount of setup code | Medium | Medium |
| Manual (code-based) | Use the OpenTelemetry API directly to create your own spans, metrics and events | Highest | Full |
Think of getting around a new city. Zero-code is taking a guided bus tour: you see all the main sights with no planning. Library-based is renting a bike and choosing some stops yourself. Manual is walking with your own map to reach exactly the small cafe you want. Most visitors use more than one.
These levels are not rivals. A very common and healthy approach is to start with auto-instrumentation to get broad coverage in a day, and then add manual spans and attributes in the few places where business meaning matters, such as “apply discount” or “approve loan.”
An instrumentation library is a package that knows how to trace one specific library, such as one particular web framework. Auto-instrumentation agents are essentially large bundles of these, switched on automatically. This is why the two ideas are closely related.
4Internal Working: How Can It Work Without Changing My Code?
This sounds like magic, but it is a clever and well-known set of tricks.
The key idea is that a running program can be changed slightly as it loads, or its libraries can be wrapped at the moment they are brought in. The agent gets in early, before your application does its work, and quietly adds tracing around the libraries you use. Each language has its own way of doing this.
flowchart TD
A["Application starts with agent attached"] --> B["Agent loads first and registers its hooks"]
B --> C["Application loads a library such as a web framework"]
C --> D{"Does the agent know this library?"}
D -->|Yes| E["Agent wraps key methods with tracing logic"]
D -->|No| F["Library runs unchanged and nothing is traced"]
E --> G["Application code runs normally"]
G --> H["Wrapped method runs and creates a span through the API"]
H --> I["SDK processes the span and exporter sends it out"]
Different languages, different tricks
Java Agent and Bytecode
A single agent file is attached when the program starts. As each class loads, the agent can rewrite its compiled form so that key methods are wrapped with tracing. Your source code is untouched.
Wrapper Command and Patching
A helper command starts your program and swaps in traced versions of library functions at run time. A companion tool can detect which libraries you use and suggest matching packages.
Load-Time Hooks
A setup file is loaded before the application. It intercepts the loading of known modules and wraps them with tracing logic. A ready-made bundle covers many popular libraries.
Profiler and Startup Hooks
Environment settings tell the .NET runtime to load a profiler that modifies code as it loads. Many .NET libraries also already emit tracing data of their own, which OpenTelemetry simply listens to.
A Different Story
Go programs are compiled ahead of time, so run-time tricks are harder. Options include watching the program from outside using a kernel technology called eBPF, special build steps, or lightweight wrapper libraries.
Ruby, PHP and More
Other languages offer their own packages or extensions that hook into frameworks and drivers. The idea is similar, though the maturity and coverage vary.
Think of a hotel where a cleaner enters each room before the guest arrives and places a small, quiet sensor behind the door, the window and the minibar. The guest, which is your application, never notices. When the guest uses the door, the sensor records it. The cleaner is the agent, and the sensors are the wrapped library methods.
The agent must be attached before your application loads its libraries. If it starts too late, the libraries are already loaded and unwrapped, and nothing will be traced. “Nothing shows up” is very often a start-order problem.
5What You Get: A Tour of the Output
Let us follow one web request through an auto-instrumented service to see exactly what appears.
sequenceDiagram
participant U as Customer
participant S as Orders Service with Agent
participant D as Database
participant P as Payment Service with Agent
participant C as Collector
U->>S: HTTP request to place order
S->>S: Agent starts a server span
S->>D: Database query
S->>S: Agent records a client span for the query
S->>P: HTTP call with trace headers added by agent
P->>P: Agent reads headers and starts a child server span
P-->>S: Response
S-->>U: Response
S->>C: Export spans and metrics
P->>C: Export spans and metrics
The typical catch
Incoming Requests
A span for every web request received, with the method, route and status code.
Outgoing Calls
A span for every call made to another service, so you can see how long each dependency took.
Database Calls
A span for each query, often with the type of database and a cleaned-up version of the statement.
Queues and Streams
Spans for sending and receiving messages, with the trace identity carried in message headers.
Runtime Metrics
Facts about the program itself, such as memory use, garbage collection and thread counts, in languages that support it.
Failures and Exceptions
Failed calls are marked, and exceptions are often recorded as events on the span.
Production Example: Instant Dependency Map
A platform team attaches the same agent to fifty existing services in one week. Without a single code change, their tracing tool draws a full map of which service calls which, and highlights a slow database as the common cause of checkout delays. The map would have taken months to build by hand.
What you do not get
Auto-instrumentation knows about libraries, but not about your business. It can tell you that “a database call took 900 milliseconds.” It cannot tell you that “this was the loyalty discount calculation for a gold-tier customer.” That kind of meaning must be added by you, using the manual approach, which we will see in a later chapter.
6Configuration: Settings Instead of Code
Because you are not writing code, almost everything is controlled by settings, usually environment variables.
An environment variable is a named setting that is given to a program when it starts, like a note stuck on the door before it enters the room. OpenTelemetry defines a standard set of these names, so the same ideas work across languages and agents.
| Setting | What It Controls | Why It Matters |
|---|---|---|
OTEL_SERVICE_NAME | The name of your service | Without it, data may appear as “unknown service” |
OTEL_RESOURCE_ATTRIBUTES | Extra labels, such as environment and version | Lets you filter by production or testing, and by release |
OTEL_EXPORTER_OTLP_ENDPOINT | Where to send the data | Usually the address of your Collector |
OTEL_TRACES_SAMPLER | How many traces to keep | Controls cost on busy services |
OTEL_PROPAGATORS | Which header formats to use between services | Needed to keep traces connected through older systems |
| Enable or disable switches | Turn individual library instrumentations on or off | Removes noise or avoids problems with a specific library |
Exact setting names for enabling or disabling a specific library differ between languages and versions. Always confirm them in the documentation for the agent you are using, rather than guessing from another language.
Why settings are a strength
Settings can be changed by an operations team without touching the application. They can be managed in one place, such as a deployment template, and applied to many services at once. This is the reason why large companies like auto-instrumentation: policy lives with the platform, not with every developer.
7Deployment: From One Laptop to a Thousand Pods
How the agent actually gets attached in real environments.
On a single machine or in a container
For a simple setup, you change the start command or the container’s startup settings so that the agent is loaded. In a container image, you place the agent file or package inside the image, and set the environment variables in the deployment settings.
On Kubernetes with the OpenTelemetry Operator
Kubernetes is a popular system for running many containers. The OpenTelemetry Operator is an add-on for Kubernetes that can inject auto-instrumentation into your applications for you. You mark a workload with a simple annotation, and the Operator does the rest. It supports several languages, such as Java, Node.js, Python and .NET, and some others in different ways.
flowchart TD
A["Team adds an annotation to a workload"] --> B["Kubernetes creates a new pod"]
B --> C["Operator webhook intercepts pod creation"]
C --> D["Operator adds an init container holding the agent"]
D --> E["Operator adds environment variables and shared volume"]
E --> F["Application container starts with agent attached"]
F --> G["Telemetry is sent to the Collector"]
G --> H["Collector forwards to the tracing and metrics backends"]
Imagine a security guard at the door of a building who hands every visitor a small tracking badge as they enter. The visitors do not have to bring their own badge or know how it works. The Operator is that guard, and the annotation is the sign on the visitor’s invitation that says “please give this person a badge.”
Other places it runs
- Serverless functions: cloud providers offer ready-made layers that add OpenTelemetry instrumentation to functions without code changes. For example, AWS provides managed layers through its OpenTelemetry distribution.
- Web servers and proxies: there are modules for servers such as Apache HTTP Server and Nginx, which trace requests at the front door.
- Virtual machines: use startup scripts or system service settings to attach the agent and set the environment variables.
- Vendor distributions: many monitoring vendors and cloud providers ship their own builds of the agents, with their defaults pre-set. These are still based on OpenTelemetry.
8Compiled Languages and eBPF: The Go Question
Not every language can be patched while it runs.
Languages like Java and Python run inside a managed environment that allows code to be wrapped as it loads. Go, Rust and C++ are compiled straight into machine code, so there is no runtime layer to hook into. This makes zero-code instrumentation harder, and the solutions are different.
eBPF-Based Approach
- Watches the program from outside, using a feature of the Linux kernel
- Needs no code change or recompilation
- Can work across many languages at once
- But needs a supported Linux system and special permissions
- Has less detail and weaker context passing than in-process methods in some cases
Library-Based Approach
- You add wrapper libraries around popular frameworks, such as web servers and clients
- Gives rich, accurate detail and full context passing
- Works anywhere, including non-Linux systems
- But needs small code changes and a rebuild
- Coverage depends on which libraries you wrap
Tools in this area are developing quickly, including work inside the OpenTelemetry project itself. Check the current status of the options for your language before deciding. For Go, many teams today choose the library-based approach for production, and use eBPF for quick visibility.
9Combining Auto and Manual Instrumentation
The best results come from using both, each for what it does best.
flowchart LR
subgraph APP["One Application Process"]
AUTO["Auto-instrumentation: web, database and client spans"]
MAN["Manual code: business spans, events and attributes"]
API["OpenTelemetry API"]
SDK["SDK provided by the agent"]
end
AUTO --> API
MAN --> API
API --> SDK
SDK --> OUT["Exporter and Collector"]
A sensible way to combine them
Start With Auto
Attach the agent and get the full skeleton of requests, calls and queries.
Find the Gaps
Look at real traces and ask, “What would I want to know at three in the morning that is not here?”
Add Attributes to Existing Spans
Use the API to fetch the span that auto-instrumentation already created, and add business details like the customer tier or the order type.
Add Spans for Key Business Steps
Create manual spans for important operations that the agent cannot know about, such as “calculate price.”
Keep Manual Work Small
Resist the temptation to wrap everything. A few meaningful additions are worth more than hundreds of noisy ones.
When an agent already provides the SDK, your application code should usually depend only on the lightweight API, not on a second copy of the SDK. Running two SDKs at once can cause duplicate data, conflicts or confusing behaviour. Check your agent’s guidance for how to combine them in your language.
Production Example: Auto Plus a Dash of Manual
A travel booking company attaches an agent to its search service. Traces show the web request, a cache call and three calls to airline systems. The team then adds just two things by hand: an attribute for the number of passengers on the main span, and a custom span around the fare-ranking step. Now engineers can answer “why are group bookings slow?” with one query, using a mix of automatic and manual data.
10Performance, Noise and Cost
An agent is easy to attach, but it is not free.
Where the cost comes from
- Startup time: agents that rewrite code as it loads, such as the Java agent, can make a program start more slowly, sometimes by several seconds. This matters for short-lived jobs and serverless functions.
- Memory and CPU: creating, storing and exporting spans uses some resources. The cost grows with the amount of traffic and the number of spans.
- Data volume: automatic coverage can produce a lot of spans, especially on services with many database calls. Storage and network costs follow.
- Noise: health checks, internal timers and chatty libraries may flood your traces with spans nobody needs.
How to keep it under control
Keep a Share of Traces
Use sampling settings to keep only a portion of traces on busy services, and tail sampling in a Collector to keep the interesting ones.
Drop the Noise
Exclude health-check endpoints and other useless spans, either with agent settings or with a Collector filter.
Turn Off What You Do Not Need
Disable instrumentations for libraries that create noisy or low-value spans.
Send Data in Groups
The SDK’s batch processor groups data to keep network use small. Make sure it is enabled, which is usually the default.
Measure Before You Roll Out
Run a load test with and without the agent in a test environment, so you know the real overhead for your service.
Go Gradually
Start with a few services, watch their health, and then widen. Do not switch on everything on the busiest day of the year.
Production Example: The Health Check Flood
After enabling an agent on a busy service, a team noticed that almost half of all stored spans were from a load balancer calling the health endpoint every second. A single filtering rule removed them. Their trace storage cost dropped immediately, and the real traffic became much easier to read.
11Advantages, Disadvantages and Trade-offs
No approach is perfect. Here is an honest look at both sides.
Advantages
- Fast to adopt: useful data in hours, not months
- No code changes: developers keep working normally
- Wide coverage: traces libraries you do not own
- Consistent: every service follows the same naming rules
- Central control: platform teams change behaviour through settings
- Automatic context passing: traces stay connected across services
Disadvantages
- Limited meaning: knows libraries, not your business
- Can be noisy: may create more data than you need
- Overhead: slower startup and extra resource use
- Coverage gaps: unsupported or very new library versions may not be traced
- Less transparent: harder to understand what it is doing inside your process
- Version coupling: upgrades can change span names and attributes
Key trade-offs
| Decision | Option A | Option B |
|---|---|---|
| Approach | Auto only, for speed and low effort | Auto plus manual, for business meaning |
| Scope of instrumentation | Everything on by default, for maximum visibility | Only chosen libraries on, for lower cost and less noise |
| Where to attach | In the container image, simple and self-contained | Through the Operator, central but with more moving parts |
| Upgrade policy | Always latest, for fixes and coverage | Pinned versions, for stable dashboards |
| Data collected | Capture more details for easier debugging | Capture less to protect privacy and cost |
12Security and Privacy Considerations
An agent sees a lot. That is its job, and also its risk.
It Runs Inside Your Process
An agent has deep access to your application. Use only official or trusted builds, and verify where they come from.
Pin and Review Versions
Pin agent versions, scan them like any other dependency, and review release notes before upgrading.
Query Strings Can Hold Secrets
Web addresses sometimes contain tokens or personal details. Check what your agent records, and remove sensitive parts.
Statements May Include Values
Many agents replace the actual values in queries with placeholders by default, but confirm this for your setup, since real values could expose personal data.
Capture Headers Carefully
Capturing request headers can record cookies and authorisation tokens. Only capture a short, approved list of safe headers.
Scrub in the Collector
Use Collector processors to remove or mask sensitive attributes as a second line of defence.
Because auto-instrumentation records so much by default, you may collect personal data without realising it. Review a sample of real traces from a test environment before enabling production, and check them against your privacy rules, such as GDPR.
13Monitoring, Logging and Troubleshooting
When nothing shows up, or too much shows up, here is how to find out why.
Common reasons for “I see no data”
- The agent was not actually attached. Check that the start command or startup setting is really in use in the running environment.
- It started too late. If libraries loaded before the agent, they were not wrapped.
- Wrong destination. The endpoint setting may point to the wrong address, port or protocol.
- Network blocked. A firewall or security rule may prevent the application from reaching the Collector.
- Unsupported library or version. Your framework may not be covered by the agent.
- Everything sampled out. A sampling setting may be dropping all traces.
- Missing service name. Data may be arriving, but under an unexpected name.
Things to watch once it is running
- Startup time and memory: compare before and after attaching the agent.
- Dropped data: watch for dropped spans from full queues, a sign that your export path cannot keep up.
- Span volume per service: a sudden jump after a deployment suggests a changed instrumentation or a new noisy library.
- Broken traces: traces made of one or two spans suggest that context is not passing through some hop.
- Agent logs: most agents have a debug mode that prints what they are doing. Turn it on briefly in a test environment, and then off again.
- Silent services: alert when a service that normally sends telemetry suddenly sends none.
Production Example: The Upgrade That Renamed Everything
A team upgraded their agent to a newer version. Overnight, their dashboards went blank, because the new version used updated standard names for web request attributes. The data was fine, but the dashboards were still searching for the old names. A careful reading of the release notes and a short period of sending both old and new names would have avoided the surprise.
14Design Patterns and Anti-Patterns
Habits worth copying and traps worth avoiding.
Good design patterns
Platform Default
A platform team provides a standard way to attach the agent, so every new service starts with telemetry already on.
Sidecar Collector
Send agent data to a nearby Collector, which then handles batching, filtering and routing, keeping the application light.
Auto for Breadth, Manual for Depth
Use agents for coverage everywhere, and manual spans only for the most valuable business steps.
Config as Code
Keep agent settings in version control with reviews, since small changes can alter large amounts of data.
Canary Rollout
Attach the agent to a small share of instances first and compare health with the rest.
Resource Standards
Agree on service names, environments and versions across all teams, set through environment variables.
Anti-patterns
The Mistake
Switching on the agent everywhere at once, with all defaults, on a busy production day.
Why It Hurts
Unknown overhead, noisy data and surprise costs arrive together, and it is hard to tell what caused any problem.
Better Approach
Roll out in stages, measure overhead, and tune sampling and filters before widening the rollout.
The Mistake
Treating auto-instrumentation as the finish line, with no manual additions at all.
Why It Hurts
You get a technical picture but no business meaning. Engineers cannot answer questions about customers, orders or features.
Better Approach
Add a small number of business attributes and spans to the most important flows.
The Mistake
Running an agent and a hand-built SDK setup in the same application without a plan.
Why It Hurts
You may get duplicate spans, clashing settings or two exporters sending the same data.
Better Approach
Choose one source of the SDK. Let the agent provide it, and keep application code limited to the API.
The Mistake
Upgrading agent versions automatically without testing.
Why It Hurts
Span names, attributes and behaviour can change, silently breaking dashboards and alerts.
Better Approach
Pin versions, read release notes, test in a staging environment and upgrade deliberately.
The Mistake
Capturing full request headers, bodies or query strings just in case.
Why It Hurts
It can leak passwords, tokens and personal data into your monitoring system and break privacy rules.
Better Approach
Capture only a short, approved list of safe fields, and scrub the rest in the Collector.
15Best Practices and Common Mistakes
A practical checklist for a healthy rollout.
Best practices
- Set a clear service name. Always set the service name, environment and version through settings.
- Send to a Collector. It gives you filtering, redaction, batching and the freedom to change backends later.
- Start small. Try one non-critical service first, then grow.
- Measure the overhead. Compare startup time, memory, CPU and latency before and after.
- Filter noise early. Remove health checks and other low-value spans on day one.
- Review real traces. Look at sample traces for sensitive data and missing details before going wide.
- Pin and test versions. Keep agent versions fixed, and upgrade on purpose.
- Add business meaning. Use attributes and a few spans to make traces answer business questions.
- Keep traces connected. Check that proxies and gateways pass the trace headers along.
- Document the setup. Write down how the agent is attached and which settings are standard.
Common mistakes beginners make
- Expecting auto-instrumentation to understand business logic.
- Attaching the agent too late, so libraries are already loaded.
- Forgetting to set the service name, and finding data under “unknown.”
- Assuming every library and version is supported.
- Leaving all defaults on and being surprised by cost and noise.
- Mixing an agent’s SDK with a second manually installed one.
- Upgrading and finding that dashboards no longer match the new attribute names.
16Real-World and Industry Examples
How auto-instrumentation shows up in tools and platforms you may already know.
Kubernetes Teams and the Operator
Teams running many services on Kubernetes use the OpenTelemetry Operator so that a single annotation on a workload adds an agent to it. A platform team can turn on tracing for dozens of services with one reviewed change, which is far easier than asking every developer team to edit code.
AWS Lambda and Managed Layers
For serverless functions, AWS offers prebuilt layers based on its distribution of OpenTelemetry. Adding the layer to a function and setting a few options can produce traces for the function and its calls to other services, without editing the function’s code. Because functions are short-lived, the cost of startup time is something to watch.
The Java Agent in Large Enterprises
Large companies with big Java estates, such as banks and retailers, often rely on the Java agent because it covers a wide range of frameworks, database drivers and messaging libraries at once. It lets them add tracing to old applications whose source code few people still understand.
Vendor Distributions
Observability vendors and cloud providers such as Grafana, Elastic and AWS offer their own distributions of OpenTelemetry agents with sensible defaults and easy setup for their platforms. They are built on the same open standard, so data can still be moved to other tools if needed.
Proxies and Web Servers
Modules for web servers and proxies, such as Nginx and Apache HTTP Server, can create the first span of a trace right at the front door and pass the trace headers to the services behind it. This gives teams a complete view starting from the edge of their network.
A typical journey for a growing company
Pilot
Attach an agent to one or two services in a test environment and review the traces.
Tune
Filter noise, set sampling, fix naming and remove sensitive data.
Roll Out
Use a platform mechanism, such as the Operator, templates or layers, to attach the agent widely and in stages.
Add Business Meaning
Add attributes and a few spans to critical flows such as checkout and sign-in.
Operate
Version the settings, test upgrades, watch overhead and review traces regularly.
17Frequently Asked Questions
Quick answers to questions beginners ask most.
18Summary and Key Takeaways
If you remember only a few things, remember these.
Key Takeaways
- Auto-instrumentation attaches, it does not rewrite. An agent or package wraps well-known libraries and produces traces, metrics and logs with little or no change to your code.
- It works through language-specific tricks. Java rewrites classes as they load, Python and Node.js wrap modules, .NET uses a profiler, and Go needs different approaches such as eBPF or wrapper libraries.
- You get the skeleton, not the story. Requests, calls, queries and messages are covered, but business meaning needs manual attributes and spans.
- Settings replace code. Service name, endpoint, sampling and propagation are set through standard environment variables, so platform teams can control many services at once.
- Attach early and roll out carefully. The agent must start before your libraries load, and it should be introduced in stages, with sampling and filtering to control cost and noise.
- Combine auto and manual. Use auto for breadth and a small amount of manual work for depth, sharing one API and one SDK.
- Protect data and pin versions. Review what the agent captures, scrub sensitive values, and test upgrades because names and behaviour can change.