What Is the OpenTelemetry API?
A friendly, ground-up guide to the part of OpenTelemetry that your code actually talks to: what it is, why it is separate from everything else, how it works inside, and how to use it without hurting your system.
Imagine you run a food delivery app. On Friday evening, thousands of people tap “Order now” at the same moment. Suddenly, one customer says the app took twenty seconds to confirm an order. Behind that single tap, your system may have talked to forty different small programs: one checks the menu, one checks the restaurant, one charges the card, one finds a driver. Which one was slow? Without a good way to look inside, you are guessing in the dark.
This is the problem OpenTelemetry was built to solve. It gives every program a common way to say, “Here is what I did, how long it took, and whether it worked.” The part of OpenTelemetry that your code talks to is called the OpenTelemetry API. In this guide, we will learn what it is step by step, using simple words and everyday examples. By the end, you will understand not only what it does, but why it was designed the way it was.
1The Problem: Why Can’t We See Inside Our Software?
Before learning any tool, it helps to feel the pain it removes.
Software is no longer one big program
Many years ago, an application was often one large program running on one computer. If it was slow, you opened it up, looked at the logs, and found the problem. Today, most big products are built from many small services that talk to each other over the network. Netflix, Uber and Amazon each run hundreds or thousands of such services. A single click from a user can travel through dozens of them.
When something breaks or slows down in such a system, the cause can hide anywhere. The slow part may not even be the service the user was talking to. It might be a database three steps away. Looking at one service alone tells you almost nothing about the whole story.
What is observability?
Observability is the ability to understand what is happening inside a system by looking at the information it sends out. Think of a doctor. A doctor cannot open you up every time you feel unwell. Instead, the doctor checks your temperature, heartbeat and blood pressure. Those readings are signals that tell the story of your inner health. Software needs the same kind of signals.
Think about a courier parcel. When you order something online, you receive a tracking number. Every time the parcel passes a warehouse, a scan is recorded: where it was, when, and how long it stayed. If the parcel is late, you can look at the scans and see exactly where it got stuck. Observability gives your software requests their own tracking numbers and scans.
The mess before OpenTelemetry
Before OpenTelemetry, every monitoring vendor had its own way of collecting this information. If you chose Vendor A, you installed Vendor A’s libraries all over your code. If you later wanted to move to Vendor B, you had to find and rewrite all that code. This is called vendor lock-in, and it is like building your house with plugs that only fit one electricity company.
Worse, the libraries your application depends on, such as web frameworks and database drivers, could not easily include monitoring code, because they did not know which vendor you would pick. The industry needed one shared, neutral language for this job. That language is OpenTelemetry.
2So, What Is the OpenTelemetry API?
The shortest honest answer, and then the details that make it make sense.
OpenTelemetry (often shortened to OTel) is an open-source project that defines a standard way to create, collect and send telemetry data. Telemetry is simply information that a system sends out about itself, such as timings, counts and messages. OpenTelemetry is not one single tool. It is a family of pieces that work together.
The OpenTelemetry API is the set of interfaces, which are like menus of allowed actions, that your code uses to produce telemetry. It answers questions like: “How do I say that a piece of work has started?” and “How do I record that one more order was placed?” The API defines the words and sentences. It does not decide where the data goes or how it is stored.
The four pieces of the OpenTelemetry family
Many beginners mix these up, so let us separate them clearly.
The API
The interfaces your code and libraries call. It is small, stable and has almost no work of its own to do.
The SDK
The real implementation behind the API. It decides what to keep, adds extra details, batches the data and hands it to exporters.
OTLP and Exporters
OTLP is the standard message format and protocol for sending telemetry. Exporters are the parts of the SDK that actually send it.
The Collector
A separate program that receives telemetry from many applications, cleans it up, and forwards it to one or more storage backends.
API versus SDK: the most important idea
A wall socket in your home is the API. Any lamp, fan or phone charger can plug into it without knowing where the electricity comes from. The power station and the wires behind the wall are the SDK. You can switch from one electricity company to another, and your lamp never notices. If the power station is switched off, the socket is still there; plugging in a lamp simply does nothing.
This split is the heart of the design. Libraries and application code depend only on the API. The person who owns the final application decides which SDK to install, how to configure it, and where to send the data. Because of this, a library author can add monitoring once, and it works for everyone, with any vendor, without forcing a choice.
| Question | API | SDK |
|---|---|---|
| Who uses it? | Application code and library authors | The application owner, usually at startup |
| What does it contain? | Interfaces and safe do-nothing defaults | Real processing, sampling, batching and exporting |
| How often does it change? | Very rarely, because breaking it would hurt everyone | More often, as features and performance improve |
| What if it is missing? | Not possible, your code needs it to compile | Telemetry calls quietly do nothing |
| Does it send data anywhere? | No | Yes, through exporters |
3A Short History
Knowing where something came from explains many of its design choices.
In the years before 2019, two open-source projects were trying to solve the same problem in different ways. OpenTracing focused on a neutral API for tracing. OpenCensus, which grew out of ideas used inside Google, offered both tracing and metrics with its own libraries. Developers were confused. Which one should they pick? Library authors were stuck, because supporting both meant doing the work twice.
In 2019, the two communities decided to join forces and create a single project, and OpenTelemetry was born. It became part of the Cloud Native Computing Foundation (CNCF), the same neutral home that looks after Kubernetes. Since then, it has become one of the most active projects in that foundation, with contributors from cloud providers, monitoring vendors and large end-user companies.
Before 2019: Two Competing Projects
OpenTracing and OpenCensus each solved part of the problem. Teams had to pick a side.
2019: The Merger
The two projects combined into OpenTelemetry, keeping the best ideas of both and adding a clean API and SDK split.
Early 2020s: Tracing and Metrics Mature
The tracing and metrics APIs reached stable status in the major languages, giving teams confidence to adopt them in production.
Later: Logs Join the Family
A logging API was added so that logs could carry the same identifiers as traces and be connected to them.
Today: The Industry Default
Major clouds and monitoring vendors accept OpenTelemetry data, and popular frameworks include it out of the box.
Signal stability differs by language and changes over time. Always check the official status page for your language before depending on a newer feature, such as profiling, which was still evolving at the time of writing.
4Core Concepts: The Three Signals
A signal is a type of telemetry. The API has a separate set of tools for each type.
Imagine a hospital patient. The doctor looks at three different kinds of information. A timeline of what happened during the visit (admitted, scanned, treated, discharged). A set of numbers measured over time, like temperature and heart rate. And the nurse’s written notes. OpenTelemetry has the same three kinds of signals.
Traces
The story of one request as it travels through your system. Made of timed steps called spans.
Metrics
Numbers measured over time, such as requests per second, memory used, or how long responses take.
Logs
Timestamped text records of events, such as “payment declined for order 8841”.
Baggage and Context
Not signals themselves, but the carrier system that moves identifiers and small notes between steps and services.
Why three signals and not one?
Each signal answers a different question. Metrics tell you that something is wrong: “errors jumped from 1 percent to 9 percent.” Traces tell you where it is wrong: “the payment step is taking eight seconds.” Logs tell you why it is wrong: “the card provider returned a timeout error.” A doctor who only had temperature readings would be missing most of the picture. Real troubleshooting uses all three, and OpenTelemetry lets them share identifiers so you can jump from one to another.
Production Example: A Checkout Slowdown
An online store sees a metric alarm: checkout latency has doubled. An engineer opens a trace of a slow checkout and sees that the “reserve stock” step takes most of the time. Clicking the logs linked to that exact trace shows repeated database lock warnings. Three signals, one story, solved in minutes instead of hours.
5The Tracing API in Depth
Tracing is where most people begin, and it introduces ideas used everywhere else.
Trace and span
A trace is the complete journey of one request through your system. A span is one timed piece of work inside that journey. Each span has a name, a start time, an end time, and a few extra details. Spans are connected in a parent and child relationship. If “Place order” calls “Charge card,” then “Charge card” is a child span of “Place order.” Together, spans form a tree that shows the whole story.
Picture a school trip. The whole trip is the trace. Each activity, such as the bus ride, the museum visit and the lunch stop, is a span. The museum visit may contain smaller spans: the guided tour and the gift shop. If the trip ran two hours late, you can look at the spans and see that the lunch stop alone took ninety minutes.
The main characters of the tracing API
TracerProvider
The starting point. It hands out tracers. An application usually has one, set up once at startup.
Tracer
The object used to create spans. A tracer is named after the library or component that uses it, so you can tell who produced what.
Span
One timed operation. You start it, add details, and end it. Forgetting to end it is a classic mistake.
SpanContext
The small identity card of a span: its trace ID, its span ID, flags such as “sampled,” and optional trace state. This is what travels between services.
What can you attach to a span?
- Attributes: key and value details, like the HTTP method, the restaurant ID or the database name. They help you search and filter later.
- Events: timestamped notes inside a span, such as “cache miss” or “retry started.”
- Links: connections to other spans that are related but not parent or child. For example, a batch job span can link to the many message spans it processes.
- Status: one of three values: unset (nothing special said), ok (explicitly successful) or error (something failed).
- Exceptions: a standard way to record that an error happened, including its type and message.
Span kinds: who is talking to whom?
Every span has a kind that describes its role in a conversation. This helps backends draw correct service maps and calculate things like “time spent waiting on the network.”
| Kind | Meaning | Simple Example |
|---|---|---|
| Server | Handling an incoming request from a remote caller | The order service receiving “place order” |
| Client | Making an outgoing request to a remote service | The order service calling the payment service |
| Internal | Work that stays inside one process | Calculating the final price |
| Producer | Sending a message to a queue for later work | Placing a “send email” message on a queue |
| Consumer | Receiving and processing a queued message | The email worker reading that message |
A span that is never ended is never sent, and it can leak memory in some setups. Always make sure the span ends, even when an error happens halfway through the work.
6The Metrics API in Depth
Metrics are cheap, compact and perfect for dashboards and alarms.
A metric is a number that is measured and tracked over time. The metrics API has a MeterProvider that hands out Meters, and a meter creates instruments. An instrument is a specific kind of measuring tool. Choosing the right instrument is the main skill in metrics, so let us look at each.
Think of a car dashboard. The odometer only ever goes up, like a counter. The fuel gauge shows the current level and can go up or down, like a gauge. The trip summary that says “most of your speeds were between 40 and 60” is like a histogram. Each tells you something different, and you would not want to use the wrong one.
| Instrument | What It Measures | Everyday Example |
|---|---|---|
| Counter | A value that only goes up | Total orders placed |
| UpDownCounter | A value that can rise and fall by steps | Number of users currently in a queue |
| Histogram | A spread of values, grouped into ranges | How long each request took |
| Gauge | The latest value at a moment in time | Current room temperature |
| Observable (async) versions | Values read on demand when the SDK asks | Current memory used by the program |
Synchronous versus asynchronous instruments
A synchronous instrument is used at the moment something happens. When an order is placed, you add one to the counter right there. An asynchronous (observable) instrument works the other way around. You register a small function once, and the SDK calls it on a schedule to read the current value. This is perfect for things you cannot easily tie to one event, like the size of a memory pool.
Attributes on metrics
Just like spans, metric measurements can carry attributes. A counter of orders can be split by payment method or by country. This is powerful, but it hides the biggest trap in all of metrics, called cardinality. Every unique combination of attribute values creates a separate time series that must be stored. If you add a user ID as an attribute, and you have ten million users, you have just created ten million series. Costs and memory can explode.
Only use attributes with a small, known set of values on metrics, such as region, status class or payment type. Things like user IDs, order IDs and full URLs belong on traces or logs, not on metrics.
7The Logs API and Baggage
Two quieter parts of the API that are often misunderstood.
The Logs API
Developers have been writing logs for decades, and most teams already use a logging library they like. OpenTelemetry does not ask you to throw it away. The Logs API defines a LoggerProvider, a Logger and a LogRecord, which is one log entry with a timestamp, a severity level, a body and attributes. The key benefit is that a log record can automatically carry the trace ID and span ID of the work that was happening when it was written.
It is important to know the intended audience. The Logs API is mainly meant for people who build log bridges or appenders, which are small adapters that take records from an existing logging library and forward them into OpenTelemetry. Most application developers keep using their familiar logger, and a bridge quietly connects it. This is a thoughtful choice, because it lets you adopt OpenTelemetry without rewriting every log line.
Baggage
Baggage is a set of key and value pairs that travels along with a request from service to service. Unlike span attributes, which stay in the span where you put them, baggage moves with the request and can be read by every service downstream.
Imagine a sticky note attached to a parcel. Every warehouse the parcel passes can read the note, such as “fragile” or “gift for customer tier gold”. The note travels with the parcel, but nobody writes it into the tracking scans unless they choose to copy it there.
An important detail: baggage is not automatically added to spans or metrics. A service has to read it and copy it over on purpose. Teams often use baggage for things like a customer tier or an experiment name, so that all downstream services can tag their own telemetry consistently.
Baggage travels in plain request headers, so it can be seen by any service that handles the request, including third parties. Never put secrets, passwords or personal data in baggage.
8Context and Propagation: The Glue
This is the idea that makes a trace more than a pile of unrelated spans.
What is context?
The Context is an object that carries the current information about “where we are” in a request, such as the active span and the baggage. When you start a new span, the API looks at the current context to find the parent. This is how a child span knows who its parent is, without you passing the parent around by hand.
Different languages store the current context in different ways, because each language handles “who is running right now” differently. Some use thread-local storage, some use special async-aware storage, and some, such as Go, ask you to pass a context object explicitly through your function calls. The API hides these differences behind a consistent idea.
What is propagation?
Context only lives inside one program. To keep a trace connected across the network, the identifiers must be written into the outgoing request and read back by the next service. This is called propagation. The writing step is called inject, and the reading step is called extract.
It is like a relay race. The baton is the trace identity. Each runner must hand it to the next runner, or the team loses track of who is running the race. Inject is handing over the baton. Extract is receiving it.
The standard format
To make sure services written in different languages understand each other, OpenTelemetry uses the W3C Trace Context standard by default. It defines a request header named traceparent. This header carries four things: a version, a 128-bit trace ID shared by the whole trace, a 64-bit span ID of the calling span, and a flags field that includes whether the trace is being sampled. A second header, tracestate, can carry extra vendor-specific hints. A separate W3C Baggage header carries baggage.
Because this is an open standard, a Java service can call a Go service, which calls a Python service, and the trace stays whole. Older formats, such as B3 from the Zipkin world, are also supported through propagator plugins, which helps when you are migrating step by step.
sequenceDiagram
participant U as User Browser
participant G as API Gateway
participant O as Order Service
participant P as Payment Service
U->>G: HTTP request
G->>G: Start server span and trace ID created
G->>O: Call with traceparent header
O->>O: Extract context and start child span
O->>P: Call with traceparent header
P->>P: Extract context and start child span
P-->>O: Response
O-->>G: Response
G-->>U: Response
In the diagram above, the gateway creates the trace ID because it is the first stop. It passes the identity to the Order Service inside a header. The Order Service extracts it and starts a child span that shares the same trace ID. It then does the same when calling the Payment Service. Later, a backend collects all the spans with that trace ID and assembles the full tree.
9Architecture and Components
Let us put every piece on one map and see who talks to whom.
flowchart LR
subgraph APP["Your Application Process"]
CODE["Application Code"]
LIB["Instrumented Libraries"]
API["OpenTelemetry API"]
SDK["OpenTelemetry SDK"]
PROC["Processors and Samplers"]
EXP["OTLP Exporter"]
end
CODE --> API
LIB --> API
API --> SDK
SDK --> PROC
PROC --> EXP
EXP --> COL["OpenTelemetry Collector"]
COL --> B1["Tracing Backend"]
COL --> B2["Metrics Backend"]
COL --> B3["Logging Backend"]
Read the diagram from left to right. Your own code and the libraries you use both call the API. Neither knows or cares what is behind it. The SDK sits behind the API and receives every call. Inside the SDK, samplers decide which traces to keep and processors add helpful details or group the data into batches. The exporter then sends batches out using OTLP.
Most production setups then send data to the Collector, a standalone program. The Collector can receive data in many formats, drop noisy data, add extra labels, and route different signals to different backends. Teams often run a Collector next to each application, and another central one that aggregates everything. The backends themselves, such as Jaeger, Prometheus, Grafana, or a commercial vendor, are outside OpenTelemetry. They simply store and display the data.
Resource: who is sending this data?
Every piece of telemetry needs to say who produced it. In OpenTelemetry, this description is called a Resource. It contains attributes such as the service name, service version, the cloud region and the host. The most important one is service.name. Without it, a backend shows your data as “unknown service,” which is not helpful at all. The resource is set in the SDK, not in the API, because it describes the application as a whole and not a single operation.
Instrumentation scope
When a tracer or meter is created, it is given a name and optionally a version. This is the instrumentation scope. It tells you which library or module created the data, for example a web framework versus your own checkout code. When something looks strange, this helps you find out which piece of code is responsible.
10Internal Working: The Magic of the No-Op
How can a library safely use the API when nobody has installed anything behind it?
The most clever idea inside the API is the no-op implementation. “No-op” means “no operation,” which is a fancy way of saying “does nothing.” When your code asks the API for a tracer and no SDK has been installed, the API returns a do-nothing tracer. You can start spans, add attributes and end them, and nothing happens. No data is stored, no errors are thrown, and almost no time is spent.
flowchart TD
A["Library calls API to start a span"] --> B{"Is an SDK registered?"}
B -->|No| C["Return no-op span"]
C --> D["Call returns almost instantly"]
B -->|Yes| E["SDK creates a real span"]
E --> F["Sampler decides keep or drop"]
F -->|Keep| G["Span recorded and later exported"]
F -->|Drop| H["Span not recorded but context still passes along"]
A no-op is like a light switch that is not wired to any bulb. You can flip it all day and nothing breaks. The moment an electrician connects the wiring, the same switch starts doing real work. The person who installed the switch never had to know in advance.
Why this design matters
This gives library authors the freedom to add instrumentation without fear. A database driver can include spans for every query. If you do not use OpenTelemetry, you pay almost nothing. If you do, you get detailed traces with no extra effort. It also means the API has no hard dependency on the heavy parts, so adding it to a library does not drag in large amounts of other code.
How the SDK plugs in
When the application starts, the owner builds the SDK, configures it, and registers it as the global provider. From that moment, calls that used to go to the no-op now go to real objects. Libraries that asked for a tracer earlier usually get a lightweight proxy that quietly switches to the real tracer once it is available, so startup order problems are handled gracefully in many languages.
Sampling and the flag that travels
Recording every request at huge scale is expensive. Sampling means keeping only a portion of traces. A common choice is head sampling, where the decision is made when the trace begins, for example keeping ten percent. Another is tail sampling, usually done in the Collector, where the decision is made after the trace is finished, so you can keep all slow or failed ones. The API itself does not sample. It only carries the sampled flag in the span context so all services in a trace agree on the decision.
11Data Flow and Lifecycle of a Span
Follow one span from birth to a dashboard.
flowchart TD
S1["Get a Tracer from the TracerProvider"] --> S2["Start a span"]
S2 --> S3["Add attributes and events"]
S3 --> S4{"Did the work fail?"}
S4 -->|Yes| S5["Record exception and set error status"]
S4 -->|No| S6["Leave status as unset or ok"]
S5 --> S7["End the span"]
S6 --> S7
S7 --> S8["SDK batches finished spans"]
S8 --> S9["Exporter sends batch using OTLP"]
S9 --> S10["Collector or backend stores it"]
Start
Code asks for a tracer and starts a span. The API checks the current context to find the parent and give the new span its identity.
Enrich
While the work is running, code adds attributes like the customer tier, and events like “retry attempt 2”.
Finish
When the work is done, the span is ended and its end time is stamped. If the work failed, the error is recorded first.
Batch
The SDK puts finished spans into a queue and groups them, because sending one at a time would waste network effort.
Export
The exporter sends the batch in the background. Importantly, this must never slow down your real request.
Store and Search
A backend receives the spans, joins them by trace ID, and lets engineers search, filter and view them as a tree.
A few design rules protect your application during this journey. The SDK uses bounded queues, which means that if the exporter falls behind, old data is dropped rather than letting memory grow forever. Telemetry code is also designed not to throw exceptions into your business logic. A good rule used by the whole project is that observability must never be the reason your service goes down.
12Semantic Conventions: Speaking the Same Words
A shared dictionary that makes data from different teams comparable.
Imagine two teams record the HTTP method. One writes “method”, another writes “http_verb”, and a third writes “verb”. A dashboard that wants to show “slow POST requests” would have to know all three names. Semantic conventions solve this by defining standard names and meanings for common things, such as HTTP requests, database calls, messaging systems, and cloud environments.
For example, the standard says what the attribute for an HTTP method is called, what the attribute for a response status code is called, and how to name a span for an incoming web request. Following these rules means that dashboards, alerts and vendor features work out of the box, because everybody uses the same words.
Think of road signs. A stop sign is red and octagonal in almost every country. A driver in a new city does not need to learn a new language to understand it. Semantic conventions are road signs for telemetry.
Some conventions are still being renamed or stabilised as the project matures, especially in newer areas. Before building dashboards that depend on specific attribute names, check the version of the conventions your instrumentation follows, so that a library upgrade does not silently change names.
13Manual and Automatic Instrumentation
Two ways to get telemetry out of your code.
Instrumentation means adding the code, or the hooks, that produce telemetry. There are two main approaches, and most real systems use both.
Automatic Instrumentation
- Attaches to popular frameworks and libraries with little or no code change
- Gives instant coverage of web requests, database calls and outgoing calls
- Great for getting started in a day
- May be less detailed about your own business logic
Manual Instrumentation
- You decide exactly which operations deserve a span or metric
- Lets you capture business meaning, like “discount applied”
- Uses the API directly in your code
- Needs more effort and care to keep consistent
A very sensible path is to start with automatic instrumentation to get a broad picture quickly, then add manual spans and metrics only for the business steps that matter most, such as checkout, search or sign-in. In Java, for example, a special agent can attach at startup. In other languages, instrumentation packages are added and enabled for each library. In every case, they all talk to the same API underneath, so the data looks uniform.
14Advantages, Disadvantages and Trade-offs
No tool is perfect. Knowing the costs helps you decide with open eyes.
Advantages
- Vendor neutral: change your backend without rewriting code
- One standard: traces, metrics and logs share identifiers
- Safe for libraries: the no-op design costs almost nothing when unused
- Wide support: many languages, frameworks, clouds and vendors
- Stable API: a small surface that avoids breaking changes
- Open and community driven: no single company controls it
Disadvantages
- Learning curve: many new terms, and API versus SDK confuses beginners
- Uneven maturity: language support and signals are not equally stable everywhere
- Moving conventions: naming standards can change between versions
- Not a backend: you still need to pick and run a place to store and view data
- Overhead exists: an enabled SDK still uses some CPU, memory and network
Key trade-offs
| Decision | Option A | Option B |
|---|---|---|
| How much to sample | Keep everything for full detail, but pay more for storage | Keep a small share to save cost, but risk missing rare problems |
| Where to sample | At the start (head), which is cheap and simple | At the end (tail), which is smarter but needs more infrastructure |
| How to instrument | Automatic for speed and breadth | Manual for precision and business meaning |
| How to send data | Directly from the app to the backend, which is simple | Through a Collector, which adds a hop but gives control and safety |
| Metric attributes | Rich attributes for detailed slicing at higher cost | Few attributes for lower cost and less detail |
15Security Considerations
Telemetry is data about your system, and it can accidentally become a leak.
It is easy to forget that traces and logs often contain real information. A well-meaning developer might record the full web address of a request, including a secret token in the query string. Or they may save a customer email as a span attribute. That data is then stored in a monitoring system that many more people can access than the original database. Treat telemetry with the same care as any other sensitive data.
Keep Personal Data Out
Do not put names, emails, card numbers, passwords or tokens into attributes, events, log bodies or baggage. Use internal IDs if you must identify something.
Redact in the Pipeline
Use Collector processors or SDK settings to remove or mask sensitive fields as a second line of defence.
Be Careful at Boundaries
Incoming trace headers come from outside. A malicious caller can send fake or huge values. Decide whether to trust, restart or limit incoming context at public edges.
Encrypt and Authenticate
Send telemetry over encrypted connections and require authentication between agents, Collectors and backends.
Cap Sizes
Use attribute count and length limits so that one bad value cannot bloat memory or storage.
Control Who Can View
Apply role-based access to your dashboards and trace viewers, as they may reveal business details.
Rules such as GDPR and similar privacy laws apply to telemetry as well. If a trace can identify a person, it may count as personal data and must be handled accordingly, including how long you keep it.
16Monitoring, Logging and Metrics: Watching the Watcher
Your telemetry pipeline is a system too, and it needs to be healthy.
It sounds funny, but you must monitor the system that does the monitoring. If the exporter is silently failing, your dashboards will go flat, and you may think everything is calm when in fact you are blind. Both the SDK and the Collector can report their own health, and you should look at these signals regularly.
- Dropped data: count how many spans, metrics or logs are discarded because queues are full. A growing number means the pipeline cannot keep up.
- Export failures: watch for failed or retried sends to the Collector or backend.
- Queue size and delay: long queues suggest slow exporters or a struggling backend.
- Collector resource use: keep an eye on its CPU and memory, since a busy Collector can become a bottleneck.
- Missing services: set an alert when a service that normally reports telemetry suddenly sends nothing.
- Telemetry cost: track the volume of data per service, because a single change can multiply it overnight.
Production Example: The Silent Gap
A team once noticed their error dashboard showed zero errors for a whole afternoon. It looked like a miracle. In reality, a configuration change had broken the exporter, so no data left the application. A simple alert for “this service sent no spans for ten minutes” would have caught it immediately.
17Design Patterns and Anti-Patterns
Proven habits to copy, and common mistakes to avoid.
Good design patterns
Facade
The API itself is a facade: a simple front that hides complex machinery. Keep your own code talking only to the API.
Instrument at the Boundaries
Create spans where work enters or leaves a service or process: incoming requests, outgoing calls, queue messages and database queries.
Decorator or Middleware
Wrap handlers once in a shared layer so every endpoint gets a span, instead of editing each one by hand.
Propagate Everywhere
Always pass context through threads, async tasks and message queues so the trace stays connected.
Agent plus Gateway Collectors
Run a light Collector near each application, and a central one for heavy tasks like tail sampling and routing.
Follow Conventions
Use standard names and units so every team’s data fits together on shared dashboards.
Anti-patterns
The Mistake
Putting high-cardinality values, such as user IDs or request IDs, as attributes on metrics.
Why It Hurts
Each unique value creates a new time series. Storage cost, memory use and query time all grow rapidly, and some backends will start rejecting data.
Better Approach
Keep metric attributes to small, fixed sets. Put detailed identifiers on traces and logs, where one record per event is expected.
The Mistake
Creating a span for every tiny function call.
Why It Hurts
Traces become huge and noisy, hiding the real story. Overhead and storage costs grow, and engineers cannot find what matters.
Better Approach
Create spans for meaningful units of work, usually those that cross a boundary or have a meaningful duration. Use events for small details inside a span.
The Mistake
A library that depends on the SDK or configures exporters on its own.
Why It Hurts
It forces choices on the application owner, may conflict with their setup, and can create duplicate or broken data.
Better Approach
Libraries should depend only on the API. Leave the SDK and exporter setup entirely to the final application.
The Mistake
Breaking the context chain: starting background work or sending messages without carrying the current context.
Why It Hurts
The trace splits into disconnected fragments. You see many small traces instead of one story, and root-cause hunting becomes slow.
Better Approach
Pass or capture the context when handing work to another thread, task or queue, and inject it into message headers.
18Best Practices and Common Mistakes
A practical checklist to keep your telemetry useful and affordable.
Best practices
- Name your service clearly. Set a meaningful service name and version on every application, so data is never labelled “unknown.”
- Name spans by operation, not by data. Use a name like “GET orders by id” rather than a name that includes the actual order number. Names with changing values create endless unique names.
- Always end spans and record errors. Use structures that guarantee the span ends, even when something fails.
- Mark failures properly. Set the error status and record the exception, so error views and alerts work.
- Use the right instrument. Counters for totals, histograms for durations, gauges for current levels.
- Use units and clear names. A metric named only “time” is confusing. Say what it measures and in which unit, following conventions.
- Start with sampling in mind. Decide your keep rate early and make sure all services follow the same parent-based decision.
- Use a Collector. It gives you a safe place to filter, redact and route data without redeploying applications.
- Link logs to traces. Make sure log records carry trace and span IDs, so engineers can jump between them.
- Test your telemetry. Treat important spans and metrics as part of the product, and verify them in test environments.
Common mistakes beginners make
- Thinking the API alone sends data. Without an SDK, nothing leaves the application.
- Installing the SDK but forgetting to register it globally, so libraries still see the no-op.
- Forgetting that background threads and async tasks may lose the current context.
- Adding every possible attribute “just in case,” which increases cost and privacy risk.
- Assuming that sampled-out traces are free. The context still travels, and some work still happens.
- Treating OpenTelemetry as a replacement for a backend. It produces and moves data. It does not store or show it.
If you are unsure what to measure first, use the “golden signals” idea: how many requests (traffic), how many fail (errors), how long they take (latency), and how full your resources are (saturation). Four well-chosen measurements already tell you most of the health story.
19Real-World and Industry Examples
How the ideas show up in systems you may already know.
Uber and Jaeger
Uber built the distributed tracing system Jaeger to understand requests across its huge network of services. Jaeger later became a CNCF project, and its maintainers now recommend using OpenTelemetry for instrumentation, with Jaeger serving as one of the places to store and view the resulting traces. This shows the split well: OpenTelemetry produces the data, and Jaeger is a backend.
Cloud Providers
Amazon offers the AWS Distro for OpenTelemetry, a supported build of the components, and tools such as X-Ray and CloudWatch can receive the data. Google Cloud and Microsoft Azure also accept OpenTelemetry data in their monitoring services. For customers, this means one instrumentation approach can work across clouds.
Kubernetes and Service Meshes
Kubernetes components have added tracing built on OpenTelemetry, so that operators can follow requests inside the cluster itself. Popular proxies and meshes such as Envoy can also emit traces that join the same trace as your application spans, because they speak the standard propagation headers.
Frameworks and Libraries
Widely used frameworks, such as Spring in the Java world, connect their own observability layers to OpenTelemetry. Communication libraries such as gRPC also support it. When these libraries use the API, applications get detailed telemetry from the first day, and no vendor-specific code is required.
Monitoring Vendors
Commercial and open-source observability platforms such as Datadog, New Relic, Honeycomb and Grafana accept OTLP data. A company can start with one vendor and later move or add another by changing Collector settings, not by re-instrumenting hundreds of services. That freedom is the main promise of the whole project.
A typical journey for a growing company
Start Small
Enable automatic instrumentation on two or three key services and send data to a Collector.
Standardise
Agree on service names, resource attributes and semantic conventions across teams.
Add Business Spans and Metrics
Use the API directly for checkout, search and payments, with carefully chosen attributes.
Control Cost
Introduce sampling, tail sampling in the Collector, and data redaction as volume grows.
Connect Everything
Link logs, metrics and traces, build alerts on user-facing goals, and monitor the pipeline itself.
20Frequently Asked Questions
Quick answers to the questions beginners ask most.
21Summary and Key Takeaways
If you remember only a few things, remember these.
Key Takeaways
- The API is the language, not the engine. It defines how code creates traces, metrics and logs, but it does not process or send them. That is the SDK’s job.
- The API and SDK split gives freedom. Libraries depend only on the API, and application owners choose the SDK, exporters and backend, so there is no vendor lock-in.
- The no-op default makes instrumentation safe. If no SDK is present, API calls do almost nothing, so libraries can include telemetry at almost no cost.
- Context and propagation connect the dots. Trace IDs travel in the standard traceparent header, turning many separate spans into one story across services.
- Choose signals and instruments wisely. Metrics tell you that something is wrong, traces show where, and logs explain why. Keep metric attributes low in cardinality.
- Protect and watch your pipeline. Keep secrets and personal data out of telemetry, use a Collector for control, and monitor for dropped or missing data.
- Start simple and grow. Begin with automatic instrumentation, follow semantic conventions, then add business-specific spans and metrics where they matter most.