Before touching OpenTelemetry itself, it helps to be completely clear on what “observability” means and what the three pillars of telemetry actually are.

What is observability?

Observability is the ability to understand what is happening inside a running system just by looking at the data it produces from the outside, without having to guess or add new debugging code every time something breaks. A system is observable when an engineer, faced with an unexpected problem they have never seen before, can answer “why is this happening” using only the information the system already emits — not by attaching a debugger to a live production server or adding print statements and waiting for the bug to happen again.

The three pillars: traces, metrics, and logs

Observability data is traditionally split into three related but distinct categories, each answering a different kind of question.

Traces capture the journey of a single request as it moves through a system, showing every service it touched, in what order, and how long each step took. A trace is made up of smaller units called spans, where each span represents one unit of work — for example, one span for “look up the user,” a nested span for “query the database,” and another for “call the payment service.”

Metrics are numeric measurements taken over time, such as how many requests per second a service is handling, how much memory it is using, or how many errors occurred in the last minute. Metrics are cheap to store and excellent for spotting trends and setting alerts, but on their own they cannot explain exactly why a specific request failed.

Logs are timestamped, free-form text records emitted at specific points in code, such as “user 4821 failed login: incorrect password.” Logs are the most detailed and flexible of the three, but also the hardest to search through at scale without some way to connect a given log line back to the specific request it belongs to.

What is OpenTelemetry, specifically?

OpenTelemetry, often shortened to “OTel,” is an open-source, vendor-neutral collection of tools, application programming interfaces, and standards for generating, collecting, and exporting traces, metrics, and logs from software. It is not a monitoring product itself — it does not provide dashboards, alerting, or long-term storage. Instead, it is the common plumbing that generates and moves telemetry data in a standard format, so that any monitoring or observability backend that understands that format can receive and display it. It was formed by merging two earlier, separately governed projects, and today is maintained as a large, vendor-neutral open-source project under a shared governance umbrella.

It is worth being precise about what OpenTelemetry is not, since the surrounding ecosystem contains tools that solve adjacent but different problems. OpenTelemetry is not a monitoring dashboard, not an alerting system, and not a data warehouse. It also has no opinion about how long telemetry data should be kept, how it should be visualized, or what should trigger a page to an on-call engineer — those decisions belong entirely to whatever backend a team chooses to send the data to. OpenTelemetry’s job stops the moment data has been correctly generated, formatted, and delivered.

Why not just build a custom logging solution instead?

Many teams historically solved observability by writing their own custom logging format and building internal tools to search through it. This works reasonably well inside a single service, but breaks down once a request crosses service boundaries, because there is rarely a consistent, agreed-upon way to link a log line in Service A to a related log line in Service B for the same request. Adopting a shared standard like OpenTelemetry, with built-in context propagation, solves exactly this cross-service correlation problem from the start, rather than requiring every team to independently reinvent an ad-hoc version of the same idea.

This separation between API, SDK, instrumentation, and exporter is the single most important design decision behind OpenTelemetry. It means a team can instrument their code once, using the vendor-neutral API, and later change which backend receives that data — from one monitoring vendor to another, or to a self-hosted open-source tool — by changing only configuration and an exporter, without touching a single line of application code.

A brief note on history and governance

OpenTelemetry came into existence by merging two earlier, separately maintained open-source tracing projects that had each built a loyal following but were starting to fragment the ecosystem by competing for the same instrumentation libraries. Rather than one project “winning,” the two communities combined their efforts into a single project, contributed to a large, vendor-neutral open-source software foundation, so that no single company controls its direction and any interested organization can participate in its governance and roadmap.

Understanding how a single piece of telemetry data actually moves from a line of code to a dashboard reveals why OpenTelemetry is built the way it is.

Context propagation: the thread that ties everything together

The single hardest problem OpenTelemetry solves is context propagation — making sure that when Service A calls Service B, which calls Service C, all three services agree that they are working on the same logical request. This is done by attaching a small amount of identifying information, most importantly a trace identifier and a span identifier, to every outgoing network call. When Service B receives a request from Service A, it reads that identifying information from the incoming request headers and uses it to create its own spans as children of the span that made the call, rather than starting a brand-new, disconnected trace. Without this propagated context, every service’s telemetry would look like an isolated island with no way to reconstruct the full journey.

graph LR
    subgraph App["Instrumented Application"]
    A["Application
Code"]
    B["OTel API"]
    C["OTel SDK"]
    end
    subgraph Local["Local Machine / Sidecar"]
    D["OTel Collector"]
    end
    subgraph Backend["Observability Backend"]
    E[("Traces
Storage")]
    F[("Metrics
Storage")]
    G[("Logs
Storage")]
    end
    A --> B
    B --> C
    C -- "OTLP protocol" --> D
    D --> E
    D --> F
    D --> G
        

Spans in detail

A span is the fundamental unit of a trace. Every span has a name describing the operation it represents, a start time and an end time, a unique span identifier, a reference to its parent span (if it has one), and a set of key-value attributes describing details relevant to that operation — for example, which database table was queried, or which HTTP route was hit. A span can also record events that happened during its lifetime, and its final status, marking whether the operation succeeded or failed. A collection of related spans, all sharing the same trace identifier, together forms one complete trace.

The OpenTelemetry Collector

Rather than having every single application send its telemetry directly to a monitoring backend, most real deployments route data through an intermediate component called the Collector. The Collector receives telemetry from many applications, and can then process it — filtering out noisy data, adding extra metadata, batching many small pieces of data into fewer larger network calls — before forwarding it onward to one or more backends. This design means an application only needs to know how to talk to a nearby Collector using one standard protocol, while the Collector takes on the responsibility of knowing how to talk to whatever specific backend or backends a company happens to use.

The OTLP protocol

OpenTelemetry defines its own wire format and network protocol, generally referred to as OTLP, for transmitting traces, metrics, and logs between an application’s SDK and a Collector, or between one Collector and another. Because OTLP is an open, published standard rather than something owned by a single vendor, any tool that speaks OTLP can, in principle, exchange telemetry with any other tool that also speaks OTLP, which is what makes the vendor-neutral promise practically achievable rather than just theoretical.

Sampling

A system handling millions of requests per day cannot realistically record a full, detailed trace for every single one without overwhelming both the network and the storage backend. Sampling is the deliberate decision to record only a subset of traces in full detail, while still tracking overall counts accurately. A simple approach picks a fixed percentage of requests at random; more advanced approaches decide whether to keep a trace only after seeing how it turned out, so that traces containing errors or unusually slow requests are always kept even if most ordinary, fast requests are not.

Metrics and logs follow the same underlying pattern

Although traces tend to get the most attention because of how visually compelling a connected request diagram is, metrics and logs move through the same overall pipeline. A metric such as a request counter is recorded by the SDK, aggregated over a short time window rather than sent one value at a time, and exported through the same OTLP protocol to the same Collector. Logs, once tied into OpenTelemetry’s logging support, gain an important extra capability: a log line generated while a particular span was active can automatically be tagged with that span’s trace and span identifiers, meaning a single click from a slow or failed trace can jump straight to the exact log lines produced during that specific request, rather than the surrounding flood of unrelated log lines from every other request.

Here is the full journey a single user request takes as it becomes a complete, connected trace visible on a dashboard.

graph TD
    A["User request arrives
at Service A"] --> B["Service A's instrumentation
starts a new root span"]
    B --> C["Service A calls
Service B over HTTP"]
    C --> D["Trace context is attached
to the outgoing request headers"]
    D --> E["Service B reads the
incoming trace context"]
    E --> F["Service B starts a
child span linked to Service A's span"]
    F --> G["Service B finishes its work
and closes its span"]
    G --> H["Service A finishes and
closes its own span"]
    H --> I["Completed spans are batched
by the SDK"]
    I --> J["Batches are exported
to the Collector via OTLP"]
    J --> K["Collector processes and
forwards data to backend"]
    K --> L["Backend assembles all spans
sharing one trace ID"]
    L --> M["Complete trace is
visible on a dashboard"]
        

Automatic vs. manual instrumentation

Getting telemetry out of an application can happen in two complementary ways. Automatic instrumentation relies on libraries that attach themselves to common frameworks — a web server, a database client, a messaging library — and generate spans for standard operations like “handle an incoming HTTP request” or “run a database query” without a developer writing any extra code. Manual instrumentation is when a developer deliberately adds a few lines of code around a specific piece of business logic they particularly care about, such as “validate this customer’s discount code,” to capture detail that generic automatic instrumentation could never know to look for.

Batching and asynchronous export

Sending telemetry data over the network the instant each span finishes would slow down the very application being observed, since every request would now wait on an extra network call before finishing. Instead, the SDK collects finished spans in memory and sends them in batches, on a background thread, at a regular interval or once a batch reaches a certain size. This keeps the overhead added to the actual user-facing request extremely small, while still delivering telemetry to a backend within a few seconds in most configurations.

Resource attributes: knowing where data came from

Every piece of telemetry an SDK produces is stamped with a set of resource attributes describing the environment it came from — which service, which version of that service’s code, which deployment environment, and often which specific host or container instance. Without this stamping, a backend receiving telemetry from thousands of application instances across dozens of services would have no reliable way to group, filter, or compare data by service or version, which is why resource attributes are typically configured once, centrally, rather than left to chance in each individual piece of code.

Correlating the three signals together

The real power of the three pillars shows up once they are used together rather than separately. A metric dashboard might first reveal that error rates on a particular service jumped at 2:14 PM. An engineer can then look at traces from around that exact time, filtered to that service, to find a specific failing request. From that one trace, a single click reveals the exact log lines emitted during that request, showing the precise error message and any relevant business context. This three-step narrowing — from “something is wrong” to “here is exactly what happened and why” — is the practical payoff that justifies the upfront effort of adopting a shared telemetry standard across an entire system.

OpenTelemetry’s core promise is vendor neutrality, and that promise shapes both its strengths and the effort it asks of adopters.

Weighing the trade-off in practice

The clearest way to see this trade-off is to compare the cost of adoption against the cost of not adopting it. Instrumenting a large, existing codebase does take real, upfront engineering time, and running a Collector fleet adds a small amount of ongoing operational work. Against that, consider the alternative: an organization locked into one vendor’s proprietary agent format discovers, often at the worst possible moment during contract renegotiation, that switching vendors would mean re-instrumenting every single service from scratch. Teams that instrument against an open standard from the start never face that particular form of leverage being used against them, because the underlying telemetry generation was never tied to any one vendor’s proprietary format in the first place.

Teams that get lasting value out of OpenTelemetry tend to follow a small set of recognizable patterns.

Pattern: instrument at the edges first

Rather than trying to add detailed manual instrumentation everywhere at once, effective teams start by enabling automatic instrumentation at the entry and exit points of each service — incoming requests, outgoing calls to other services, database queries — since this alone reconstructs the overall shape of how requests flow through a system. Deeper, hand-written instrumentation of specific business logic is then added incrementally, focused on the areas that turn out to matter most once the overall picture is visible.

Pattern: consistent, meaningful attribute naming

Establishing a shared convention for how attributes are named and what values they contain — so that “customer identifier” is always called the same thing across every service, in the same format — pays off enormously once an engineer is searching across telemetry from dozens of different services during an incident. OpenTelemetry publishes a set of semantic conventions precisely to give teams a starting point rather than inventing naming schemes from scratch.

Pattern: route everything through a Collector

Sending telemetry directly from every application straight to a backend works for a small project, but it means every application needs to know the backend’s specific address, credentials, and format, and any change to the backend requires touching every application. Routing all telemetry through a Collector first centralizes that knowledge in one place, so switching or adding a backend later becomes a Collector configuration change rather than a fleet-wide application redeployment.

Pattern: treat instrumentation as a shared platform concern

Rather than leaving each individual team to decide independently how to name services, which attributes to record, and how to configure sampling, mature organizations put a small central platform team in charge of providing a pre-configured baseline — shared libraries, agreed naming rules, sensible default sampling — that every application team then builds on top of. This avoids the common failure mode where fifty services each instrument themselves slightly differently, producing telemetry that is technically present everywhere but painfully inconsistent to search across during an actual incident.

Pattern: correlate telemetry with deployment events

Attaching a service’s current version, and the exact time a new version was deployed, to its telemetry makes it dramatically faster to answer a question that comes up constantly during incidents: “did this problem start right after our last deployment.” Without that correlation available directly in the telemetry, teams end up manually cross-referencing deployment logs against monitoring dashboards by hand, wasting time during exactly the moments when speed matters most.

Anti-pattern: treating 100 percent sampling as the safe default forever

Capturing every single trace in full detail feels like the most cautious choice, and it is genuinely useful early on while a team is still learning what their system’s normal behavior looks like. Left unchanged as traffic grows, however, it becomes an expensive habit that provides diminishing extra insight once the overall shape of typical traffic is already well understood, which is why most mature deployments eventually move to a more deliberate, tail-based sampling strategy that keeps interesting traces and reduces cost on the vast majority of routine, successful ones.

Pattern: building dashboards around the request, not the server

Older monitoring approaches were often built around individual servers — is this specific machine healthy, is its CPU too high. Teams getting real value from OpenTelemetry tend to shift their default mental model toward the request instead: how is a typical checkout request performing right now, across every service it touches, regardless of which specific machine happened to handle it. This shift matters because in modern, elastically scaled systems, individual servers come and go constantly, while the shape of a business-critical request stays meaningful over much longer periods of time.

Most OpenTelemetry adoption struggles come down to a small, repeatable set of avoidable mistakes.

Common mistake: ignoring cardinality on metrics

Attaching a high-cardinality value, such as a unique user identifier or a full request path with embedded identifiers, directly onto a metric can cause the number of distinct metric series a backend has to track to explode into the millions, sharply increasing cost and sometimes overwhelming the backend entirely. High-cardinality detail like a specific user identifier belongs on a span or a log line, where it is expected and manageable, not on a metric, which is designed for a much smaller number of distinct combinations.

Common mistake: forgetting to test instrumentation changes

Changes to instrumentation code are still code changes, and an incorrectly configured exporter or a typo in a Collector configuration file can silently stop telemetry from flowing at all, without causing any visible application error. Treating instrumentation configuration with the same testing and review discipline as any other production configuration change catches this class of mistake before it creates a costly blind spot.