A provider is the starting point for one kind of telemetry. It hands out the tools your code uses and carries the settings that shape everything those tools produce.

OpenTelemetry is an open-source project that gives software a standard way to produce telemetry, the information a program sends out about itself. Telemetry comes in several kinds, called signals. The three main signals are traces, which follow one request through many steps, metrics, which are numbers measured over time, and logs, which are time-stamped records of events.

For each signal there is one provider. The TracerProvider belongs to traces. The MeterProvider belongs to metrics. The LoggerProvider belongs to logs. The names follow a pattern that is easy to remember: the tool is named after the job, and the provider is the thing that provides it. A Tracer traces, a Meter measures and a Logger logs.

What a provider does in one sentence

A provider is a factory and a holder of settings. As a factory, it creates and gives out the Tracer, Meter or Logger that your code asks for. As a holder of settings, it knows the identity of your service, the rules for processing the data, and the destination where the data will be sent. It also controls the life of the whole signal: how it starts, how it sends remaining data and how it stops.

Where they live

Providers belong to the SDK, the engine part of OpenTelemetry, as we explain in chapter 3. Your application code and the libraries it uses talk only to the simple interfaces of the API. The providers sit behind those interfaces, quietly doing the real work. Most people write their provider set-up once, in a small startup section of the program, and never touch it again. Others never write it at all, because an automatic helper creates the providers for them.

A middleman may look like extra work, but here it removes four real difficulties.

Difficulty 1: libraries must not decide where data goes

Your program uses many libraries, such as a web framework or a database driver. Their authors want to record telemetry, but they cannot know where you want it sent. One company uses a hosted monitoring tool, another runs its own store. If every library chose its own destination, you would have a mess. Instead, library authors ask for a Tracer from whatever provider has been set up, record their data, and leave the rest to you. The provider is the single place where you decide.

Difficulty 2: settings must be in one place

Telemetry needs many settings: the name of your service, how many traces to keep, how often to send metrics, and which formats to use. Scattering these across the code would make them hard to find and easy to get wrong. A provider collects the settings for a signal in one place, set once at the start of the program.

Difficulty 3: the code must work even when nothing is set up

A library may be used in a program that has no OpenTelemetry set up at all. It must not crash. When no provider has been configured, the API quietly gives out no-op tools, which do nothing at all, at almost no cost. This lets library authors add instrumentation freely, and lets application owners switch it on later by setting up providers, with no change to the libraries.

Difficulty 4: shutdown must not lose data

Telemetry is usually collected in memory and sent in batches. When a program stops, the last batch may still be waiting. Someone must be in charge of sending that last batch and cleaning up. The provider owns the whole pipeline behind it, so it is the natural place to say “send everything now” and “stop”.

Four words explain where providers stand: API, SDK, resource and instrumentation scope.

API and SDK

OpenTelemetry splits its software into two layers. The API is a small set of interfaces, which are lists of calls that code may make, such as “start a span” or “add one to a counter”. The API does very little by itself. The SDK is the full implementation. It contains the engine that stores spans, adds up numbers, batches records and exports them. Providers are part of the SDK.

The reason for this split is that library authors should depend only on the light API, so that they do not force a heavy engine on every user. Application owners choose an SDK and set it up. If no SDK is present, the API does nothing. If an SDK is present and registered, the same API calls suddenly do real work.

flowchart TD
Lib["Your code and libraries"] --> API["OpenTelemetry API: Tracer, Meter, Logger"]
API --> Reg{"Is an SDK registered?"}
Reg -- "No" --> Noop["No-op: calls do nothing"]
Reg -- "Yes" --> SDK["SDK providers do the real work"]
SDK --> Out["Processing and export"]

Resource: who is sending

A resource is a set of labels that describes the thing producing telemetry: the service name, its version, the environment, the machine, the cloud region. It is attached to every span, metric and log the program sends, so that the backend knows who sent them. All three providers need a resource, and it should be the same for all of them.

Instrumentation scope: who is recording

Telemetry also needs to say which piece of code recorded it. This is called the instrumentation scope. It has a name, such as the name of a library or of a module in your application, and optionally a version. When your code asks a provider for a Tracer, Meter or Logger, it passes this name. The name then appears with every item the tool records, so you can tell whether a span came from the web framework or from your own checkout module.

Why the two layers protect you

Suppose a popular library hard-wired a heavy engine. Every program that used the library would carry that engine, even programs that never wanted telemetry, and two libraries with two engines would fight each other. By depending only on the thin API, libraries stay light and neutral. The application owner alone chooses the engine, and that choice applies to every library at once. This split is the quiet reason why OpenTelemetry can be adopted slowly, one service at a time, without a big rewrite.

Semantic versions and stability

The API is kept very stable on purpose, because many libraries depend on it. The SDK may change more often, since it holds options, processors and exporters. When you upgrade, you will usually be safe moving the SDK forward while the API stays steady, but still read the release notes, because the maturity of each signal is not the same in every language.

All three providers follow the same blueprint. Learn it once, and you understand them all.

Four jobs, repeated three times

Every provider does four jobs. First, it hands out tools: you ask it for a Tracer, Meter or Logger by scope name, and it returns one. Second, it holds the resource that identifies your service. Third, it owns a processing pipeline: the steps that take what the tool records and prepare it for export. Fourth, it manages life: it can be asked to send everything it holds, called a flush, and to stop completely, called a shutdown.

flowchart LR
Req["Your code asks for a tool by scope name"] --> P["Provider"]
P --> Tool["Tracer, Meter or Logger"]
P --> Res["Holds the Resource"]
P --> Pipe["Owns the processing pipeline"]
P --> Life["Controls flush and shutdown"]

What differs between the three

The blueprint is the same, but the pipeline inside each provider is different, because the signals behave differently. Traces are made of individual spans, so the TracerProvider has a sampler that decides which traces to keep, and span processors that batch and send them. Metrics are numbers that need adding up over time, so the MeterProvider has views that decide how to summarise them, and readers that collect the summaries. Logs are individual records, so the LoggerProvider has log record processors that batch and send them.

A note on timing

Providers are normally configured when they are created. You decide the resource, the sampler, the processors and the exporters at that moment. Some languages let you add a processor later, but the cleanest habit is to build the whole provider in one place and then use it for the life of the program. Changing the pipeline on a live provider is unusual and easy to get wrong.

Creating a provider step by step

Why a provider is built once

A provider is not a light object. It starts background workers, opens network connections and keeps queues and summaries in memory. Building one is like opening a branch office, and you would not open a new branch for every customer. Build it at startup and keep it until the program ends.

The TracerProvider is the oldest and best known of the three. Its pipeline decides which traces live and how they leave.

What it contains

The sampler

A busy system creates far more spans than it is worth storing. The sampler is the gatekeeper. It looks at the start of each span and says “record this” or “do not record this”. The most common production choice is the parent-based sampler wrapped around a ratio sampler. At the start of a trace, with no parent, the ratio sampler keeps a chosen share, such as one trace in ten. For later spans, the parent-based part simply follows the decision of the parent, so that a trace is either kept whole or dropped whole. The default, if nothing is chosen, is to follow the parent and to keep root spans.

The batch processor and its defaults

Sending each span over the network immediately would be wasteful and slow. The batch span processor holds finished spans in a queue and sends them in groups. Its standard settings are a queue of about two thousand spans, a delay of about five seconds between sends, and at most about five hundred spans per group. When the queue is full, new spans are dropped rather than slowing your application. You can tune these through settings, but the defaults are a good start.

flowchart LR
T["Tracer starts a span"] --> S{"Sampler"}
S -- "record" --> Sp["Span is created and recorded"]
S -- "drop" --> D["Span is not recorded"]
Sp --> E["Span ends"]
E --> BP["Batch span processor queue"]
BP --> X["Span exporter"]
X --> O["Collector or backend"]

Where propagation fits

One thing the TracerProvider does not control is propagation, the passing of trace identity between programs through headers. Propagators are set separately through the global part of the API. They work with the context rather than with the provider, but they are essential for traces to join across services, so remember to set them when you set up tracing.

Span processors in more detail

A span processor gets two chances to act on a span: once when the span starts, and once when it ends. Most processors only care about the end, because a span is complete then and can be exported. A processor that acts at the start can, for example, copy a value from the context into the span, such as the tenant name of the current customer. You may chain several processors on one provider. A common chain is one that adds extra attributes, followed by one that batches and exports.

The two built-in export processors differ in a simple way. The simple one exports each span the moment it ends, on the thread that ended it. This is easy to understand and never loses data in a crash, but it slows the application and so is only for tests and learning. The batch one is the production choice. It keeps its own background worker and a queue, so the application never waits for the network.

Span limits explained

A span can collect attributes, events and links as it runs. Without a ceiling, a loop that adds an attribute on every pass could grow one span to enormous size. Span limits set a maximum number of each, and a maximum length for text values. A common default for attributes is about one hundred and twenty-eight per span. Extra items are quietly discarded, and the span carries a count of how many were dropped. This protects memory, and it also protects your backend from oversized messages.

Choosing a sampling strategy

Head sampling, done in the provider, is fast but cannot know whether a trace will turn out slow or failed. Tail sampling, done in the Collector, can know, but needs more memory and a place where all spans of a trace arrive together. Many teams combine a modest head ratio with tail rules for errors.

The Tracer is what your code actually holds. Understanding how to get one, and how to name it, avoids a surprising number of mistakes.

Getting a tracer

To record a span, your code asks the TracerProvider for a Tracer, giving a name. The name should identify the code that is doing the instrumenting, not the whole application. A library uses its own package name. In an application, a team may use the name of a module, such as the checkout module or the payment module. Optionally, the request includes a version, and in newer versions of the specification, a schema address and a few attributes.

Asking for the same Tracer twice with the same details returns the same one, or an equivalent one, so you may fetch a tracer wherever it is convenient. It is also cheap. Tracers are safe to use from many threads at once, and you normally keep one in a variable for each module.

What the Tracer creates

The Tracer starts spans. A span has a name, a start time, an end time, a status, a kind such as server or client, and attributes. A new span usually becomes a child of the span that is current in the context, which is how the family tree of a trace is built. When the span ends, it is handed to the provider’s processor for export.

Why the name matters

The tracer’s name travels with every span as its instrumentation scope. In your tracing tool, you can then filter by the scope to see only spans from a certain library, or to find which module is producing too many spans. A good name makes later investigation easy, and a vague name, such as “app”, wastes this benefit.

The timeline of one span

Span kinds and why they matter

Every span has a kind that says its role in a conversation between programs. A server span covers handling a request that came in. A client span covers sending a request out. An internal span is work inside one program. Producer and consumer spans describe sending and receiving messages through a queue. Backends use these kinds to draw service maps and to compute request rates, so choose them honestly.

Span status and errors

A span can end as unset, ok or error. Mark a span as an error when the operation failed, and record the failure details as an event on the span. Avoid setting error for normal outcomes such as a “not found” answer to a search, since this fills dashboards with false alarms. Keep the rule simple: an error means the work did not do what it was meant to do.

Metrics are numbers that must be added up over time. The MeterProvider therefore has a different shape: readers and views instead of samplers and span processors.

Why metrics need a different pipeline

A trace records every single request. A metric does not. If your shop handles ten thousand orders a minute, it would be silly to send ten thousand separate numbers. Instead, the SDK aggregates them, which means it adds them up in memory and keeps only a summary, such as “ten thousand orders in this minute”. The MeterProvider owns this summarising machinery.

What it contains

Readers: push and pull

A metric reader is what triggers collection. In the push style, a periodic reader wakes up on a timer, collects all summaries, and hands them to an exporter that sends them out. In the pull style, nothing is sent until an outside system, such as a Prometheus server, comes and asks. The provider can have several readers at once, for example one pushing to a Collector and one serving a pull endpoint. Each reader gets its own copy of the data.

Views: reshaping the data

A view is a rule that says “for metrics that match this description, do it this way”. It can match by instrument name, by type, or by the scope that created it. The rule can rename the metric, change how it is aggregated, set the histogram bucket boundaries, or keep only certain attributes. Views are the main tool for controlling cost and the number of series, because you can remove a high-detail attribute such as a user identifier before it multiplies your data.

flowchart LR
M["Instruments record measurements"] --> V["Views reshape and filter"]
V --> A["Aggregation in memory"]
A --> R1["Periodic reader, every N seconds"]
A --> R2["Pull reader, on request"]
R1 --> E1["Metric exporter"]
R2 --> E2["Prometheus endpoint"]
E1 --> B["Collector or backend"]

Cardinality in plain words

Cardinality is the number of different combinations of attribute values a metric has. A counter of orders with an attribute for the country has as many series as there are countries, perhaps a hundred. Add an attribute for the customer identifier, and it explodes into millions. Each series uses memory in your program and money at your backend. The cardinality limit and views exist to protect you from this.

Temporality: cumulative and delta

Imagine a counter that has counted one hundred orders so far. A cumulative report says “the total since the program began is one hundred”. A delta report says “since my last report, there were twelve more”. Different backends prefer different styles. Prometheus-style systems expect cumulative totals. Some commercial systems prefer deltas because they use less memory in the application. The exporter or reader can be told which style to use, and the SDK handles the arithmetic.

Histograms and bucket boundaries

A histogram counts how many measurements fell into each range, called a bucket. For request durations, the ranges might be under five milliseconds, five to ten, ten to twenty-five, and so on. From the buckets, a backend can estimate the median or the ninety-ninth percentile. The default bucket boundaries suit many cases but not all. If your requests usually take two seconds, default boundaries aimed at milliseconds will put everything in the top bucket and the graph will be useless. A view lets you set boundaries that match your reality. Newer exponential histograms adjust their ranges automatically and are a good option when your backend supports them.

What happens when a reader collects

On each collection, the reader asks the provider for the current summaries from every instrument. Synchronous instruments simply report what they have added up. For observable instruments, the callbacks run at that moment, so they should be quick and never block. The results are bundled with the resource and scope information and passed to the exporter. Because collection is on a timer, a callback that takes too long delays the whole report.

Several readers, several destinations

Because every reader receives its own copy of the data, a single MeterProvider can feed several destinations. You might push to the Collector every thirty seconds while also exposing a pull endpoint for a local Prometheus scraper. The cost is a little extra memory, since each reader keeps its own state, especially in cumulative style.

A Meter is a small workshop that makes instruments. Each instrument is a specialised measuring tool.

Getting a meter

Just like a Tracer, a Meter is requested from its provider with a scope name and optional version. The Meter itself records nothing. It creates instruments, and you use the instruments to record. You usually create each instrument once, near the start of the module, and keep it, because creating an instrument again and again is wasteful.

The instrument family

Synchronous and asynchronous

A synchronous instrument is used by your code at the moment something happens: an order arrives, so you add one to the counter. An asynchronous instrument, also called observable, is different. You give it a small function, a callback, and the SDK calls it whenever a reader collects. This suits things that you read rather than cause, such as the current memory use or the size of a cache.

Choosing the right one

Ask three questions. Do I count events as they happen? Use a counter. Do I care about how long or how large each event was? Use a histogram. Do I read the state of something at a moment? Use a gauge or an observable. A histogram is the right choice for durations, because averages hide slow requests, while a histogram lets you see the slowest few percent.

flowchart TD
Q["What do you want to record?"] --> A{"Counting events?"}
A -- "Only increases" --> C["Counter"]
A -- "Can go up and down" --> U["UpDownCounter"]
Q --> B{"How big or how long each time?"}
B --> H["Histogram"]
Q --> S{"Current state at a moment?"}
S --> G["Gauge or Observable"]

Units, names and descriptions

Each instrument has a name, a unit and a description. The unit, such as seconds or bytes, lets backends show graphs with the right labels and convert values sensibly. Use base units where possible, for example seconds rather than milliseconds, because the conventions prefer them and tools expect them. A short description helps the next engineer understand what a mystery number means.

Attributes on measurements

When you record a measurement, you may attach attributes, such as the payment method or the HTTP status class. Each unique combination of values becomes its own series. Choose attributes whose values come from a small, known list. A status class such as success or failure is excellent. A customer name is not. The next chapters show how views and the cardinality limit protect you if someone forgets this rule.

Logs are the youngest signal in OpenTelemetry, and the LoggerProvider is built with one special idea in mind: you probably already have a logging library.

Why logs are different

For traces and metrics, OpenTelemetry offered a brand-new way of working. But logging already existed for decades, with libraries in every language, and millions of lines of code already call them. Asking everyone to rewrite all of that would never work. So OpenTelemetry took a gentler path. It lets you keep your current logging library and adds a bridge that carries those log lines into the OpenTelemetry pipeline.

What the LoggerProvider contains

The bridge, in plain words

The part that connects your logging library to OpenTelemetry is called an appender or handler, depending on the language. It listens to the logging library. Each time your code writes a log line, the appender converts it into an OpenTelemetry log record and hands it to a Logger from the LoggerProvider. The log record then goes through the processors and exporters like any other data.

The API that appender authors use is called the Logs Bridge API. It is deliberately meant for the authors of these bridges, not for everyday application developers. As an application developer, you keep writing logs the way you always did, and you only install and enable the appender for your library.

flowchart LR
App["Your code writes a log line"] --> Lib["Existing logging library"]
Lib --> Ap["OpenTelemetry appender or handler"]
Ap --> L["Logger from LoggerProvider"]
L --> LP["Log record processor"]
LP --> LE["Log record exporter"]
LE --> B["Collector or backend"]

The big benefit: automatic trace links

When the appender builds a log record, it can copy the current trace identifier and span identifier from the context into it. This means each log line is tied to the exact request that produced it. In your backend, you can open a slow trace and see the log lines from that very request, or start from an error log and jump to its trace. This link is the strongest reason to send logs through OpenTelemetry.

Log record processors

The LoggerProvider is the home of log record processors, which work like span processors. A simple one exports at once, and a batch one groups records in a background queue. Log processors also have a handy extra use: because they see every record before export, they can modify or filter them. A processor can add attributes, mask a sensitive field, or drop records that are not worth sending. This is a useful safety net, though cleaning at the source remains better.

Volume is the main challenge

Traces are sampled and metrics are summarised, but logs are usually sent one by one. A busy service may write thousands of lines each second, so logs are normally the biggest cost of the three signals. Good habits help a lot: log at the right severity, avoid logging inside tight loops, write structured data rather than long sentences, and do not log the same fact at many layers. Sending only warnings and errors from busy modules, while keeping debug lines local, is a common compromise.

Structured logs and attributes

A structured log keeps details as separate named attributes, such as the order number and the payment method, instead of burying them in a sentence. Structured logs are much easier to search and count in the backend. When the appender converts a log line, it maps the fields your library already provides into attributes, so adopting structured logging in your library pays off directly in OpenTelemetry.

The Logger is the tool the appender uses. Knowing what it builds helps you read the logs that arrive at your backend.

The log record

A log record is the unit of the logs signal. It carries a time stamp, a severity (such as debug, info, warning, error), a body with the message, a set of attributes, the trace and span identifiers if a span was active, and the resource and scope supplied by the provider and logger. Because the structure is standard, any backend can show logs from any language in the same way.

Events

OpenTelemetry also describes events, which are log records with a defined name and a standard set of attributes, used for things such as a user action or a browser event. They travel through the same LoggerProvider. Their exact API and naming have been evolving, so check the current documentation for your language before building on them.

Severity and filtering

Most logging libraries already let you choose a minimum severity. That filtering happens before the bridge, so debug lines that your library drops never reach OpenTelemetry. This is the first and cheapest place to control volume. Later, a processor or the Collector can filter further.

Correlation in practice

For the trace link to work, the log must be written while a span is active in the context. A log line written in a background thread that has lost the context will carry no identifiers. In busy programs, make sure the context is passed along when work moves between threads, and check a few sample logs to confirm that identifiers appear. If they do not, the problem is usually context propagation, not the LoggerProvider.

Severity numbers

OpenTelemetry stores severity as a number in ranges, so that different libraries’ levels can be compared. Trace and debug sit at the low end, then info, warning, error and fatal. The appender maps your library’s levels onto this scale. This is why an error from one language and an error from another look the same in your backend, even though the libraries named them differently.

The three providers are separate, but they must tell the same story about who you are.

Why sharing matters

Each provider attaches a resource to its data. If the TracerProvider says the service is “checkout” and the MeterProvider says “checkout-service”, your backend will treat them as two different services. You would lose the ability to move from a slow trace to the matching metrics or logs. Correlation depends on identical identity.

How to share it safely

The simplest approach is to build the resource once at startup and pass the same object to all three providers. Most SDKs also read the resource from environment settings, namely the service name and a list of extra attributes. When the environment supplies identity, every provider reads the same source and agrees without extra effort. Autoconfiguration and agents do this for you.

What belongs in a resource

flowchart TD
R["One Resource: service name, version, environment, host"] --> TP["TracerProvider"]
R --> MP["MeterProvider"]
R --> LP["LoggerProvider"]
TP --> T["Traces carry the resource"]
MP --> M["Metrics carry the resource"]
LP --> L["Logs carry the resource"]
T --> B["Backend can join all three"]
M --> B
L --> B

Service name and instance identity

The service name groups all copies of a service together. The service instance identifier separates one copy from another, which helps when only one pod misbehaves. Many SDKs generate a random instance identifier at startup if none is provided. If each provider generated its own random value, signals from the same process would look as if they came from different instances, which is one more reason to build the resource once and share it.

Detectors

Writing every attribute by hand is tedious. Resource detectors are small helpers that discover facts from the environment, such as the host name, the operating system, the container identifier or the cloud region, and add them to the resource. Using detectors keeps your identity accurate and saves effort. Review what they add, since some details can be too long or sensitive for your taste.

Creating a provider is not enough. Libraries must be able to find it, and that is the job of registration.

The global registry

Libraries are written long before your application exists, so they cannot receive your provider as an argument. Instead, the API keeps a global slot for each signal. When you register a provider, it goes into that slot. When any code asks the global API for a Tracer, Meter or Logger, the answer comes from the registered provider. This is how a library you did not write ends up using your configuration.

Set once only

The global slot can usually be set only once. A second attempt is ignored or logs a warning, depending on the language. This prevents two parts of a program from fighting over the global provider, but it also means the order of start-up matters. Register early, before libraries start asking for tools.

Passing providers directly

Registration is not the only way. You may also keep the provider in your own variable and ask it directly for tools, without touching the global slot. This is common in tests, in programs that run several isolated parts, and in code that avoids global state on purpose. Instrumentation libraries usually allow you to pass a provider to them, so they can use yours.

What happens without registration

If nothing is registered, the API supplies no-op tools. Spans have valid identifiers when a parent exists, so context can still pass through, but nothing is recorded or exported. This is safe, but silent. A quick check during development is to confirm that a first span, metric or log really reaches your backend or a console exporter.

Dealing with start-up order

Because registration happens once and early, the order of start-up work matters. Frameworks often create their instrumented clients while loading. If those clients ask for a tracer before your provider is registered, they may keep a no-op tracer for life. A reliable approach is to put telemetry set-up at the very top of the program entry point, or to load it before the application code, as agents and launchers do. In some environments, a proxy tracer is handed out that later switches to the real one, but you should not depend on it unless the documentation of your language promises it.

A short checklist for registration

Data waiting in memory is lost when a program ends suddenly. The lifecycle controls decide whether your last seconds of telemetry arrive.

The three stages

Flush versus shutdown

ForceFlush means “send everything you are holding, now, but keep running”. It is useful before a short pause, such as the end of a serverless function call. Shutdown means “flush, then stop for good”. After shutdown, the provider no longer accepts new data. Both normally accept a time limit, so a stuck network cannot freeze your exit forever.

Who needs this most

Long-running servers lose very little, because the batch delay is a few seconds and they usually stop gracefully. The programs at greatest risk are short-lived ones: command-line jobs, scheduled tasks, serverless functions and containers that are killed quickly. If such a program ends before the next batch send, its last data is gone. The fix is to flush and shut down all three providers before exit, and to give the platform enough time at termination.

sequenceDiagram
participant App as Application
participant P as Provider
participant Pr as Processor
participant Ex as Exporter
participant Bk as Backend
App->>P: Create and register at startup
App->>P: Record data while running
P->>Pr: Pass data to the processor
Pr->>Ex: Send batches on timer
Ex->>Bk: Deliver
App->>P: Shutdown on exit
P->>Pr: Flush remaining data
Pr->>Ex: Final batch
Ex->>Bk: Deliver last data
P-->>App: Done

Serverless and short-lived programs

A serverless function may be frozen the moment it returns its answer. A batch timer set for five seconds will then never fire, and spans wait in the queue until the function is thawed or destroyed. The usual fixes are to flush all providers before the function returns, or to use a platform extension or Collector that receives data immediately. Flushing adds a small delay to each call, so measure the cost. Command-line tools and scheduled jobs need the same care, with a flush at the end of the main routine.

Graceful termination in containers

When a container platform wants to stop your service, it first sends a polite stop signal and waits a grace period, often thirty seconds, before killing it. Your program should catch the signal, stop accepting work, finish current requests, then flush and shut down the providers. If the grace period is shorter than your flush timeout, the platform will cut you off in the middle. Make the timeouts agree: the export timeout must be shorter than the grace period.

What shutdown does not do

Shutdown flushes what the provider holds. It cannot recover data that was already dropped from a full queue, and it cannot send data if the network is down. It is a best effort. For stronger guarantees, rely on a Collector with persistent storage close to the application.

Let us follow one customer request through all three providers, from the first call to the backend screen.

The story of one request

A customer presses “Pay”. The web framework, instrumented by a library, asks the global API for a Tracer. The API hands over the tracer made by the registered TracerProvider. A span starts, the sampler says yes, and the span becomes the current span in the context. While the request runs, your code increases an order counter from a Meter, and a database call adds a child span. A payment failure causes a log line, and the appender turns it into a log record that already carries the trace and span identifiers.

Three parallel pipelines

Each signal now travels its own pipeline inside its own provider. Spans finish and enter the span processor queue. The counter value sits in the aggregation memory until the periodic reader wakes up. The log record enters the log record processor queue. Timers then push each batch to its exporter. All three exporters usually point to the same OpenTelemetry Collector, a separate program that receives, processes and forwards telemetry, which then sends everything on to the backend.

flowchart LR
Req["Customer request"] --> Fw["Instrumented code"]
Fw --> TP["TracerProvider: sampler, span processor, exporter"]
Fw --> MP["MeterProvider: views, reader, exporter"]
Fw --> LP["LoggerProvider: log processor, exporter"]
TP --> Col["OpenTelemetry Collector"]
MP --> Col
LP --> Col
Col --> Bk["Backend: traces, metrics, logs"]

Where the three signals touch

The providers do not talk to each other. They are connected by two shared things: the resource, which identifies the service, and the context, which carries the current span. The appender reads the context to stamp logs with trace identifiers. Some metrics features, called exemplars, copy a trace identifier into a metric sample, so a spike on a graph can lead you to a sample trace. Everything else is independent.

Why independence is a strength

Since the pipelines are separate, a problem in one rarely hurts the others. If the log exporter cannot reach the network, spans and metrics still flow. You can also tune each one differently: sample traces at ten percent, collect metrics every thirty seconds and keep all error logs.

The idea is the same everywhere, but the class names and the set-up style change from language to language.

Common names

The names below are accurate to the best of our knowledge, but SDKs evolve quickly, so confirm them in the documentation of your version.

Three ways to set up

Environment variables

OpenTelemetry defines a standard set of environment variables, so the same names work across languages. They set the service name, resource attributes, which exporter to use for each signal, the sampler, the endpoint of the Collector, and the export timing. This is a big help in containers, because operations teams can change behaviour in the deployment files without touching the program.

Which to choose

Begin with autoconfiguration or an agent, because it gives you correct defaults and a shared resource for free. Move to manual set-up only when you need something special, such as custom views, a custom sampler, several exporters, or several providers in one process.

flowchart TD
Start["Need telemetry"] --> Q1{"Can you attach an agent or launcher?"}
Q1 -- "Yes" --> Z["Use zero-code: providers built for you"]
Q1 -- "No" --> Q2{"Are the standard options enough?"}
Q2 -- "Yes" --> A["Use autoconfiguration with environment variables"]
Q2 -- "No" --> M["Build the three providers by hand"]
Z --> Fin["Add custom spans and metrics where needed"]
A --> Fin
M --> Fin

What can go wrong with autoconfiguration

Autoconfiguration is convenient, but mistakes in environment variables are easy to make and hard to spot. A misspelt variable name is simply ignored. A wrong endpoint leads to silent export failures. A sampler value in the wrong format may fall back to a default. When behaviour surprises you, print the effective configuration, which many SDKs can do through their diagnostic logging, and compare it with what you intended.

Mixing agents and manual code

Teams often attach an agent and then add a manual provider by habit. This can lead to double registration, duplicated spans or two competing exporters. Decide who owns the providers. If an agent owns them, your code should use only the global API. If your code owns them, do not also attach an agent that builds its own set.

Telemetry must never harm the application it watches. Providers are designed with this promise in mind.

Cost on the hot path

The hot path is the code that runs for every request. Starting a span, adding to a counter and writing a log should cost very little. Providers keep this small by doing the heavy work, such as batching, serialising and network sending, on background threads. Your request thread only places a small item in a queue and moves on.

Bounded memory

Every queue is limited. When the span queue or the log queue is full, new items are dropped instead of making your service wait. This is a deliberate trade: losing a little telemetry is better than slowing customers. Metrics are naturally bounded because they are summaries, and the cardinality limit stops them from growing without end.

Reducing volume

Scaling out

When you run hundreds of copies of a service, each has its own providers, so each sends its own data. This is fine, but the number of connections to your backend can grow large. A common design is to send everything to a nearby Collector, which batches and forwards on. That keeps the application’s set-up simple and moves heavy decisions out of the application.

What if the destination is down

Exporters retry for a limited time and then drop data. The queue absorbs short outages. For longer outages, a Collector with a persistent queue gives extra safety. In every case the application keeps running normally, and that is the essential promise.

flowchart LR
H["Hot path: request thread"] --> Q["Bounded queue"]
Q -- "full" --> Dr["Drop new items, count them"]
Q --> W["Background worker thread"]
W --> Ex["Exporter with timeout and retry"]
Ex --> Col["Collector"]
Col --> Bk["Backend"]

Thinking about the Collector as a shock absorber

A local Collector sits near your application, often on the same machine or in the same cluster. Applications send to it over a fast local link and return to work. The Collector can retry against a distant backend for much longer than an application would be willing to. It can also add common attributes, remove sensitive ones, batch efficiently and balance load across several backends. For these reasons, many production guides recommend always sending through a Collector, and keeping the provider settings in the application as simple as possible.

Measuring the overhead

The cost of instrumentation is usually small but not zero. Before turning on broad tracing in a very hot service, measure latency and CPU with and without it, under realistic load. Pay attention to the number of spans created per request: a hundred tiny spans cost much more than ten meaningful ones. A useful rule is to create spans for operations that matter to an investigation, such as calls across a network boundary, and not for every small function.

Memory and the queue sizes

Each queued span or log record holds attributes in memory. A queue of two thousand small spans is trivial, but a queue of two thousand huge spans is not. Combine queue sizes with span limits and attribute discipline. If you raise the queue size to avoid drops, you are also raising worst-case memory use, so test it in a staging environment under pressure.

Telemetry is a copy of your system’s inner life, so it deserves the same care as any sensitive data.

Security of the data

Spans, metrics and logs can contain customer names, addresses, tokens and query text. Treat attributes as potentially sensitive. Never put passwords, keys, full card numbers or session tokens into them. Avoid recording full request bodies. Where you must keep an identifier, prefer a hashed or shortened form. Redaction in the Collector gives a second layer of defence.

Security of the connection

Exporters send data across the network, so use encrypted connections to the Collector or backend. Where the Collector or backend needs authentication, supply headers or certificates through the exporter’s settings, and keep those secrets in a secret store, not in the code or in plain deployment files. Restrict who can reach the Collector endpoint.

Security of the set-up

Remember that the environment variables controlling telemetry can also change behaviour. An attacker with the power to change them could redirect your data to another endpoint. Limit who can edit deployment settings, and review them like any other production configuration.

Monitoring the monitors

You should watch the health of the providers themselves. Useful signs include the number of spans dropped from full queues, the number of failed exports, export latency, queue fill level, and the memory used by metric series. Many SDKs expose internal diagnostics through their own logging. Turn these on in a staging environment, and alert on repeated export failures in production.

Cost control

Most monitoring bills grow from three causes: too many traces kept, too many metric series, and too many verbose logs. Each of the three providers has a matching control, as the previous chapter showed. Review volume regularly, and treat a sudden jump as a sign that a new attribute or a new debug line has slipped in.

Retention and access to telemetry data

Think about how long you keep telemetry and who may read it. Debug-level logs from last year rarely help anyone, but they may hold private details. Set retention periods in your backend, limit access by role, and treat exports of telemetry for outside parties with the same care as any customer data export. Where laws about personal data apply to you, check with your legal and security teams what telemetry may contain and how long it may stay.

Every design choice has a price. Here is an honest view of the provider model.

Key trade-offs

Reading the trade-offs correctly

There is no single best setting. A small internal tool may sample everything, because traffic is low. A large retail site may keep one trace in a hundred and rely on tail sampling in the Collector for errors. The right answer depends on traffic, budget and how quickly you need to answer questions during an incident.

When the provider model feels heavy

For a tiny script, setting up three providers can feel like too much. In such a case, use the all-in-one helper of your language or a launcher, or turn on only the one signal you need. The model scales down by simply leaving things out, and it scales up by adding readers, processors and several providers. That flexibility is worth the small learning cost.

These lessons come from the way teams actually run OpenTelemetry in production.

Helpful patterns

Anti-patterns

Best practices at a glance

Common mistakes and quick fixes

Testing code that uses providers

Good tests prove that the right spans, metrics and logs are produced. Build a test-only provider with an in-memory exporter and a simple processor, pass it to the code under test, run the action, and then read what the exporter captured. Because the simple processor exports at once, no waiting is needed. Check names, attributes, status and parent-child links. For metrics, use a manual reader that you can trigger on demand, instead of waiting for a timer. This approach avoids global state, which is another reason to let code accept a provider as an input where practical.

Naming and consistency across teams

In an organisation with many teams, agree on a short guide: how services are named, which environment names are used, which attributes are required on every resource, and which attributes are forbidden. Put this guide in shared startup code or deployment templates, and use a checker in review to catch violations. A small amount of agreement early prevents the painful clean-up of thousands of inconsistent dashboards later.

Upgrading safely

Telemetry libraries change over time, and parts of the logs and event areas have been moving fast. Pin versions, read release notes before upgrading, and upgrade API, SDK and instrumentation packages together. After an upgrade, run your console-exporter check again to prove all three signals still flow, and watch dashboards for changes in names or units.

Let us watch the three providers at work in an imagined online shop called ShopFast. The company and numbers are made up for teaching.

Day one: setting up

The platform team sets standard environment variables in a shared template: the Collector address, the sampler as parent-based with a ratio of one in ten, and a metric interval of thirty seconds. Each service adds only its own name. The result is a consistent resource across all three signals, with no code changes.

Day thirty: a surprise bill

A developer adds a customer identifier as an attribute on a checkout counter. Within a day the number of metric series rises sharply, and the backend bill follows. The team adds a view in the provider settings that drops that attribute from the metric, and adds a code review rule against unbounded attributes. The cardinality limit also protects the service memory while the fix is prepared.

Day sixty: a lost final error

A nightly invoice job sometimes fails, but its last error never reaches the backend. The cause is that the job exits right after logging. The fix is a shutdown step that flushes the LoggerProvider and the TracerProvider before exit, and a few seconds of grace time at container termination. Errors from the job then appear reliably, each linked to its trace.

Day ninety: a single investigation

On a busy evening, the checkout latency graph rises. An on-call engineer clicks a point on the histogram and opens an exemplar trace. The trace shows a slow call to the payment service. From the span, she opens the correlated logs, which show “payment gateway timeout” with the same trace identifier. The problem is found in minutes, because three providers produced three signals that share one identity.

How the industry uses these ideas

The provider model is the base of OpenTelemetry SDKs used by many large organisations and by the commercial monitoring vendors that accept OpenTelemetry data. Cloud platforms and managed observability services publish their own distributions that preconfigure the providers with the right exporters. Teams commonly run a fleet of Collectors, and tune sampling and views centrally. Details differ by vendor, so always check your vendor’s documentation for supported versions and exporters.

sequenceDiagram
participant Eng as On-call engineer
participant Dash as Dashboard
participant Tr as Trace view
participant Lg as Log view
Eng->>Dash: Notice latency spike on histogram
Dash->>Tr: Open example trace from the spike
Tr-->>Eng: Slow payment call found
Eng->>Lg: Open logs for the same trace id
Lg-->>Eng: Gateway timeout message
Eng->>Eng: Root cause identified in minutes

Lessons from the ShopFast story

A second picture: a small team

Consider a three-person start-up with one service and no platform team. They choose autoconfiguration with a hosted backend. They set the service name, the backend address and a token through environment variables, and they leave sampling at its default. For their traffic, that is enough. When traffic grows, they change only the sampler variable. They never touched the provider code, yet they benefited from every concept in this guide. Understanding providers still matters, because it tells them which variable to change and where to look when a graph is blank.

Three desks, one identity, and a careful closing routine: that is the whole story in short.

The TracerProvider, MeterProvider and LoggerProvider are the SDK’s central desks for traces, metrics and logs. Each hands out the tool your code uses and holds the settings that decide how data is processed and sent. Code and libraries talk only to the API, and the registered providers make the calls real. If you set them up early, give them one shared resource, choose sensible processors and limits, and flush on exit, you get dependable and connected telemetry at a cost you can control.