What Is the OpenTelemetry SDK?
A plain-English guide to the engine that lives inside your application and creates, shapes, and ships traces, metrics, and logs, and to how it fits into real production systems.
Picture this. It is a busy evening, and customers of an online store are complaining that checkout feels slow. Some say the page hangs. Others say it works fine. Your team opens a dashboard and sees that the website is “up”, yet something is clearly wrong. Which of the twenty services behind the checkout button is the slow one? Is it the payment service, the database, or a network hop in between?
To answer that, every service needs to tell its own story: what it did, how long each step took, and what went wrong. The data that carries this story is called telemetry. The piece of software that creates, organizes, and sends this telemetry from inside your running program is the OpenTelemetry SDK.
In this guide, you will learn what the SDK is, why it exists, how its parts fit together, what happens inside it when your code runs, and how to use it safely at the scale of millions of requests. No prior knowledge is needed. Every new word is explained the first time it appears.
1What Is OpenTelemetry, and Where Does the SDK Fit?
Before we zoom in on the SDK, we need the big picture. The SDK is one piece of a larger project, and its job only makes sense once you see the whole journey of a piece of data.
First, what is telemetry?
Telemetry means data that a system sends out about itself. A heart monitor in a hospital is a good example. It does not fix the patient, but it reports pulse, blood pressure, and oxygen so doctors can act. In software, telemetry reports things like “this request took 240 milliseconds”, “the payment call failed”, or “memory use is rising”.
The ability to understand what is happening inside a system just by looking at the data it produces is called observability. Teams use it to find bugs, fix slowdowns, plan capacity, and prove that a new release did not make things worse.
So what is OpenTelemetry?
OpenTelemetry (often shortened to OTel) is an open-source, vendor-neutral framework for creating and moving telemetry. It is hosted by the Cloud Native Computing Foundation (CNCF), the same foundation that hosts Kubernetes. It gives the industry one common set of rules, interfaces, tools, and a shared data format, so that any application can describe itself in the same way no matter which monitoring product is used later.
One point confuses almost every beginner, so let us say it clearly. OpenTelemetry is not a place where telemetry is stored, and it is not a dashboard. It does not draw charts. It is the set of pipes and standards that carry data from your code to whichever tool does the storing and the charting.
And the SDK?
The SDK (Software Development Kit) is the part of OpenTelemetry that actually runs inside your application process. It is a library written for each programming language, such as Java, Python, Go, JavaScript, .NET, Ruby, PHP, and Rust. When your code says “a request just started” or “record this number”, the SDK is the part that decides what to keep, adds useful labels such as the service name, gathers data into batches, and sends it away over the network.
Think of a national weather network. The rules for how a thermometer reading must be written down are the standard. The weather station on a hilltop, with its sensors, recorder, and radio, is the SDK. The central weather office that stores all readings and draws the forecast maps is the backend. You can swap the central office for a different one, and the stations keep working exactly as before.
The journey of your data
The diagram below shows the full path. Your code calls the API. The SDK does the real work. Exporters send the data out, often through an optional middle service called the Collector, and finally a backend stores and shows it.
flowchart TB
App["Your Application and Libraries"]
API["OpenTelemetry API (interfaces only)"]
SDK["OpenTelemetry SDK (real implementation)"]
Exp["Exporters (OTLP and others)"]
Col["OpenTelemetry Collector (optional)"]
BE["Observability Backend (storage and dashboards)"]
App --> API
API --> SDK
SDK --> Exp
Exp --> Col
Col --> BE
Read the figure from top to bottom. Your application never talks to the backend directly. It only talks to the API. That simple rule is the reason you can change your monitoring vendor later without touching your business code.
Remember it as pipes and plugs. OpenTelemetry standardizes the plugs and the pipes. The backend you choose is the tap at the end of the pipe.
2The Problem OpenTelemetry Was Built to Solve
Every good standard exists because the world before it was painful. Understanding that pain makes the design of the SDK feel natural instead of random.
Life before a standard
In the early days of monitoring, each vendor shipped its own agent or library. If you used Vendor A, your code was full of Vendor A calls. If you later wanted Vendor B, you had to find and rewrite thousands of lines. Teams call this vendor lock-in, and it made switching so expensive that people simply stayed with tools they no longer liked.
Libraries had it even worse. The author of an HTTP client or a database driver wanted to add tracing, but which vendor should they support? Supporting all of them was impossible, so most libraries supported none, and developers had to patch tracing in by hand.
Imagine travelling between countries when every country has a different wall socket. You need a bag full of adapters, and some devices simply will not charge. A universal plug standard fixes this. OpenTelemetry is the universal plug for monitoring data.
Two competing projects, then one
Two open-source efforts tried to fix this. OpenTracing defined a common tracing interface but did not include a working implementation. OpenCensus, which grew out of work at Google, shipped both an interface and an implementation and also covered metrics. Having two standards created a new kind of confusion. In 2019 the two communities joined forces and created OpenTelemetry, taking the best ideas of both.
How the story unfolded
Vendor-specific agents
Each monitoring product had its own library. Switching products meant rewriting instrumentation code.
OpenTracing and OpenCensus
Two open projects appeared with different designs. Better than lock-in, but still split.
The 2019 merger
The two projects combined into OpenTelemetry under the CNCF, with one specification and one set of language SDKs.
Stability, signal by signal
Traces reached stable status first, then metrics, then logs. Each language SDK matures on its own schedule, so always check the status page for your language.
Broad adoption
Most major monitoring vendors and cloud providers now accept OpenTelemetry data, and many tools build directly on it.
That one sentence is the promise of OpenTelemetry. The SDK is how the promise is kept inside each application.
3The Two Halves: API and SDK
The most important idea in OpenTelemetry is that every language ships two separate packages. Knowing which one does what will save you from the most common design mistakes.
The API: the promise
The API (Application Programming Interface) is a set of names and shapes. It says “you can start a span”, “you can add one to a counter”, “you can emit a log record”. It describes what can be done, but it does not contain the machinery that does it. On its own, the API is a harmless no-op. Calls to it do nothing and cost almost nothing.
The SDK: the delivery
The SDK is the real implementation behind those promises. It creates actual spans, measures real numbers, applies sampling rules, batches data, and exports it. The application owner installs and configures the SDK once, usually at program start.
In a restaurant, the waiter’s order pad is the API. Anyone can write on it, and the format is the same everywhere. The kitchen is the SDK. It decides how the order is cooked, in what sequence, and how fast. A guest writing an order does not need to know how the kitchen is arranged.
| Question | API | SDK |
|---|---|---|
| What is it? | Interfaces and no-op defaults | The working implementation |
| Who depends on it? | Application code and shared libraries | Only the final application (or its startup code) |
| If it is missing | Nothing breaks, data is simply not produced | The API stays no-op and no telemetry leaves the process |
| What does it control? | How code describes events | Sampling, batching, limits, export, resources |
| How often does it change? | Rarely, with strong compatibility promises | More often, as features and options grow |
Why the split matters
Suppose a popular database driver adds tracing. If it depended on a full SDK, it would force every user to accept that SDK’s version, settings, and weight. Because it depends only on the light API, it can ship tracing to everyone. Teams that do not care about telemetry pay almost nothing. Teams that do care install the SDK and instantly see data from the driver, with no change to the driver itself.
You may also hear the word distribution. A distribution is a vendor’s or company’s pre-configured package of the standard SDK, bundled with defaults for their environment. It is still the same SDK underneath, only wrapped for convenience.
If you are writing a reusable library, depend on the API only. Never pull the SDK into a library. The decision of where telemetry goes belongs to the application owner, not to library authors.
4Core Concepts: Signals, Spans, Metrics, and Logs
A signal is a type of telemetry. OpenTelemetry defines three main ones, and each answers a different question about your system.
Traces
Answer “what happened to this one request, and where did the time go?” A trace follows a single request across every service it touches.
Metrics
Answer “how is the system doing overall?” Numbers such as requests per second, error counts, and memory use, summed or averaged over time.
Logs
Answer “what exactly did the program say at that moment?” Timestamped messages, ideally structured and linked to a trace.
Baggage
Not a signal itself. A small set of key-value pairs that travels along with a request so downstream services can read it.
Traces and spans
A span is one timed piece of work, such as “handle the checkout request” or “run this database query”. A trace is the family tree of spans that belong to one request. The first span is the root. Every other span has a parent, so the whole trace forms a tree that shows what called what, and for how long.
Track a parcel that is shipped across the country. The whole journey is the trace. Each stop, such as “picked up”, “sorted at the hub”, “loaded on the truck”, and “delivered”, is a span with its own start and end time. Put the stops together and you can see exactly where the parcel waited too long.
What a span contains
Trace ID and Span ID
A trace ID is shared by every span in the same request. A span ID is unique to one span. The parent span ID links child to parent.
Name, start, and end
A short human-readable name for the operation, plus exact start and end times, so duration can be computed.
Span kind
Says the role of the span: server, client, producer, consumer, or internal. It helps tools draw service maps correctly.
Status
Unset, OK, or Error. It tells you whether the work succeeded, so you can filter for failures quickly.
Attributes
Key-value labels such as the HTTP method, the route, or a customer tier. They make traces searchable.
Events and links
Events mark a point in time inside a span, such as “cache miss”. Links connect a span to other traces, useful in batch or queue work.
Metrics and logs in one minute
A metric is a number measured repeatedly over time. Unlike a trace, which describes one request, a metric describes the crowd. It is cheap to store because many events are squeezed into a few numbers. A log record is a single timestamped message, often with a severity such as info, warning, or error. We will explore both in depth later.
Attributes and semantic conventions
An attribute is a simple key and value, such as the name of the HTTP route and its value. If every team invents its own key names, a dashboard cannot understand them. OpenTelemetry solves this with semantic conventions, which are agreed names and meanings for common things, such as HTTP requests, database calls, messaging, and cloud resources. Following them means your data works with tools and dashboards you did not build.
Before inventing a new attribute name, search the semantic conventions. If a standard name exists, use it. Your future self and every tool in your toolbox will thank you.
5Context and Propagation: How a Trace Crosses Service Borders
A trace is only useful if it stays connected as the request hops from service to service. The glue that does this is called context propagation.
Context: the invisible backpack
Context is a small package of information that follows your code as it runs. Its most important content is “the span that is currently active”. When your code starts a new span, the SDK looks in the context, finds the current span, and makes it the parent. This happens automatically, even when work is passed between threads or async tasks, provided the language integration supports it.
Propagation: handing the backpack to the next service
Inside one process, context lives in memory. Between two services, it has to travel across the network. Propagation is the act of writing the context into the outgoing message (this is called inject) and reading it back on the receiving side (this is called extract). A component called a propagator does this job.
The default standard is W3C Trace Context. It uses an HTTP header named traceparent that carries four parts, separated by dashes: a version, the trace ID, the ID of the calling span, and a flags field that includes the “sampled” bit. A companion header named tracestate can carry extra vendor-specific hints. Older formats such as B3 and Jaeger headers are also available for systems that have not moved yet.
At a music festival, you get a wristband at the gate. Every stage, food stall, and exit recognizes it. A request carries the trace ID like a wristband. Every service it visits can read the band and say “this person belongs to group 4F2A”, so all their activity can be linked together later.
What the journey looks like
sequenceDiagram
participant U as User Browser
participant G as API Gateway
participant O as Order Service
participant P as Payment Service
participant C as Collector
U->>G: HTTP request
G->>O: Forward request with traceparent header
O->>P: Call with same trace ID and a new span ID
P-->>O: Response
O-->>G: Response
G-->>U: Response
G->>C: Export spans using OTLP
O->>C: Export spans using OTLP
P->>C: Export spans using OTLP
Notice two things. First, each service creates its own spans and exports them on its own schedule. Nobody collects the full trace in one place until the backend stitches them together using the shared trace ID. Second, the trace ID stays the same all the way, while every hop gets a new span ID with the previous one as its parent.
Baggage: a note passed along the way
Baggage is a small set of key-value pairs that travels with the request in a separate header. For example, you could attach the customer tier at the front door, and a service five hops away could read it without looking it up again. Baggage is not copied onto spans automatically. Code or a processor must choose to read it and add it as an attribute.
Why traces break
The most common support question about tracing is “why is my trace cut in half?” The causes are almost always simple:
- A proxy, gateway, or message queue strips the
traceparentheader. - One service in the middle has no instrumentation, so it never reads or forwards the header.
- Two services use different propagation formats and cannot read each other.
- A background job starts without restoring the context that was saved with the message.
Never blindly trust trace headers that come from the public internet. A caller can send any trace ID or force the sampled flag on. At your public edge, many teams start a fresh trace and only link to the incoming one.
6Architecture and Components of the SDK
The SDK is not one big blob. It is a small set of cooperating parts, each with one clear job. Learn these names once, and every language’s documentation will feel familiar.
Think of a post office sorting center. A manager hires clerks, a screener decides which parcels are worth recording, a sorting team groups parcels into crates, and a truck carries each crate to its destination. The SDK has the same shape: providers, samplers, processors, and exporters.
The tracing pipeline at a glance
flowchart LR
TP["TracerProvider"] --> T["Tracer"]
T --> S["Span"]
SM["Sampler"] -. decides .-> S
R["Resource"] -. describes .-> S
S --> SP["SpanProcessor (Batch)"]
SP --> Q["In-memory Queue"]
Q --> SE["SpanExporter (OTLP)"]
SE --> OUT["Collector or Backend"]
A tracer creates a span. The sampler decides whether it is worth recording. The resource stamps it with facts about where it came from. When the span ends, a processor places it in a queue, and an exporter sends queued spans away in groups. Let us meet each part properly.
TracerProvider
The factory and the central control room for tracing. It holds the configuration (sampler, processors, resource) and hands out tracers. Most programs create exactly one and register it globally at startup.
MeterProvider and LoggerProvider
The same idea for metrics and logs. A MeterProvider hands out meters that create instruments. A LoggerProvider hands out loggers that emit log records.
Sampler
Decides, at the moment a span starts, whether to drop it, record it only locally, or record and export it. This is how you control data volume and cost.
Resource
A set of attributes describing the thing producing telemetry: service name, version, host, container, cloud region. It is attached once and applies to every signal.
SpanProcessor and LogRecordProcessor
Hooks that run when a span starts and ends (or a log record is emitted). The batch version collects items and ships them in groups. The simple version ships each item right away.
Exporters
Convert data into a transport format and send it. The standard choice is the OTLP exporter. Others can write to the console for debugging or speak older protocols.
MetricReader and Views
A reader collects measurements from the SDK, usually every set interval, and hands them to an exporter. A view lets you rename, filter, or regroup metrics before they leave.
IdGenerator
Creates trace IDs and span IDs. Defaults are random. Some environments, such as AWS X-Ray, expect IDs with a special format, so a custom generator can be plugged in.
Propagators
Inject and extract context at service borders, as described in the last chapter. W3C Trace Context and Baggage are the usual pair.
Limits
Caps on how many attributes, events, and links a span may carry, so one bad code path cannot create a giant span. Typical defaults are in the low hundreds per span.
Create the providers once, near the very start of your program, and register them globally. If code asks for a tracer before the provider is registered, it may receive a no-op tracer and silently produce nothing.
7How the SDK Works Internally: The Life of a Span
Let us follow one span from birth to delivery. This is the single best way to understand what the SDK does for you behind the scenes.
Startup configuration
When the program starts, you build a resource, choose a sampler, attach a batch processor with an OTLP exporter, and register the provider. After this moment, the API calls inside your code start doing real work.
A request arrives
Instrumentation for your web framework extracts any incoming trace header, then asks a tracer to start a span. The SDK reads the active context to find the parent, if there is one.
IDs and sampling
The SDK generates a span ID, reuses or creates the trace ID, and asks the sampler for a decision. There are three outcomes: drop, record only, or record and sample. A dropped span becomes a cheap placeholder that still passes context along, so child work stays consistent.
Work happens
While the span is open, your code can add attributes, record events, mark errors, or set the status. These are simple in-memory operations, so they are fast. Limits guard against runaway growth.
The span ends
The SDK stamps the end time, freezes the span, and calls every registered processor. A sampled span is handed to the batch processor. An unsampled one is simply forgotten.
Queueing
The batch processor drops the span into a bounded in-memory queue. Your request thread returns at once. It never waits for the network.
Batch export
A background worker wakes up on a timer, or when enough spans have piled up, takes a batch off the queue, and passes it to the exporter. The exporter serializes the batch and sends it, and retries when the failure looks temporary.
Flush and shutdown
When the program is closing, you ask the provider to flush and shut down so that the last queued spans are sent before the process disappears.
Why the queue is bounded
The queue has a maximum size on purpose. Specification defaults are roughly a queue of 2,048 spans, batches of up to 512 spans, and a send attempt about every five seconds. If the backend is slow and the queue fills up, new spans are dropped instead of piling up in memory. This is a deliberate trade: the SDK values the health of your application over the completeness of its telemetry. A monitoring tool that crashes your service would be worse than useless.
Simple versus batch processing
| Question | Simple processor | Batch processor |
|---|---|---|
| How it sends | Exports each span the moment it ends | Collects spans, then exports groups |
| Network cost | High, one call per span | Low, one call per many spans |
| Effect on request speed | Can slow requests down | Almost none |
| Risk of loss | Low if the process stays alive | Spans in the queue are lost if the process dies without a flush |
| Best use | Local debugging and tests | Production |
Short-lived programs, such as command-line tools, scheduled jobs, and serverless functions, may end before the next batch timer fires. Always call flush and shutdown before exit, or the final spans vanish.
The specification asks SDKs to be fail-safe. If exporting fails, or a configuration value is wrong, the SDK should report the problem through an internal error handler and carry on. It should not throw errors into your business logic.
8Data Flow and Lifecycle: From Your Code to a Dashboard
The SDK finishes its job when data leaves the process. But where it goes next shapes cost, safety, and reliability, so let us follow the data to the end.
OTLP: the common language
OTLP (OpenTelemetry Protocol) is the standard way of sending traces, metrics, and logs between SDKs, the Collector, and backends. It can run over gRPC, a fast binary protocol, usually on port 4317. It can also run over plain HTTP, usually on port 4318, with paths named for each signal. Because so many tools accept OTLP, it is the safest default exporter.
The Collector: a flexible middle layer
The OpenTelemetry Collector is a separate program that receives telemetry, changes it, and forwards it. It is built from three kinds of parts that you connect into pipelines: receivers take data in, processors change it (batching, filtering, scrubbing, sampling), and exporters send it on. Think of it as a mail room inside a large office building. Departments hand over their envelopes, and the mail room sorts, stamps, and dispatches them.
flowchart LR
A1["Service A with SDK"] --> AG1["Collector Agent (on node 1)"]
A2["Service B with SDK"] --> AG1
A3["Service C with SDK"] --> AG2["Collector Agent (on node 2)"]
AG1 --> GW["Collector Gateway (central: redaction, tail sampling)"]
AG2 --> GW
GW --> TB["Tracing Backend"]
GW --> MB["Metrics Backend"]
GW --> LB["Logs Backend"]
In the figure, each application sends data to a small agent Collector running on the same machine or node. The agent handles quick tasks and local buffering. Agents then forward to a central gateway Collector, which applies company-wide rules and fans data out to different backends.
Direct export or Collector?
| Concern | SDK exports directly | SDK sends via Collector |
|---|---|---|
| Setup effort | Smallest | One more service to run |
| Backend credentials | Stored in every application | Kept in one place |
| Retries and buffering | Limited to the in-process queue | Larger, separate buffers |
| Scrubbing sensitive data | Each app must do it | One central rule set |
| Tail sampling | Not possible | Possible |
| Switching backends | Change every app | Change one config |
Direct export is fine for local development and tiny systems. For anything serious, most teams place a Collector in between because it moves heavy work out of the application and gives the platform team a single control point.
The full lifecycle of one data point
Produce
Your code or auto-instrumentation creates a span, metric measurement, or log record through the API.
Shape
The SDK samples, enriches with the resource, and applies limits and views.
Ship
A batch or periodic reader hands data to an OTLP exporter that sends it away.
Route
The Collector receives it, cleans it, optionally samples it again, and sends it to one or more backends.
Store and query
The backend stores it, indexes it, and lets people search, chart, and alert on it. It is eventually deleted according to a retention rule.
9Sampling: Keeping the Useful Data and Paying Less
Sampling is the SDK feature that matters most to your bill. It is also the one beginners understand least, so we will go slowly.
Why sample at all?
Consider a busy service handling 10,000 requests per second, where each request creates 20 spans. That is 200,000 spans every second, or about 17 billion spans a day. If each span averaged only 500 bytes, you would produce around 8.6 terabytes of trace data daily from one service. Storing and searching all of it is rarely worth the price, because most requests are boring and nearly identical.
Sampling means keeping only a fraction of the traces. Done well, you keep the ones you care about and still get a true picture of the whole.
A factory has a quality inspector. One style of inspector checks every tenth item as it leaves the line, no matter what. Another waits until the end of the day and then pulls out every item that looks faulty or took suspiciously long to make. The first is head sampling. The second is tail sampling.
Head sampling: decide at the start
Head sampling decides when the root span is created, before anything is known about how the request will turn out. It is cheap and happens inside the SDK. Its weakness is that it cannot know in advance which requests will fail or be slow.
| Sampler | What it does | When to use it |
|---|---|---|
| Always on | Keeps every trace | Development, low-traffic systems |
| Always off | Keeps nothing | Disabling tracing for a component |
| Trace ID ratio | Keeps a fixed share, based on the trace ID | High-traffic production, when you need a simple cap |
| Parent based | Follows the decision already made by the parent | Almost always, wrapped around another sampler |
The ratio sampler works from the trace ID itself, so two services using the same ratio will agree on which traces to keep. The parent-based wrapper is crucial because it keeps a trace whole. Without it, service B might drop the middle of a trace that service A decided to keep.
flowchart TB
S["New span requested"] --> Q{"Has a parent?"}
Q -- No --> R["Root sampler decides (for example a 10 percent ratio)"]
Q -- Yes --> P{"Parent sampled flag set?"}
P -- Yes --> K["Keep this span"]
P -- No --> DR["Drop this span"]
R -- Selected --> K
R -- Not selected --> DR
Read the figure like a flowchart at a security gate. A request that comes with a parent follows the parent’s choice. Only a brand-new trace faces the ratio test.
Tail sampling: decide at the end
Tail sampling waits until a trace is finished, then decides. It can keep every trace that contains an error, every trace slower than two seconds, and a small share of the normal ones. It is far smarter than head sampling, but it needs a Collector that holds all the spans of a trace in memory until the decision is made. That means all spans of one trace must reach the same Collector instance, usually through a load balancer that routes by trace ID.
A practical recipe: use a parent-based ratio sampler in the SDK to cap volume. Add tail sampling in the Collector only if you must guarantee that all errors and slow requests are kept.
Trace sampling does not affect metrics. If you compute request counts and error rates from sampled spans, your numbers will be wrong. Use SDK metrics for rates, and use traces for explaining individual cases.
10Metrics in Depth: Instruments, Aggregation, and Cardinality
Metrics work very differently from traces. Instead of recording each event separately, the SDK squeezes many events into a few numbers. Understanding that squeeze is the key to using metrics wisely.
A car dashboard shows three kinds of readings. The odometer only goes up and counts total distance. The fuel gauge goes up and down and shows a current level. The trip computer can tell you how your speeds were spread across the journey. Metric instruments follow the same ideas.
The instruments
An instrument is the object your code uses to report a measurement. It is created from a meter, and each type fits a different shape of data.
| Instrument | What it measures | Example |
|---|---|---|
| Counter | A value that only goes up | Total orders placed |
| UpDownCounter | A value that can rise and fall by steps | Items currently in a queue |
| Histogram | The spread of many values | Request duration |
| Gauge | The latest value at a moment | CPU temperature |
| Observable (async) versions | Values read by a callback when the SDK asks | Current memory in use |
The observable instruments are special. Instead of your code pushing a number each time something happens, the SDK calls your callback at collection time and you return the current value. This is ideal for things that exist independently of requests, such as memory use or open connections.
The metric pipeline
flowchart LR
I["Instrument (Counter, Histogram, Gauge)"] --> M["Measurement"]
M --> V["View (optional rename, filter, regroup)"]
V --> AG["Aggregation (Sum, Last Value, Histogram)"]
AG --> MR["MetricReader"]
MR --> ME["MetricExporter"]
ME --> DST["Collector or Backend"]
Aggregation and temporality
Aggregation is the method used to combine measurements: add them together (sum), keep the latest (last value), or count them into buckets (histogram). Histograms deserve a special note. They tell you not just the average, but how many requests were fast, medium, and slow, which is what you need to compute percentiles such as the 99th. Newer SDKs also support exponential histograms, which use buckets that adapt automatically and stay compact.
Temporality describes what the numbers are measured against. With cumulative temporality, each export reports the total since the process started. With delta temporality, each export reports only what changed since the last export. Some backends, such as Prometheus-style systems, expect cumulative data. Others prefer deltas. Choose to match your backend.
A metric reader decides when data is collected. The usual reader collects on a timer, with a default of about one minute. Another type waits to be asked, which is how pull-based systems scrape data.
The cardinality trap
Cardinality means the number of different label combinations a metric has. If you tag a request counter with the HTTP route and status code, you may have a few dozen combinations, which is fine. If you also tag it with the user ID, you suddenly have one time series per user, perhaps millions. Storage cost and query time explode. Newer SDKs include a cardinality limit per metric and fold the overflow into a special bucket, but the safest approach is never to put unbounded values such as user IDs, session IDs, or full URLs into metric attributes.
High-cardinality labels belong on traces and logs, where each record stands alone. Keep metric labels to small, fixed sets such as region, route template, and status class.
Finally, exemplars let a metric point carry a link to a sample trace. When you see a spike in latency on a chart, you can jump straight to an example slow request. This is one of the most helpful bridges between metrics and traces.
11Logs and Correlation: Connecting the Three Signals
Logs are the oldest signal, and almost every application already has logging in place. OpenTelemetry does not ask you to throw that away.
The logs bridge
Instead of replacing your logging library, OpenTelemetry offers a logs bridge. A small adapter sits inside your existing logger, such as the standard logging module of your language. Every time your code writes a log line, the adapter turns it into an OpenTelemetry log record. The SDK then adds the resource, passes the record through a log processor, and exports it with the same OTLP pipeline as the other signals. Application developers keep calling the logger they already know.
Why correlation changes everything
When a log record is written while a span is active, the SDK copies the current trace ID and span ID onto the record. This one small act is powerful. From a slow trace you can jump to every log line written during that request, and from a suspicious error log you can open the whole trace that produced it. Instead of three separate tools, you get one connected story.
Imagine a detective’s board. The photo of the crime scene is the trace, the list of witness statements is the logs, and the neighborhood crime statistics are the metrics. Red string connecting items to a common case number makes the board useful. The trace ID is that case number.
Prefer structured logs
Write fields, not sentences. A record with named fields such as order ID and result can be filtered and counted. A long free-text line cannot.
Use levels honestly
Debug, info, warning, and error should mean something. If everything is an error, nothing is.
Watch the firehose
Logs are often the largest and most costly signal. Filter noisy debug output in the Collector or at the logger.
Never log secrets
Passwords, tokens, and personal data should never reach a log record. Once exported, they are hard to take back.
12Instrumentation and Configuration: Putting the SDK to Work
The SDK does nothing until something feeds it data and someone configures it. This chapter covers both: how telemetry gets produced, and how you control the SDK without rewriting code.
Instrumentation: who creates the spans?
Instrumentation is the code that calls the API to create telemetry. There are three common styles, and most real systems mix them.
| Style | How it works | Effort | Detail you get |
|---|---|---|---|
| Automatic (zero-code) | An agent or startup hook patches popular frameworks and drivers while the program loads | Very low | Good coverage of web, database, and messaging calls |
| Library instrumentation | You add a ready-made package for one framework or client | Low | Detailed for that one library |
| Manual | You write API calls around your own business steps | Higher | Exactly what matters to your business, such as “apply discount code” |
A sensible path is to start automatic, see what you get in a week, then add manual spans only where the automatic picture is too coarse. Several languages offer a startup agent: the Java agent attaches to the program at launch, and Python offers a wrapper command that runs your program with instrumentation switched on. On Kubernetes, the OpenTelemetry Operator can inject these agents into your pods based on a simple annotation, so developers change nothing.
Configuration without code changes
The specification defines standard environment variables that every SDK understands. This lets the platform team configure telemetry through deployment files while developers focus on features.
OTEL_SERVICE_NAME
The name of this service. It becomes the main label on all telemetry. If you forget it, data is filed under an “unknown service” name that is hard to find.
OTEL_RESOURCE_ATTRIBUTES
Extra facts about the service, such as version and deployment environment, written as comma-separated pairs.
OTEL_EXPORTER_OTLP_ENDPOINT
Where to send data, usually the address of your Collector. Related settings choose the protocol and add authentication headers.
OTEL_TRACES_SAMPLER
Chooses the sampler, such as a parent-based ratio. A companion setting carries the argument, such as the ratio value.
OTEL_PROPAGATORS
Lists the header formats to use, for example W3C Trace Context and Baggage. Change it only when you must talk to older systems.
OTEL_SDK_DISABLED
Turns the SDK off without removing instrumentation. A handy emergency lever.
Resource: tell the truth about where data comes from
A good resource answers the questions “which service, which version, which environment, which machine?” Without these, a spike on a chart cannot be tied to a release or a region. Many SDKs include resource detectors that fill in cloud, container, and Kubernetes facts automatically. Add your own service version, and you can compare behavior before and after every deployment.
Pick one naming rule for services across the whole company and write it down. Names like “orders”, “order-service”, and “OrderSvc” for the same thing split your data into three piles.
13Security: Keeping Telemetry from Becoming a Leak
Telemetry describes the inner life of your systems. In the wrong hands, or with the wrong content inside, it becomes a risk. Treat it as sensitive production data.
A flight data recorder captures everything that happens in a cockpit. It is incredibly valuable for investigators, which is exactly why it is built to be protected and handled carefully. Your telemetry pipeline records the cockpit of your business and deserves similar respect.
Sensitive data in attributes
Database statements, URLs with tokens, email addresses, and request headers often hold secrets or personal data. Capture only what you need and scrub the rest.
Encrypt the wire
Send OTLP over TLS between SDKs, Collectors, and backends. Plain connections can be read by anyone on the path.
Authenticate the receiver
A public, open Collector port invites strangers to flood you with fake data. Require tokens or mutual TLS, and keep the Collector on a private network where you can.
Untrusted trace headers
Outside callers can send forged trace IDs or force sampling on. At the public edge, restart or link traces instead of trusting them blindly.
Baggage crosses boundaries
Baggage is copied into outgoing requests, including calls to third-party services. Never place secrets or personal data in it.
Agents run with full power
A zero-code agent runs inside your process with the same permissions as your application. Download it from official sources, pin versions, and review upgrades.
Telemetry as an attack surface
Attacker-controlled values used as labels can create a flood of unique metric series and exhaust storage. Limit label values and enforce limits.
Limit who can look
Traces reveal your architecture and customer behavior. Apply role-based access and audit logs to your observability tools.
Scrubbing in the right place
The most reliable defense is to remove sensitive data twice. First, avoid putting it into spans and logs in the code. Second, add Collector processors that delete or hash known risky attributes before anything reaches storage. The second layer catches mistakes made by tired developers on a deadline.
Do not capture full request and response bodies as span attributes by default. They are large, often private, and almost never needed for finding performance problems.
14Performance, Reliability, and Monitoring the Monitor
An observability system that slows down your service, or silently stops working, is a bad deal. Here is how the SDK protects your application and how you can protect the SDK.
What does it cost at runtime?
Creating a span, adding attributes, and ending it are in-memory operations designed to be fast. The costs that matter are the number of spans per request, the number and size of attributes, and the work of serializing and sending batches. The exact overhead depends on your language, traffic, and settings, so measure it in your own environment under realistic load instead of trusting a single number from anyone else’s blog.
Fewer spans, smaller spans
Skip tiny internal functions. A span per loop iteration adds cost and noise without adding insight.
Bounded queues
Queues and attribute limits cap memory growth. Tune the queue size only after you see drops.
Batch and compress
Large batches with compression cut bandwidth. Send to a nearby Collector instead of a distant backend.
Sample deliberately
Sampling is the strongest lever for both cost and overhead, since dropped spans cost almost nothing.
Reliability when things break
If the Collector goes down, the SDK keeps trying for a while, fills its queue, and then starts dropping data. Your application continues to serve users. Once the Collector returns, new data flows again, but whatever was dropped is gone. To reduce loss, run several Collector instances behind a load balancer, give agents their own buffers, and keep export timeouts short so that threads do not wait for long.
Watching the watcher
You must monitor the telemetry pipeline just as carefully as the applications it observes. The Collector publishes its own metrics about how much data it received, queued, refused, and failed to send. Many SDKs also report diagnostic logs or internal counters on dropped spans and failed exports. Alert when export failures climb, when queues stay close to full, or when the number of incoming spans from a service suddenly falls to zero. A silent drop to zero often means broken configuration, not a quiet day.
Scale the Collector layer horizontally. Stateless agents and gateways can be copied freely. Only tail sampling needs care, because it requires every span of a trace to land on the same instance.
15Advantages, Disadvantages, and Trade-offs
No tool is perfect. A good architect knows both what a technology gives and what it costs. Here is an honest balance sheet for the OpenTelemetry SDK.
Advantages
- Vendor neutrality: change backends without rewriting instrumentation.
- One consistent model for traces, metrics, and logs, with shared context and resource.
- Wide language coverage and a huge library of ready-made instrumentation.
- Strong community, steady development, and broad support from vendors and cloud providers.
- Configuration through standard environment variables and the Collector.
- Library authors can add telemetry safely through the lightweight API.
Disadvantages
- A big surface area to learn: signals, processors, samplers, readers, and the Collector.
- Maturity differs by language and signal, so features and stability vary.
- Naming conventions evolve, which can cause attribute names to change between versions.
- Poor sampling or labels can create heavy costs very quickly.
- Running and tuning a Collector fleet is real operational work.
- Auto-instrumentation can surprise you by creating more data than expected.
Key trade-offs to decide consciously
| Decision | Choose the first when | Choose the second when |
|---|---|---|
| Head sampling or tail sampling | You want simplicity and a hard cap on volume | You must keep every error and slow trace |
| Direct export or Collector | Tiny systems and quick experiments | Production systems with many services |
| Automatic or manual instrumentation | You want quick broad coverage | You need business-level detail |
| Cumulative or delta metrics | Your backend expects running totals | Your backend prefers per-interval changes |
| Rich attributes or lean attributes | Debugging power matters more than cost | Cost, privacy, and speed matter more |
16Design Patterns and Anti-patterns
Teams who run OpenTelemetry at scale keep rediscovering the same good ideas and the same traps. Learn from them to save months of trial and error.
Patterns worth copying
Agent plus gateway
A light Collector next to every application node, feeding a central gateway. Local buffering with central control.
API-only libraries
Shared libraries depend on the API and leave SDK choices to the application owner.
Parent-based ratio sampling
A simple ratio at the edge, honored by every downstream service, so traces stay whole.
Resource-first naming
Every service sets its name, version, and environment from deployment settings, so all signals line up.
Business attributes
Add a small number of meaningful fields, such as plan type or order channel, to the root span of important flows.
Central scrubbing
A Collector processor removes or hashes sensitive fields for all services at once.
Anti-patterns to avoid
The Mistake
Adding unbounded values, such as user IDs, email addresses, or full URLs, as labels on metrics.
Why It Hurts
Each new value creates a brand-new time series. Storage and query costs grow without limit, dashboards slow down, and the backend may reject the data.
Do This Instead
Keep metric labels to small fixed sets, such as route template and status class. Put the detailed identifiers on traces or logs.
The Mistake
Forgetting to flush and shut down the SDK in short-lived programs, such as scheduled jobs and serverless functions.
Why It Hurts
The batch queue is in memory. If the process ends first, the most recent spans, often the ones around a crash, are lost.
Do This Instead
Call flush and shutdown on the providers before exit, and leave enough time for the final export to finish.
The Mistake
Creating a span for every tiny function or loop iteration “just in case”.
Why It Hurts
Traces turn into unreadable walls of thousands of spans, overhead grows, and the single slow step you need is buried in noise.
Do This Instead
Create spans around meaningful units of work: incoming requests, outgoing calls, database queries, and important business steps. Use span events for small details inside a span.
17Best Practices and Common Mistakes
If you remember only one chapter for your first week with the SDK, make it this one. These habits prevent the majority of problems.
Set the resource first
Define service name, version, and environment before anything else. Everything else depends on this identity.
Start with automatic instrumentation
Get broad coverage quickly, look at the result, then add manual spans for the gaps.
Route through a Collector
Even a single Collector gives you a place to scrub, batch, retry, and switch backends later.
Choose a sampling plan before launch
Pick a parent-based ratio and review it as traffic grows. Do not wait for the first surprise bill.
Follow semantic conventions
Standard names make your data usable by tools and dashboards written by others.
Correlate logs with traces
Make sure trace and span IDs appear on log records so you can jump between signals.
Monitor the pipeline itself
Alert on export failures, full queues, and sudden drops to zero incoming data.
Pin and test versions
Upgrade SDK and instrumentation packages on purpose, and test in staging, because attribute names and defaults can change.
Common mistakes and their fixes
| Mistake | What you will see | Fix |
|---|---|---|
| No service name set | Data under an “unknown service” label | Set the service name through the standard variable |
| Provider created too late | No spans from early startup code | Register providers before anything asks for a tracer |
| Wrong endpoint or protocol | No data and quiet export errors | Check the address, port, and gRPC versus HTTP setting |
| Headers stripped by a proxy | Traces broken in the middle | Allow the trace headers through every hop |
| Sampling set separately in each service | Partial traces with missing pieces | Wrap the sampler in a parent-based one |
| Rates computed from sampled spans | Wrong request and error counts | Use SDK metrics for rates |
| Using the SDK inside a library | Version clashes and forced settings | Depend on the API only |
18Real-World and Industry Examples
Theory becomes clearer with stories. First, a walk-through of the slow checkout from the start of this guide. Then, a look at how the wider industry uses OpenTelemetry.
An illustrative story: the slow checkout
Imagine an online store whose checkout runs through an API gateway, an order service, a payment service, and an inventory database. Every service uses the SDK, sets its service name, and sends spans to a nearby Collector agent. A support ticket reports slow checkouts. An engineer filters traces by duration and opens a slow one. The trace shows the order service took 2.4 seconds in total, and 2.1 seconds of that was one query inside the inventory step. The span attributes reveal the query ran without an index after last week’s release, and the service version attribute confirms the timing. The fix takes minutes. Without a shared trace ID, the team would have spent hours guessing which of the four services was at fault. (This is a made-up example to show the workflow, not a report from a real company.)
Jaeger: from in-house tool to open-source standard
Jaeger, a widely used tracing backend, was created at Uber and later became a graduated CNCF project. The Jaeger project has encouraged users to move to OpenTelemetry SDKs for instrumentation, and newer Jaeger versions are built on the OpenTelemetry Collector framework. It is a good example of the industry converging on one instrumentation layer while backends compete on features.
Kubernetes: the platform instruments itself
Kubernetes control-plane components can emit traces in OTLP format. That means the same tools and Collectors you use for your applications can help you investigate slow cluster operations, such as API server requests.
Cloud providers and vendors
Major cloud providers and monitoring vendors either accept OTLP directly or ship their own distribution of OpenTelemetry. AWS, for example, offers the AWS Distro for OpenTelemetry, which packages the Collector and SDK components with settings for AWS services, and the SDK ecosystem includes an ID generator and a propagator designed for AWS X-Ray. This is why a team can write instrumentation once and still choose among several destinations.
Proxies, gateways, and service meshes
Edge proxies such as Envoy can start and continue traces and can export them in OpenTelemetry-compatible form. This helps cover services you cannot modify, and it often supplies the very first span of a trace at the front door.
19Frequently Asked Questions
Quick answers to the questions that appear most often when people first meet the SDK.
20Summary and Key Takeaways
You started with a slow checkout and no way to see inside. You now know the machinery that makes the inside visible.
Key Takeaways
- The SDK is the engine inside your app. It implements the OpenTelemetry API, and it creates, samples, labels, batches, and exports traces, metrics, and logs.
- API and SDK are separate on purpose. Libraries depend on the API only, and the application owner installs and configures the SDK once.
- Context propagation connects the story. A shared trace ID, carried in headers such as
traceparent, ties spans from many services into one trace. - Sampling controls cost. Use parent-based ratio sampling for simplicity, and add tail sampling in a Collector when you must keep every error and slow trace.
- Mind metric labels. Keep cardinality low, and keep high-detail identifiers on traces and logs.
- Use a Collector in production. It centralizes scrubbing, retries, routing, and backend choices, and it keeps heavy work out of your applications.
- Treat telemetry as sensitive production data. Encrypt it, authenticate receivers, scrub secrets, and monitor the pipeline itself.