What Does It Mean to Correlate Logs with Traces in OpenTelemetry?
A plain-English guide to the small identifiers that connect a sentence in your logs to the exact step of the exact request that wrote it, and how to make that connection work in real systems.
Imagine a customer writes to support: “My payment failed at 10:05.” An engineer opens the logs and searches for 10:05. The system was busy, so there are thousands of lines from that minute. Some say “payment declined”, some say “retrying”, some say “timeout”. Which of them belong to this customer? Which belong to the thousand other people who paid at the same moment? Without a clear link, the engineer is reading a pile of sentences that came from many stories mixed together.
Now imagine that every log line carries a small tag that says, “I was written during request number 4bf9…, inside step number 00f0…”. The engineer pastes that request number into the search box, and only the lines of that one story appear, in order. This is what it means to correlate logs with traces.
In this guide, you will learn what correlation is, what the tags are made of, how the OpenTelemetry SDK attaches them, how they travel across services, and how to avoid the mistakes that quietly break them. No prior knowledge is needed. Every new word is explained the first time it appears.
1What Does It Mean to Correlate Logs with Traces?
To correlate two things means to connect them so that, from one, you can find the other. Here, the two things are a log and a trace.
A quick look at each side
A log is a written message that a program produces at a moment in time, such as “payment declined for order 8841”. A trace is the full story of one request as it travels through your system, made of timed steps called spans. A log tells you what the program said. A trace tells you what the request did, step by step, and how long each step took.
The connection
Correlating a log with a trace means that the log record carries the ID of the trace, and the ID of the specific span, that were active when the message was written. With those two IDs, a tool can take you from a log line to the exact step of the request it belongs to, and from a step of the request to every log line written inside it. The link works in both directions.
Think of a hospital. A patient has a chart number. The blood test result, the doctor’s notes, and the heart-rate monitor readings are all stamped with that chart number. Without it, the papers are just loose sheets in a big pile. With it, one number brings together everything about one patient. The trace ID is the chart number for one request, and logs are the notes and test results stamped with it.
Remember the short form: correlation is “stamping every log with the request it belongs to”. Everything else in this guide is how to stamp reliably and how to use the stamps.
2Why Logs and Traces Need Each Other
Each signal is good at something and weak at something else. Correlation joins the strengths together.
What traces are good at, and what they miss
A trace is excellent for showing structure and timing. You can see that the order service called the payment service, and that the payment service spent 900 milliseconds waiting for a fraud check. But a span usually carries only a few short labels. It rarely holds the detailed message that explains why the fraud check was slow, such as “rule set reloaded from disk” or “connection pool exhausted”.
What logs are good at, and what they miss
A log can hold rich, specific detail: the exact error text, the values of important variables, a warning from a library. But a log on its own does not know which request it belongs to, and it does not show how long things took relative to each other. In a busy system, logs from thousands of requests are interleaved, like the voices of thousands of people in a crowded hall.
Shape and timing
Who called whom, in what order, and how long each step took. Perfect for finding where time went.
Thin detail
Spans carry short labels. They rarely contain the long explanations or exact messages that a developer needs.
Rich detail
Exact messages, values, stack traces, and warnings from every library in the program.
No request context
Without IDs, a line does not say which request wrote it, so lines from many requests blur together.
The size of the problem
Consider a service that handles 1,000 requests per second, where each request writes 20 log lines. That is 20,000 log lines every second. In one minute, there are 1.2 million lines. Searching those by time alone to find the 20 lines of a single request is like finding a particular grain of rice in a bag of rice. With a trace ID, the same search is a single exact lookup.
Picture a courtroom where a hundred people are speaking at once, and a stenographer writes down every sentence without names. Later, a judge asks, “What did the witness say?” Nobody can tell. If each sentence is marked with the speaker’s number, the transcript becomes a clear conversation. Trace IDs are those speaker numbers.
3Core Vocabulary: The Words You Need
A small set of words appears in every conversation about correlation. Learn them once, and the rest will be easy.
| Term | Plain meaning | Example |
|---|---|---|
| Log record | One structured log entry, with a time, a severity, a message, and labels | “payment declined” at 10:05:12 |
| Trace | The story of one request across all services | One checkout from click to confirmation |
| Span | One timed step inside a trace | “charge card” |
| Trace ID | A random number shared by every span of one trace | A 32-character hexadecimal string |
| Span ID | A random number that identifies one span | A 16-character hexadecimal string |
| Trace flags | Small switches, including whether the trace is being sampled | “sampled: yes” |
| Active span | The span that is currently in progress for the running code | The span for the request being handled |
| Context | An invisible package that follows the code and holds the active span | The “backpack” of the current request |
| Severity | How serious a log is | Debug, info, warning, error |
| Resource | Facts about the thing producing telemetry | Service name, version, host |
| Logs bridge | An adapter that turns lines from an existing logging library into OpenTelemetry log records | A plug-in for your language’s logger |
One more word: signal
OpenTelemetry calls each kind of telemetry a signal. The main three are traces, metrics, and logs. This guide focuses on the pairing of two of them, logs and traces, with a short look at metrics near the end.
Trace IDs and span IDs are not secrets and do not identify a person. They are random labels. Treat them like a ticket number at a bakery, which is useful for matching and not for proving who someone is.
4Anatomy of an OpenTelemetry Log Record
To see where correlation lives, let us open a log record and look inside. OpenTelemetry defines the shape of a log record carefully, so that every tool reads it the same way.
Timestamp
When the event happened. A second time field can record when the system first observed it, which helps when logs are collected from files later.
Severity text and number
A word such as “ERROR” and a standard number, so tools can sort and filter across languages.
Body
The message itself. It can be plain text or a structured set of fields.
Attributes
Key and value labels about this one record, such as the order ID or the error code.
Trace ID
The ID of the trace that was active when the log was written. This is the first half of correlation.
Span ID
The ID of the exact span that was active. This is the second half of correlation.
Trace flags
Switches copied from the active span, including the sampled flag, which says whether the trace is being kept.
Resource and scope
The resource says which service and host wrote the log. The scope says which library or module did.
The three “link” fields
The trace ID, the span ID, and the trace flags are the heart of correlation. They are normal, built-in fields of the log record, not a custom label you invent. That is important. Because they are part of the standard shape, every OpenTelemetry-aware backend knows where to find them and can turn them into a clickable link between a log line and its trace.
Think of a parcel label. It has the sender, the weight, and a description of what is inside, but the most useful part for tracking is the tracking number. The trace ID and span ID are the tracking number on every log record. Everything else describes the parcel, and the tracking number lets you find its journey.
If your log lines have “trace_id” and “span_id” only inside the message text, that is correlation by habit. If they sit in the dedicated fields of the log record, that is correlation by design. Aim for the second.
5How Correlation Works: The Idea in One Picture
The mechanism is simple. When your code writes a log line, the SDK looks at which span is active at that moment and copies its IDs into the log record.
flowchart LR
REQ["Incoming request"] --> SP["Active span (trace ID and span ID)"]
SP --> LG["Code writes a log line"]
LG --> LR["Log record with trace ID, span ID, and message"]
LR --> BE["Backend stores logs and traces"]
BE --> UI1["From a trace: show its logs"]
BE --> UI2["From a log: open its trace"]
Read the figure from left to right. A request arrives, and a span is started for it. While that span is active, your code writes a log line. The SDK copies the span’s trace ID and span ID into the new log record. Later, the backend stores logs and traces side by side. Because both carry the same IDs, a person can start from either side and jump to the other.
Three keys that can connect signals
| Key | How precise | What it connects |
|---|---|---|
| Trace ID | Very precise | All logs of one request, across every service |
| Span ID | Most precise | Logs written inside one exact step |
| Resource and time | Approximate | Logs and traces from the same service and instance at about the same moment |
The first two keys are the real thing. The third is a fallback that helps when IDs are missing, and it is covered in Chapter 11.
Nobody has to write the IDs by hand. In a normal setup, you write ordinary log lines, and the SDK or its logging integration adds the IDs automatically.
6Inside the SDK: From a Log Call to an Exported Record
Let us follow one log line from your code to the network, and see exactly where the IDs are attached.
sequenceDiagram
participant C as Your Code
participant L as Logging Library
participant B as OpenTelemetry Log Bridge
participant S as SDK LoggerProvider
participant X as Context
participant P as Log Record Processor
participant E as OTLP Exporter
C->>L: Write "payment declined"
L->>B: Hand over the log event
B->>S: Emit log record
S->>X: Which span is active right now?
X-->>S: Trace ID, span ID, trace flags
S->>P: Record with IDs, resource, and time
P->>E: Batch of records
E->>E: Send to Collector or backend
Your code writes a log
You call your normal logger. Nothing about this line is special.
The bridge catches it
A small adapter inside your logging library notices the event and turns it into an OpenTelemetry log record, with severity, body, and attributes.
The SDK reads the context
When the record is emitted without an explicit context, the SDK looks at the current context and finds the active span. It copies the trace ID, span ID, and trace flags.
The resource is added
The SDK attaches the resource, such as service name and version, so the record also says where it came from.
The processor queues it
A log record processor collects records in a queue. The batch version groups them so the network is used efficiently.
The exporter sends it
The exporter sends the batch in the OTLP format to a Collector or backend. The program never waits for the network on the logging line.
The IDs are taken at the moment the record is created. If your code hands the log message to another thread and creates the record there, the active span may be gone. Create the record while the span is still active.
7Logs Bridges and Integrations
You do not have to replace the logging library your team already uses. OpenTelemetry is designed to plug into it.
The logs bridge idea
OpenTelemetry has a logs API, but it is meant mainly for people who build logging libraries, not for everyday application code. For application developers, the recommended path is a bridge or appender: a small piece that sits inside your existing logger and forwards each log event to the OpenTelemetry SDK. You keep calling the logger you already know. The bridge does the rest.
A travel plug adapter lets your old device work in a new country without changing the device. A logs bridge is an adapter that lets your existing logging library speak OpenTelemetry without changing a single log line in your code.
Two ways the IDs reach your logs
| Approach | How it works | Good for |
|---|---|---|
| Bridge to the SDK | The log event becomes an OpenTelemetry log record with the IDs in dedicated fields, and is exported through OTLP | New setups that send logs directly to a Collector or backend |
| Add the IDs to the logger’s own context | The instrumentation puts the trace ID and span ID into the logger’s context, so they appear in the normal log output, such as a JSON line or a formatted text line | Existing setups that write logs to files or standard output |
The second approach is common and very practical. Many logging frameworks have a place for “extra fields attached to every line during this request”, often called a mapped diagnostic context. Instrumentation, including some automatic agents, can fill that place with the current trace ID and span ID, so your existing log format simply gains two more fields. Each language chooses its own field names, such as “trace_id” in some and a longer name in others, so check what your setup uses.
Common frameworks
Popular logging frameworks have appenders and context integrations. An automatic agent can add the IDs for you.
Standard logging module
A handler can send records to the SDK, and a logging integration can add the IDs to each record’s fields.
Built-in logging interface
The standard logging interface can be wired to the SDK, and the IDs flow through from the current activity.
Popular loggers
Instrumentation for common logger libraries can add the IDs and forward records to the SDK.
Prefer structured logging, where each line has named fields, such as JSON. Fields like the trace ID are then easy for tools to read, and you avoid fragile text parsing.
8Context: Where the Active Span Comes From
Correlation depends on one hidden helper: the context. If the context does not hold the right span at the right moment, the log gets no IDs, or the wrong ones.
A reminder about the active span
The context is a small invisible package that follows your code as it runs. One thing it holds is the active span, the span currently in progress. When a request arrives, instrumentation starts a span for it and makes it active. Every log written while it is active can read the IDs from it. When the span ends, it stops being active.
Before the span starts
Logs written here have no IDs, because no span is active yet. Start the span as early as possible, ideally in the web framework’s first step.
While the span is active
Logs get the span’s trace ID and span ID. This is the ideal case, and where nearly all your logs should fall.
Inside a child span
When code starts a child span, the child becomes active. Logs written there carry the child’s span ID but the same trace ID, which places them in a more precise step.
After the span ends
Logs written here fall back to the parent, if any is active, or get no IDs at all. Cleanup messages often fall in this gap.
When the context gets lost
Modern programs do many things at once. When work moves to another thread, a task queue, or a callback, the context has to be carried along. If it is not, logs written by that work have no IDs. They look like orphans, belonging to no request.
- Work handed to a thread pool without wrapping the context.
- Callbacks and event handlers that run long after the request returned.
- Background jobs started during a request but running on their own.
- Code in languages that pass the context explicitly, where a function simply forgot to pass it.
At a conference, each attendee wears a badge. If someone leaves the hall through a side door and comes back without the badge, the staff cannot tell which group they belong to. Code that leaves the request’s context behind returns without a badge, and its logs cannot be matched to any request.
A log line with no trace ID is not always a bug in your logging. First ask: was a span active when it was written? If the answer is no, fix the span or the context, not the logger.
9Correlation Across Services
The real magic of correlation appears when a request crosses many services. A single trace ID can gather the logs of all of them.
One trace ID, many log sources
When a request moves from one service to another, the trace context travels with it, usually in an HTTP header called traceparent. Every service that receives the request continues the same trace. That means every service writes its logs with the same trace ID, and differently numbered span IDs. A single search for that trace ID returns the log lines of the gateway, the order service, and the payment service, all in one list.
sequenceDiagram
participant G as API Gateway
participant O as Order Service
participant P as Payment Service
participant C as Collector
participant B as Backend
G->>O: Request with trace context
Note over G: Log: request received, trace T1, span S1
O->>P: Request with trace context
Note over O: Log: calling payment, trace T1, span S3
Note over P: Log: card declined, trace T1, span S5
G->>C: Logs and spans
O->>C: Logs and spans
P->>C: Logs and spans
C->>B: Everything with trace T1
Note over B: One search by T1 returns all three services
The span IDs differ because each service creates its own spans. They also tell you where inside the request each line was written: S1 belongs to the gateway’s step, S3 to the order service’s step, and S5 to the payment service’s step. Together the IDs let you rebuild the story in order: the gateway received the request, the order service called payment, and the payment service declined the card.
A courier parcel passes through a warehouse, a truck, and a local depot. Each location keeps its own notebook, but each entry is stamped with the same tracking number. If you ask the company for everything about that number, you receive the combined notes of all three places.
If one service in the chain does not forward the trace header, every log after it starts a brand-new trace ID. Your search will then return only half of the story. Check that every hop, including proxies and message queues, passes the context along.
10Two Ways to Get Logs Out of Your Application
Correlation does not depend on one particular path. Logs can leave your program in two common ways, and each handles the IDs a little differently.
flowchart LR
APP["Application with SDK and log bridge"] -- "Path A: OTLP" --> CO["Collector"]
APP -- "Path B: write JSON lines with IDs" --> FILE["Log file or standard output"]
FILE --> FR["Collector file log receiver (parses IDs)"]
FR --> CO
CO --> BE["Backend: logs and traces together"]
Path A: send log records straight from the SDK
The bridge turns each log event into an OpenTelemetry log record, with the IDs in their dedicated fields, and the SDK exports it. This path is clean because nothing has to be parsed or guessed. The downside is that if the process crashes or the network is down, queued records may be lost, and you rely on the SDK’s queue.
Path B: write lines and let the Collector read them
The application writes logs the old way, to a file or to standard output, with the trace ID and span ID as fields inside each line. In container platforms, the platform often collects standard output into files. The Collector then reads the files with a file log receiver. Parsing steps in the receiver can extract the IDs from the line and place them into the proper trace and span fields of the OpenTelemetry log record, so downstream tools treat them as real correlation fields.
| Question | Path A: direct OTLP | Path B: files and Collector |
|---|---|---|
| Where are IDs added? | By the SDK, in dedicated fields | By the logging framework, in the line, then parsed |
| Risk of parsing mistakes | Low | Medium, if formats change |
| Survives an application crash? | Records still in the queue may be lost | Lines already written to disk are kept |
| Works with old applications? | Needs a bridge | Often needs only a format change |
| Extra moving parts | Fewer | A file reader and parsing rules |
Do not use both paths for the same logs. If the application sends records directly and also writes them to a file that is collected, you will see every line twice, which doubles cost and confuses searches.
11The Fallback: Correlating by Resource and Time
Sometimes the IDs are missing, for example in old services, third-party tools, or code running outside any request. You can still narrow things down, although less precisely.
Using resource attributes
Every log record and every span carries a resource, which says which service, version, and instance produced it. If a log line has no trace ID, you can still ask for “logs from the payment service, instance 3, between 10:05:10 and 10:05:15” and look at traces from the same service and instance in that same window. This is not exact, but it shrinks an enormous pile of lines to a manageable handful.
| Level | What you match on | Precision | Typical use |
|---|---|---|---|
| Best | Span ID | One exact step | Explain a slow or failing step |
| Very good | Trace ID | One whole request | Follow a request across services |
| Okay | Service, instance, and short time window | A small group of requests | Logs written outside a request or without IDs |
| Weak | Time window only | Thousands of lines | Last resort for old systems |
A warning about clocks
Matching by time depends on clocks being close to each other. Different machines can differ by a few milliseconds, or occasionally by seconds if time synchronization is poor. Always search a window slightly wider than you think you need, and keep machine clocks synchronized.
If a package has no tracking number, a delivery clerk can still help: “It was dropped at the north depot between two and three o’clock.” She narrows the pile, and then you check each parcel by eye. It works, but it takes far longer than a tracking number.
12Business Keys: A Helpful Partner to Trace IDs
Trace IDs are perfect for machines. People, however, remember order numbers and ticket numbers. A good setup supports both.
The gap that business keys fill
A customer does not know their trace ID. They know their order number. If your logs and spans both carry the order number as an attribute, support can search by order number, find the related spans and logs, and then use the trace ID to open the full story. The business key is the front door, and the trace ID is the hallway that leads through the whole house.
Order or ticket number
Something a customer or support agent can say out loud. Put it on the root span and on important log records.
Trace ID
The machine-friendly ID that links every span and log of the request, across all services.
Span ID
The exact step where a particular log line was written.
Rules for good business keys
- Add them as attributes, with standard-looking names, on the spans and logs where they matter.
- Use only keys that are safe to show to everyone who can view your telemetry.
- Do not use them as metric labels. They have too many unique values, and they will inflate cost.
- Never replace trace IDs with business keys. One order can involve several traces, and many requests have no order at all.
Business keys such as email addresses, phone numbers, or names are personal data. Prefer opaque order numbers and internal IDs, and avoid logging personal details that you do not need.
13Span Events, Log Records, and Errors
There are two places to record “something happened at this moment”: inside a span, and in the log stream. Knowing the difference avoids duplicate work and confusing data.
Two ways to record a moment
A span event is a small, timed note attached directly to a span, such as “cache miss at 120 milliseconds into this step”. A log record is a separate entry in the log stream that merely points to the span through its IDs. Span events always travel with their span. Log records travel on their own path, and may be stored, filtered, and kept for different lengths of time.
| Question | Span event | Log record with IDs |
|---|---|---|
| Lives with | The span itself | The log stream, linked by IDs |
| Best for | A few key moments inside one step | Detailed messages, warnings, and library output |
| Volume | Low and limited per span | Can be high |
| Disappears if the trace is not sampled? | Yes, with the span | No, unless you filter logs |
| Works for old code? | Needs instrumentation | Works with existing logging |
The OpenTelemetry project has been moving toward expressing events as log records, so that there is one clear way to record a moment. Check the current guidance for your language before deciding how to record events.
An error log is not an error span
A very common surprise: writing an error-level log does not mark the span as failed. The span has its own status, which is unset, OK, or error. A trace viewer that highlights failed spans looks at that status, not at your log level. When something fails, do both: record the error on the span, with its exception details, and write the log line with the IDs attached. Then the trace shows red, and the log gives the explanation.
If a cashier writes “customer upset” in a notebook, the shop’s computer does not know a sale failed. Only when she presses the “refund” button does the system register the problem. The notebook entry is the log, and the button press is the span status. You need both to tell the full story.
14The Investigation Workflow: Using Correlation in Practice
Correlation shines during an incident. Here is the typical path an engineer follows, from an alert to the explanation.
flowchart TB
AL["Alert: error rate rising"] --> M["Open the metric chart and find the time and service"]
M --> T["Find a slow or failing trace in that window"]
T --> S["Open the trace and find the failing or slow span"]
S --> L["Show logs for that span ID"]
L --> RC["Read the exact message and find the cause"]
RC --> FX["Fix, then confirm that the metric recovers"]
Alert and metric
A metric shows that something is wrong, and when. It is fast and cheap, and it tells you which service to look at.
Pick a trace
Search traces in that time window for errors or long durations. One example of a bad request is enough to start.
Find the culprit span
The tree of spans shows which step took the time or failed. You now know where.
Jump to its logs
With one click, the tool shows only the log lines that carry that span’s IDs. You now know why.
Widen if needed
If the cause started upstream, show all logs for the trace ID across services, and read them in time order.
Going the other way
The journey also works in reverse. Suppose a developer sees a strange warning in the logs. She clicks the trace ID in the line, and the full trace opens. She now sees how the request reached that warning, how long it took, and whether it affected the user. A warning that looked harmless in isolation may turn out to sit inside a request that failed.
Practice the workflow on an ordinary day. A tool that is smooth during peacetime is far less stressful to use at three in the morning.
15A Worked Example: Finding One Failure Among Thousands of Lines
Let us do the workflow on paper with a small set of log lines. The IDs are shortened so that they are easy to read.
The raw log lines
Imagine that, within one second, these six log lines appear, from several requests mixed together.
| Time | Service | Message | Trace ID | Span ID |
|---|---|---|---|---|
| 10:05:12.100 | Gateway | request received | T1 | S1 |
| 10:05:12.105 | Gateway | request received | T2 | S9 |
| 10:05:12.140 | Order Service | calling payment | T1 | S3 |
| 10:05:12.150 | Order Service | calling payment | T2 | S11 |
| 10:05:12.900 | Payment Service | connection pool exhausted | T1 | S5 |
| 10:05:12.950 | Payment Service | payment approved | T2 | S13 |
Without IDs
Look at the message column alone. Two requests arrive, two calls to payment go out, and one payment service message is a warning while the other is a success. Which request got the warning? Order matters, but timing is close and lines overlap. If hundreds of requests were mixed in, the answer would be a guess.
With IDs
Filter by trace ID T1, and you immediately get three lines in order: the gateway received the request, the order service called payment, and the payment service reported a connection pool problem. The failing step is span S5, inside the payment service, and the delay between 10:05:12.140 and 10:05:12.900, about 760 milliseconds, is the time spent waiting there. Filter by T2 and you see a normal, quick path. Within seconds you know that request T1 had trouble because of a resource limit in the payment service.
It is like a deck of cards from two shuffled games. Without colors, you have a confusing heap. Once every card has a color on its back, you can pull out one game with a single sweep of the hand. Trace IDs are the colors on the back of the cards.
In a real system, the same filter works across millions of lines. The IDs make the search a precise lookup instead of a hunt through text.
16Sampling and Correlation: When the Trace Is Missing
At high traffic, you cannot keep every trace, so a sampler keeps a share and drops the rest. This creates a puzzle for correlation: what happens to the logs of a trace that was dropped?
The gap
A log line written during a dropped trace still carries a trace ID and a span ID, because the SDK still tracks the request’s identity even when it decides not to record the spans. The trace flags on the log say that the trace was not sampled. If you click the trace ID in that log, the backend may find nothing, because the trace was never stored. The link is valid, but the target does not exist.
flowchart TB
LG["Log record with trace ID and flags"] --> Q{"Sampled flag set?"}
Q -- Yes --> T["Trace was kept: the link opens a full trace"]
Q -- No --> N["Trace was dropped: the link finds no trace"]
N --> O1["Option: still keep the log for text search"]
N --> O2["Option: drop the log to save cost"]
Strategies
| Strategy | What happens | Trade-off |
|---|---|---|
| Keep all logs, sample traces | Every log is stored. Only some link to a stored trace. | Highest log cost, good text search, some dead links |
| Keep logs only for sampled traces | Logs whose flag says “not sampled” are dropped | Lowest cost, but you lose logs from unsampled requests |
| Always keep warnings and errors, filter the rest | Important logs survive whatever the sampling decision | A good balance, but errors may still have no trace |
| Tail sampling in the Collector | The trace is kept if it contains an error or is slow | Traces for failures exist, but needs more memory and routing |
A library keeps only some books on the shelf and puts the rest in storage. A catalog card in the reading room may point to a book that is in storage. The card is correct, but the book is not within reach. A log with the ID of a dropped trace is such a card.
If failures are your priority, use tail sampling to keep every trace that contains an error. Then the logs of failing requests will always link to a real trace.
17The Third Leg: Metrics and Exemplars
Logs and traces are two legs. Metrics are the third, and they connect to traces in their own way.
Metrics point to traces through exemplars
A metric is a number such as “99th percentile latency”. It is cheap and complete, but it does not say which requests were slow. An exemplar is a small sample attached to a metric data point that carries the trace ID and span ID of one real request that contributed to that number. When you look at a spike on a chart, you can follow the exemplar into an actual slow trace, and from there to its logs.
Metric shows the symptom
A chart says latency rose at 10:05. It is wide and cheap, and it tells you that something is wrong.
Exemplar names an example
A dot on the chart carries a trace ID, giving a real slow request that you can open.
Trace shows the step
The span tree shows which step was slow or failing.
Logs explain the reason
The span’s logs give the exact message that shows the cause.
Why you should not put trace IDs on metric labels
It may be tempting to add the trace ID as a label on a metric so that it can be linked. Do not. Every request has a unique trace ID, so labels would produce a new series for each request, and storage would explode. Exemplars exist precisely to give the link without that cost.
A school report shows the average mark of a class. It does not tell you which students struggled. A teacher might attach a note: “for example, see Maya’s exam, page three”. The note is the exemplar, a pointer to one real case that explains the number.
Exemplars depend on tracing being active and on both the SDK and the backend supporting them. Where they are not available, link metrics and traces by service name and time window instead.
18Security and Privacy
Joining logs and traces gives you power, and power needs care. A few risks are specific to correlation.
Sensitive data in logs
Logs often hold more private detail than spans do. Passwords, tokens, card numbers, and personal data must never be written to logs, and correlation makes any leak easier to find.
Untrusted trace headers
An outside caller can send any trace ID. Your logs then carry attacker-chosen IDs, which can pollute searches or hide real activity. Start a fresh trace at the public edge.
Fake log lines
If user input is written into free-text logs, an attacker can add fake lines that look like they have valid IDs. Use structured logs, and let the SDK set the IDs, never user input.
Different audiences
Logs and traces may be viewed by different teams. Joining them can expose logs to people who could see only traces, so review permissions.
Different lifetimes
Logs may have to be deleted sooner or kept longer than traces for legal reasons. Plan retention for each signal separately.
Encrypt the pipe
Send OTLP over an encrypted connection, and authenticate the Collector so that nobody can slip fake logs into your system.
Protective habits
- Scrub sensitive values in the Collector before storage, as a second line of defense after careful code.
- Never log request bodies or authorization headers by default.
- If you show a trace ID to users, for example in an error page, remember that it is only a random reference for support, and give it no special power.
- Test that log levels in production do not accidentally print debugging data.
A trace ID is not a password and must never be used as proof of identity or permission. It only matches records. Treat any access control as a separate matter.
19Performance, Cost, and Reliability
Correlation is cheap to add, but logs are often the most voluminous signal, so it pays to know where costs come from and how to watch the health of the correlation itself.
What it costs
Reading the context
Looking up the active span is a quick in-memory step. The cost of correlation is small next to formatting and writing the log.
Extra bytes per line
The IDs add roughly fifty characters of ID data plus the names of the fields to each record. Across millions of lines, that is noticeable but modest.
Batching
The batch processor groups records so the network is used efficiently, and the logging call never waits for it.
Log volume
The biggest cost is the number of log lines, not the IDs. Filtering noisy debug lines saves far more than trimming the IDs.
Reliability: what if export fails?
If the Collector is down, the SDK keeps records in a bounded queue, retries for a while, and then drops the oldest or newest records, depending on settings. The application keeps running. For logs that must never be lost, such as audit records, write them to a durable file or database as well, and do not depend on the telemetry queue alone.
Measuring the health of correlation
You can treat correlation quality as a number to track. Useful measures include the share of log records that carry a trace ID, the share of trace IDs in logs that point to stored traces, and the share of requests where logs from all services share the same trace ID. If these drop after a release, something broke propagation or logging.
The 95 percent figure is a sensible goal, not a rule from the standard. Logs written at startup, shutdown, and from background timers will naturally have no request, so 100 percent is neither expected nor needed.
20Advantages, Disadvantages, and Trade-offs
Correlation brings a large reward for a small effort, but it is not free of costs and limits.
Advantages
- Turns a hunt through text into an exact lookup.
- Works across services and languages, because the IDs travel in a standard format.
- Explains the “why” behind a slow or failing span.
- Lets you start from either side and reach the other.
- Adds only a few bytes and a quick lookup to each record.
- Works with logging libraries you already use, through bridges.
Disadvantages
- Logs written outside a span get no IDs.
- Sampling can leave logs pointing to traces that were never kept.
- Lost context in threads and callbacks silently breaks the link.
- Two export paths can create duplicate logs.
- Joined data can expose sensitive logs to a wider audience.
- Depends on every service forwarding trace context correctly.
Decisions to make on purpose
| Decision | Choose the first when | Choose the second when |
|---|---|---|
| Direct OTLP or file collection | You control the application and want clean fields | You must support older applications or want a crash-safe file |
| Keep all logs or only sampled ones | Text search and audit matter most | Cost matters most |
| Span events or log records | You need a few notes that live with the span | You want rich, filterable records with their own retention |
| Trace ID only or also a business key | Only engineers will search | Support teams will search by order or ticket number |
21Design Patterns and Anti-patterns
Teams that rely on correlation every day keep returning to the same habits. Here are the ones that work, and the traps to skip.
Patterns worth copying
Structured logs everywhere
Every line is a set of named fields. IDs, severity, and business keys are fields, not parts of a sentence.
Start the span early
Begin the request span in the first framework step, so nearly every log in the request falls inside it.
Let the SDK set the IDs
Use the bridge or the instrumentation. Never copy IDs by hand into messages.
One export path per log
Choose direct OTLP or file collection for each source and stick to it.
Errors in both places
On failure, mark the span as an error with its exception, and write a log with the IDs attached.
Central scrubbing
A Collector step removes sensitive fields from logs for all services at once.
Anti-patterns to avoid
The Mistake
Writing the trace ID by hand into the message text, for example by building a sentence that includes it, instead of using the standard fields.
Why It Hurts
Tools cannot recognize the ID as a link. Formats drift between teams, and searches depend on text patterns that break easily.
Do This Instead
Use the log bridge or instrumentation so the IDs appear in dedicated fields.
The Mistake
Sending the same logs twice, once directly through OTLP and once by collecting the console output.
Why It Hurts
Every line is stored twice, doubling cost and making counts wrong. Searches return confusing duplicates.
Do This Instead
Pick one path per source. If both are unavoidable, remove duplicates in the Collector.
The Mistake
Logging on a background thread that has lost the request’s context, then wondering why the lines have no IDs.
Why It Hurts
The most interesting lines, such as failures inside asynchronous work, become the ones you cannot link to any request.
Do This Instead
Carry the context into worker threads and callbacks, and create the log record while the right span is active.
22Best Practices and Common Mistakes
Keep this short list close. It prevents most of the trouble that teams meet when they first connect their logs and traces.
Set a clear service name
Resource facts such as service name and version make the fallback correlation possible and every record easier to place.
Use a bridge or integration
Connect your existing logging library, and avoid writing custom code to copy IDs.
Log in a structured format
Use named fields, including severity, trace ID, span ID, and important business keys.
Propagate trace context everywhere
Make sure gateways, proxies, and queues pass the trace header unchanged.
Record failures on the span and in the log
Set the span status and exception, and write an error log with the IDs.
Decide your sampling and log policy together
Choose which logs to keep for unsampled traces, and make sure failures get full traces.
Track correlation health
Measure the share of logs with IDs and the share of dead links, and alert on sudden drops.
Common mistakes and their fixes
| Mistake | What you will see | Fix |
|---|---|---|
| No log bridge or integration installed | Logs with no IDs at all | Install the logging integration for your language |
| Span started too late | Early request logs have no IDs | Start the span in the first framework step |
| Context lost in worker threads | Gaps in the logs of asynchronous work | Wrap tasks so they carry the context |
| Header stripped by a proxy | Logs after the proxy show a different trace ID | Allow and forward the trace header |
| Log parsing rule too strict | IDs stay inside the text, not in the fields | Fix the parsing rule and test with real lines |
| Both export paths active | Every log appears twice | Keep one path per source |
| Sampled-out traces with kept logs | Links that open to nothing | Use tail sampling for errors, or filter logs by flag |
23Troubleshooting: Why Are My Logs Not Linked?
When the link between logs and traces fails, the cause is almost always found in one of a few places. Work through them in order.
Walk the path from the line to the backend
Is a span active at the log line?
Open a trace of the same request and check whether the log was written inside a span. If the code runs before the span starts or after it ends, there are no IDs to copy.
Is the logging integration installed?
Confirm that the bridge or the context integration is enabled for the logger you actually use. Many programs have more than one logger.
Do the lines show the IDs?
Print a few lines locally. If the fields are empty or all zeros, the SDK found no valid span. If they are present, move on.
Does the Collector keep the IDs?
Check the parsing rules and processors. A rule may fail to extract the IDs from text or may overwrite the fields.
Does the backend know where to look?
Some tools need a setting that says which log field holds the trace ID, and which service field to match. Check that mapping.
Symptoms and likely causes
| Symptom | Likely cause | What to try |
|---|---|---|
| No IDs at all | Integration missing, or no span active | Install the integration and confirm a span exists |
| IDs are all zeros | The SDK found no valid active span | Check where the span starts and how context is carried |
| IDs exist but differ between services | Trace header lost between services | Check proxies and queues for dropped headers |
| Link opens to nothing | The trace was not sampled or the retention ended | Check the sampled flag, and align sampling and retention |
| Some logs have IDs, others do not | Different code paths, threads, or loggers | Find the paths that lose the context |
| Every line appears twice | Both export paths are on | Disable one of them |
| Time order looks wrong | Clock differences between hosts | Fix time synchronization and widen search windows |
When a letter does not arrive, the postal service checks each step: was it posted, was it sorted, was it loaded, was it delivered. Debugging correlation is the same: check if the span existed, if the IDs were copied, if the pipeline kept them, and if the backend can read them.
Test correlation with a single, simple request in a staging environment. Make one request, find its trace, and check the logs. If it works for one, then widen the test.
24Real-World and Industry Examples
First, a made-up story that shows the idea at work. Then, a look at how the wider industry uses the same connection.
An illustrative story: the coupon that broke only on Fridays
Imagine an online shop where a few customers every Friday see “coupon invalid” even though their coupons are fine. Dashboards show only a small, steady error rate. An engineer opens one failing trace from last Friday and sees that the “apply coupon” span in the pricing service took 3 seconds and ended with an error. She clicks to its logs and finds one line: “rules cache expired, reload failed, falling back to defaults”. The cause is a weekly job that locks the rules table for a few minutes every Friday evening. Without the span-level link, she would have scanned thousands of pricing logs by time. With it, the answer took minutes. (This is a made-up example to show the workflow, not a report from a real company.)
Observability platforms and the “view logs” button
Many modern observability backends show a button next to a span that says something like “view related logs”, and a clickable trace ID next to a log line. Both rely on the same fields described in this guide: the trace ID, the span ID, and the service identity from the resource.
Logging frameworks with built-in context fields
Widely used logging frameworks across languages offer a way to attach extra fields to every log line within a request. Instrumentation, including automatic agents for some languages, uses this to place the trace ID and span ID into each line without any change to application code.
The OpenTelemetry Collector as a log router
The Collector can read logs from files, from standard output collected by container platforms, and from OTLP. It can parse trace IDs out of text lines, add facts about the machine or cluster, filter noisy records, and send logs and traces to the same or different backends. It is a common central point where correlation is repaired or enriched.
Support teams and customer reference numbers
Some organizations show a short reference on error pages for customers to quote to support. Behind the scenes, that reference is tied to the trace or a business key. Support staff can then locate the request’s logs and trace within seconds, without asking the customer for technical details.
25Architecture and Components of a Correlated Logging Pipeline
A correlated log passes through several parts on its way to a screen. Naming each part makes it much easier to find where something went wrong.
flowchart LR
A["Application code"] --> LL["Logging library"]
LL --> BR["Bridge or context integration"]
BR --> LP["SDK LoggerProvider"]
CX["Context holding the active span"] -. supplies IDs .-> LP
LP --> PR["Log record processor (batch)"]
PR --> EX["OTLP log exporter"]
EX --> CO["Collector (receivers, processors, exporters)"]
CO --> LB["Log storage"]
CO --> TB["Trace storage"]
LB --> UI["Query screen: logs next to traces"]
TB --> UI
Application and logging library
Your code calls the logger it always used. The library formats the event and decides the severity.
Bridge or context integration
Turns the event into an OpenTelemetry log record, or adds the IDs to the logger’s own context.
SDK and LoggerProvider
Creates the record, reads the active span from the context, and attaches the resource.
Context
The invisible package that holds the active span. Without a correct context, there are no correct IDs.
Processor and exporter
The processor batches records. The exporter sends them using OTLP, without making your code wait.
Collector
Receives, cleans, enriches, and routes logs and traces to the right storage.
Log and trace stores
Hold each signal. They can be one product or two, as long as they agree on the ID fields.
Query and visualization layer
Reads both stores and draws the links: “view logs” on a span, and “open trace” on a log line.
Think of a newspaper office. A reporter writes a story, an editor adds the section and date, the printing room prepares the pages, a delivery van brings them to shops, and a reader finally opens the paper. If a page is missing a date, you do not blame the reader. You walk back along the chain to find which stage forgot it.
Draw this chain for your own system on a whiteboard, with real names for each stage. When a link breaks, you will know exactly which box to inspect first.
26What to Log Inside a Span, and What to Leave Out
Correlation makes each log line more useful, which makes it worth thinking about which lines deserve to exist at all.
A good log line answers three questions
A useful line says what was happening, to what it was happening, and how it ended. For example: “charge attempt for order 8841 declined, reason insufficient funds”. Because the trace ID and span ID are already attached, you do not need to repeat the name of the service or the request in the text. Keep the message short and put details in named fields.
| Worth logging | Better left out or moved |
|---|---|
| Failures, with the reason and the key IDs | Every successful step of a busy loop |
| Decisions that change behavior, such as “fallback used” | Large request or response bodies |
| Warnings from libraries and retries | Secrets, tokens, and personal data |
| State changes, such as “circuit breaker opened” | Messages with no useful content, like “here” or “done” |
| Start and end of important business operations | Timing data that spans already record |
Do not duplicate what the span already says
A span already records its name, start time, duration, and status. A log such as “starting charge” followed by “finished charge in 240 ms” repeats what the span shows and doubles your volume. Use logs for information that spans do not carry well, such as reasons, warnings, and unusual decisions.
Choose severity with care
- Error: an operation failed and someone may need to act.
- Warning: something unexpected happened, but the system coped.
- Info: a notable event in the normal life of the system.
- Debug: detail for developers, usually switched off in production.
A pilot’s flight log does not note every second of a smooth flight. It records departures, arrivals, changes of plan, and anything unusual. The route and speed are already tracked by instruments. Logs and spans share the job in the same way.
Logging inside a tight loop can create millions of lines, each stamped with perfectly valid IDs. Correlation does not make a noisy log useful. It only makes the noise easier to find and more expensive to keep.
27Setting It Up: A Step-by-Step Walkthrough
Here is the whole path, from a plain service to correlated logs and traces, as a series of small steps you can follow in order.
Name your service
Set the service name, version, and environment in the resource. All logs and traces will carry them, which also enables the fallback matching by service and time.
Turn on tracing for the entry point
Use automatic instrumentation for your web framework or messaging library, so that every incoming request starts a span before your own code runs.
Connect your logger
Install the bridge or context integration for the logging library you use, and check that the IDs appear in a test line.
Choose the export path
Either send log records directly with OTLP, or write structured lines to standard output and let the Collector read them. Do not do both for the same logs.
Configure the Collector
Add a logs pipeline next to the traces pipeline. Include batching, parsing of IDs if you use files, and rules to remove sensitive fields.
Map the fields in your backend
Tell the tool which log field holds the trace ID and span ID, and how service names match between logs and traces.
Test with one request
Make a single request, find its trace, and click through to its logs. Then go the other way, from a log line to its trace.
Measure and alert
Track the share of logs with IDs and the share of links that resolve. Alert when they drop.
Start with direct export
Use the SDK bridge and send logs and traces to one Collector. It has the fewest moving parts.
Standardize in the Collector
Give every team the same Collector configuration for logs, so field names and scrubbing rules never vary.
You do not need to change all services at once. Begin with the most important request path, such as checkout, prove that the link works end to end, and then spread the setup to other services.
28Frequently Asked Questions
Quick answers to the questions people ask most often about correlating logs with traces.
29Summary and Key Takeaways
You started with a pile of log lines and a customer who paid at 10:05. You now know how to turn that pile into one clear story.
Key Takeaways
- Correlation means stamping every log with its request. The log record carries the trace ID and span ID of the span that was active when it was written, so you can move between logs and traces in both directions.
- The SDK does the stamping. A logs bridge or logging integration reads the active span from the context and copies the IDs into dedicated fields. You keep using the logger you already know.
- Context is the hidden dependency. Logs written when no span is active, or on threads that lost the context, have no IDs. Start spans early and carry context into asynchronous work.
- One trace ID gathers all services. Because trace context travels in headers, every service writes the same trace ID, and a single search returns the whole story, provided every hop forwards the header.
- Sampling and logs must be planned together. A log may point to a trace that was never stored. Decide which logs to keep, and use tail sampling so failing requests have real traces.
- Record errors in both places. An error log does not mark a span as failed. Set the span status with its exception, and write the log with the IDs attached.
- Protect and measure the link. Scrub sensitive data, distrust outside trace headers, avoid duplicate export paths, and track how many of your logs actually carry IDs.