What Is a Trace in OpenTelemetry?
A beginner-friendly guide to the story of a single request as it travels through your software: what a trace is, how spans build it, how context travels between services, and how to read one with confidence.
Imagine you send a parcel to a friend in another city. The parcel is picked up by a courier, driven to a local depot, sorted, loaded onto a truck, flown on a plane, sorted again, loaded onto a smaller van and finally handed to your friend. If the parcel arrives two days late, you do not just want to know that it was late. You want to know where it waited. A good tracking page shows every stop, with a time next to each one, so the slow step becomes obvious. A trace in OpenTelemetry is that tracking page, but for a request travelling through your software.
In this guide we start from zero. You do not need to know what a span is, what context propagation means, or why anyone cares about identifiers. We will build each idea step by step, using simple pictures and everyday comparisons, until you can read a trace and explain it to a colleague.
1The Problem: Following One Request Through Many Services
Before we define a trace, let us feel the problem it solves.
Software used to be one big program
In the past, many applications were a single large program. When a user clicked a button, one program did all the work, from start to finish, in one place. If it was slow, an engineer could open that one program and look at it. There was only one place to look.
Modern software is a crowd
Today, a single click often wakes up dozens of separate services. Think of an online shop. When you press “Buy”, a web server receives the click, a login service checks who you are, a cart service reads your basket, a stock service checks availability, a payment service talks to a bank, and a shipping service books a delivery. Each service may run on a different computer, be written in a different language and be owned by a different team.
Now imagine the “Buy” click takes eight seconds. Which service was slow? Was it the bank? Was it a database? Was it a service waiting for another service that was waiting for a third? Each team can look at their own dashboards and say “we look fine”. Yet the customer still waited eight seconds.
Picture a relay race where each runner wears a different uniform and only knows about their own lap. The coach can see that the team finished slowly, but not which runner stumbled. A trace is like a camera that follows the baton from the first runner to the last, so you can see exactly where time was lost.
Why logs and metrics alone are not enough
Logs tell you what happened inside one service at one moment. Metrics tell you how a service is behaving on average. Neither connects the dots across services for one particular request. You can read ten log files from ten services and still not be sure which lines belong to the same customer click. Tracing exists to fill exactly this gap.
Many beginners think a trace is just a very detailed log. It is not. A log is a flat list of messages. A trace is a connected structure that shows who called whom, in what order, and how long each step took.
2So, What Is a Trace?
A plain definition first, then the reasons behind each word.
A trace is the complete record of one request or transaction as it moves through all the parts of a system. It is made from a set of smaller records called spans. Every span in the same trace shares one common identifier, called the trace ID, which is how the pieces are recognised as belonging together.
That definition has three important ideas inside it.
One request, one story
A trace represents a single journey, such as one customer clicking “Buy”, one scheduled job running, or one message being processed. It is not an average and not a summary. It is one concrete example.
Made of spans
A trace by itself has no data of its own. It is simply the collection of its spans. Each span describes one piece of work, such as one database query or one call to another service.
Held together by an identifier
All spans in a trace carry the same trace ID. That shared label is the glue. If you remove it, you have a pile of unrelated records.
Think of a trace as a whole parcel journey, and spans as the individual stops on the tracking page: “picked up”, “arrived at depot”, “loaded onto plane”, “out for delivery”. The parcel number printed on every stop is the trace ID.
3Spans: The Building Blocks of a Trace
If a trace is the story, spans are the sentences.
A span represents one unit of work with a clear beginning and a clear end. Reading a file, running a database query, calling another service, or checking a password can each be a span. Because a span has a start time and an end time, it also has a duration, which is simply how long the work took.
What a span contains
Every span carries a handful of standard pieces of information. Learn these and you will understand almost every trace you ever meet.
Name
A short, human-friendly label for the work, such as “process checkout” or “query stock level”.
Span ID
A unique identifier for this one span. No other span in the trace shares it.
Parent Span ID
The identifier of the span that caused this one. The very first span in a trace has no parent.
Start and End Time
Exact timestamps for when the work began and finished. The difference is the duration.
Kind
Describes the role of the span, for example whether it handled an incoming call or made an outgoing one.
Status
Tells whether the work succeeded, failed, or was left unset by the developer.
Attributes
Key-and-value labels that add detail, such as the web address called or the name of the database used.
Events and Links
Events mark noteworthy moments inside the span. Links connect this span to spans in other traces.
A span is like one row in a delivery driver’s logbook. It has a title (“deliver to 12 Park Road”), a start time, an end time, a note saying whether it worked (“delivered” or “nobody home”), and extra remarks such as “rang the bell twice”. Put all the rows from all the drivers together, tied to one parcel number, and you have the full trace.
Because every span records its own start and end, you can always work out where time went. A parent span that took two seconds, with one child span taking 1.8 seconds, tells you immediately that the child is where to look.
4Parents, Children and the Shape of a Trace
Spans are connected like a family tree, and that tree is what gives a trace its meaning.
When one piece of work starts another piece of work, the first span becomes the parent and the new one becomes the child. A child records its parent’s span ID, which is how the connection is remembered. The first span in a trace, with no parent, is called the root span. Together, the spans form a tree, with the root at the top and branches below it.
Let us return to our online shop. A customer clicks “Buy”. The web server handles the click, which becomes the root span. To finish the job, the web server calls the cart service, the stock service and the payment service. Each of those calls becomes a child span. The payment service in turn talks to a bank and writes to its database, creating grandchild spans.
flowchart TD
R["Root Span: Handle Buy Click"] --> A["Child: Check Login"]
R --> B["Child: Read Cart"]
R --> C["Child: Check Stock"]
R --> D["Child: Take Payment"]
B --> B1["Grandchild: Query Cart Database"]
C --> C1["Grandchild: Query Stock Database"]
D --> D1["Grandchild: Call Bank"]
D --> D2["Grandchild: Save Receipt"]
Sequential and parallel work
Some children run one after another, like washing, drying and folding laundry. Others run side by side at the same time, like three cooks preparing three dishes. A trace captures both. Parallel children overlap in time, and sequential children follow each other. This overlap is exactly what you see in the waterfall picture that tracing tools draw, which we will explore later.
The critical path
Not every span matters equally to the total wait. The longest chain of work that the customer must wait for is called the critical path. If the cart service and the stock service run in parallel, only the slower one decides how long the customer waits. Speeding up the faster one will not help at all. Traces make the critical path visible so you can spend your effort where it counts.
When investigating a slow trace, always start with the root span and walk down the longest child at each level. You will usually find the culprit in just a few steps.
5Identifiers and Span Context
Small bundles of identifiers are what let separate services recognise the same journey.
For a trace to hold together across many computers, each span must carry a few key identifiers. In OpenTelemetry, these travel together in a small bundle called the span context. It is the minimal information that must be passed along so the next service can continue the same story.
Trace ID
A long random number shared by every span in the trace. It is large enough that two traces are extremely unlikely ever to share one by accident.
Span ID
A shorter random number that identifies one specific span within the trace.
Trace Flags
Small switches that carry simple decisions, most importantly whether this trace has been chosen to be recorded.
Trace State
Optional extra notes that tracing vendors can add, kept as simple key-and-value pairs.
Think of the span context as the wristband given to a visitor at a festival. The wristband has a colour that says which event the visitor belongs to (trace ID), a serial number for this particular visit (span ID), and a stamp showing whether the visitor has VIP access (flags). Every gate checks the wristband without needing to know the visitor’s life story.
Immutable and portable
The span context is designed to be small, unchangeable once created, and easy to send over any network. That is deliberate. It is the one thing that must cross the boundary between services, so it has to be lightweight and standard. Heavy details such as long descriptions stay inside the span itself, not in the context.
Some people try to stuff extra business information, such as customer names, into identifiers or context fields. Do not do this. Identifiers should be random and meaningless, and personal information belongs, if anywhere, in carefully chosen attributes.
6Context Propagation: How a Trace Crosses Service Boundaries
This is the magic trick that makes distributed tracing possible.
Inside one program, remembering the current trace is easy, because everything shares the same memory. But when a request leaves one service and enters another, the second service knows nothing about the first. Unless something tells it, it will start a brand-new trace, and the story will be cut in half. Context propagation is the practice of carrying the span context along with every request that crosses a boundary.
Injecting and extracting
Propagation has two sides. When a service makes an outgoing call, it injects the span context into the call, usually as an extra label on the message, such as an additional header on a web request. When the receiving service gets the call, it extracts that label and uses it as the starting point for its own spans. The new spans become children of the span that made the call, and the trace continues unbroken.
sequenceDiagram
participant U as Customer
participant W as Web Service
participant P as Payment Service
participant B as Bank
U->>W: Click Buy
Note over W: Start root span, create trace ID
W->>P: Call payment, inject context
Note over P: Extract context, start child span
P->>B: Call bank, inject context
Note over B: Extract context, start child span
B-->>P: Approved
P-->>W: Payment done
W-->>U: Order confirmed
The standard format
For this to work between different teams and different tools, everyone must agree on how the context is written. The widely adopted standard is called the W3C Trace Context. It defines a specific label that carries the trace ID, the parent span ID and the flags in a fixed layout. OpenTelemetry uses this standard by default, which means services written in different languages and even tracked by different vendors can still join the same trace.
Propagators
In OpenTelemetry, the component that performs injection and extraction is called a propagator. You can configure which propagators your service uses. Using the standard one is the safest choice. Older systems sometimes use other formats, and OpenTelemetry can understand several of them, which helps during gradual migration.
Beyond HTTP
Context must also cross message queues, scheduled jobs and other asynchronous paths. In those cases, the context is attached to the message itself, so that whoever picks up the message later can continue the trace or link back to it. If any single hop forgets to pass the context along, the trace breaks into separate pieces, which is one of the most common problems beginners meet.
A request arrives with no context
The first service sees no incoming trace information, so it creates a new trace ID and a root span.
The service makes an outgoing call
Before sending, it injects the current span context into the outgoing message.
The next service extracts the context
It reads the label, recognises the trace ID, and notes the sender’s span as its parent.
New spans join the same trace
Work done by the second service is recorded as child spans carrying the same trace ID.
The pattern repeats down the chain
Each further call injects, and each receiver extracts, until the request has finished its whole journey.
A proxy, gateway or message system in the middle that strips unknown labels will silently break your traces. If your traces suddenly look chopped into small pieces, check whether something between your services is dropping the context.
7Span Kinds: The Role Each Span Plays
Not all spans do the same kind of job, so OpenTelemetry labels them.
Every span has a kind. The kind tells tools how to interpret the span and how to draw relationships between services. There are five kinds, and they describe whether work was received, sent, or done privately.
| Kind | Meaning | Everyday Example |
|---|---|---|
| Server | Handles an incoming request from a remote caller | A web service receiving a checkout request |
| Client | Makes an outgoing request to a remote service and waits | The web service calling the payment service |
| Producer | Creates a message to be processed later, without waiting | Dropping an order into a queue |
| Consumer | Processes a message that was created earlier | A worker picking the order from the queue |
| Internal | Work that stays inside one service and crosses no boundary | Calculating a discount in memory |
Why kinds matter
Tools use span kinds to draw service maps, work out which service called which, and separate network time from processing time. For instance, a client span that takes 500 milliseconds, with a matching server span of 100 milliseconds, suggests that roughly 400 milliseconds were spent on the network or waiting. Without kinds, that insight would be lost.
Think of a restaurant. The waiter taking the order from the guest is the server role. The waiter handing the order to the kitchen is the client role. Putting the order slip on a rail for the next cook is the producer role, and the cook picking it up is the consumer role. Chopping onions alone at the counter is internal work.
8Attributes, Events, Status and Links
These extra details turn a bare timing record into a rich, searchable story.
Attributes: the labels on a span
An attribute is a simple pair made of a name and a value. It adds context to a span so engineers can search and filter. For a web call, useful attributes might include the method used, the address requested and the result code received. For a database span, they might include the type of database and the shape of the query. Good attributes turn the question “which traces were slow?” into “which traces for premium customers in one region were slow?”.
Choose attribute values that you would actually search by, such as order type or region. Avoid values that are different on every single request, like a full timestamp or a random token, because they are almost useless for grouping.
Events: moments inside a span
A span event is a timestamped note inside a span, marking something noteworthy that happened at one instant. A retry being attempted, a cache miss, or an exception being thrown can each be recorded as an event. Unlike a span, an event has no duration. It is a single point in time within the span’s lifetime.
Status: did it work?
Every span has a status with three possible values: unset, ok and error. Unset is the default and means nothing special was declared. Error means the work failed. Ok is meant to be set deliberately when a developer wants to say definitively that the work succeeded. Setting the status properly is what lets tools count failures and highlight broken traces in red.
Forgetting to mark failed spans as errors is a classic oversight. The work may have raised an exception, yet the span looks perfectly healthy in the tool, hiding real problems from your dashboards and alerts.
Links: connecting separate traces
Sometimes one piece of work is related to another but is not a child of it. A batch job that processes a hundred messages might want to remember each of the hundred original traces that produced those messages. A span link is a pointer from one span to another span, possibly in a different trace. It says “I am related to that one”, without claiming a parent and child relationship.
Parent and Child
One piece of work directly causes another, and the caller usually waits or clearly owns the result.
Link
Work is related but independent, such as batch processing, fan-in from many sources, or follow-up work triggered long afterwards.
9Semantic Conventions: Speaking a Common Language
Agreeing on names is what makes traces from different teams comparable.
If one team names an attribute “http_url”, another names it “web.address” and a third names it “URL”, then searching across teams becomes a nightmare. OpenTelemetry solves this with semantic conventions, a published set of standard names and meanings for common situations such as web calls, database access, messaging and cloud services.
When your instrumentation follows these conventions, any tool that understands them can automatically recognise, for example, that a span represents a database call, show the database name, and group similar calls together. Think of it as agreeing that everyone writes dates the same way. It removes confusion and unlocks shared tooling.
Resource attributes
Describe who produced the trace, such as the service name, its version and the environment it runs in. They are attached once and apply to every span from that service.
Span attributes
Describe what a particular piece of work did, such as which route was called or which database operation ran.
Standard naming for span names
Conventions also suggest short, low-variety names, such as the route pattern of a web call rather than the full address with real identifiers in it. This keeps the number of distinct span names manageable.
Always set a clear service name. A trace where every service is called “unknown” is almost impossible to read, no matter how well everything else is instrumented.
10How Traces Are Created Inside an Application
A short tour of the machinery that turns program activity into trace data.
OpenTelemetry provides a small set of cooperating pieces inside your application. You do not have to master them, but knowing their roles helps you understand how a trace moves from “work happening” to “data on a screen”.
Tracer Provider
The factory that sets up tracing for the whole application and hands out tracers. It holds the configuration.
Tracer
The object that actually starts spans. Each library or module usually gets its own named tracer.
Sampler
Decides whether a new trace should be recorded or ignored, helping to control volume and cost.
Span Processor
Receives finished spans and prepares them for shipping, often grouping them into batches.
Exporter
Sends the prepared spans to a destination, such as a Collector or a tracing tool, using a standard protocol.
Resource
The description of the producer, such as service name and environment, attached to all outgoing spans.
flowchart LR
W["Application Work"] --> T["Tracer starts a span"]
T --> S["Sampler decides: record or ignore"]
S --> SP["Span Processor batches finished spans"]
SP --> EX["Exporter sends data"]
EX --> CO["OpenTelemetry Collector"]
CO --> BE["Tracing Backend and Dashboard"]
The active span and current context
Inside a running program, OpenTelemetry keeps track of which span is currently active. When new work starts, the tracer looks at the current context, finds the active span, and makes the new span its child. This is why you rarely need to pass parent information by hand within one service. The context quietly follows the flow of your program, including across waiting and resuming in many languages.
Spans are sent when they finish
An important detail beginners often miss: a span is normally sent to the exporter only after it has ended. This means a span that never ends, perhaps because of a programming mistake, may never be reported at all. It also means the root span, which finishes last, often appears in your tool a little after its children.
Traces are assembled by the backend
Applications do not send a “complete trace” as one package. Each service independently sends its own spans as they finish. The tracing backend collects spans from everywhere, groups them by trace ID, uses the parent span IDs to rebuild the tree, and then draws the final picture. The trace you see on screen is a reconstruction.
11Automatic and Manual Instrumentation
There are two ways to teach your software to produce traces.
Instrumentation simply means adding the ability to produce telemetry. For traces, there are two broad approaches, and most real systems use both.
Automatic Instrumentation
- Little or no change to your own code
- Quickly covers common libraries such as web frameworks, database drivers and message clients
- Follows standard naming rules automatically
- Excellent for getting a first useful picture within hours
Manual Instrumentation
- You decide exactly which business steps become spans
- You can add attributes that matter to your own domain, such as order type
- Requires developers to write and maintain extra code
- Easy to forget or to overdo, creating gaps or noise
A sensible approach
Begin with automatic instrumentation to get the skeleton of your traces: incoming requests, outgoing calls and database work. Then add manual spans and attributes only around the business steps that genuinely help you understand or debug your system, such as “calculate price” or “apply fraud rules”. This gives you strong coverage with modest effort.
Automatic instrumentation is like installing security cameras at every door of a building. You instantly see who comes and goes. Manual instrumentation is like placing a camera inside the one room where something important happens. The door cameras give breadth, and the room camera gives depth.
Creating a span for every tiny function call. Thousands of microscopic spans make traces huge, slow to load and hard to read. A good span represents meaningful work, not every line of code.
12Sampling: Choosing Which Traces to Keep
Recording every trace in a busy system is often too expensive, so we choose wisely.
A busy online shop might handle millions of requests an hour. Storing a full trace for every one would require a huge amount of space and money, and most of those traces would be dull, successful and nearly identical. Sampling means keeping only some traces while still learning what you need.
Head sampling
The keep-or-drop decision is made at the very beginning of the trace, when the root span is created. A common approach is to keep a fixed percentage, for example one in ten. The decision is then passed along in the trace flags, so every other service follows the same choice. This keeps traces complete, because either all spans are recorded or none are. It is cheap and simple, but it cannot know in advance which traces will turn out to be slow or broken.
Tail sampling
The decision is made after the trace has finished, usually in the Collector. Now the system can keep every trace that contained an error, every trace slower than a chosen limit, and only a small slice of the ordinary ones. The results are far more useful, but this approach needs more memory and a careful design, because every span of one trace must arrive at the same place before a decision is made.
| Question | Head Sampling | Tail Sampling |
|---|---|---|
| When is the decision made? | At the start of the trace | After the trace is complete |
| Can it keep all errors? | No, it cannot know yet | Yes |
| Complexity | Low | Higher |
| Resource needs | Very small | Memory to hold unfinished traces |
| Where it usually runs | In the application | In the Collector |
Head sampling is like a bouncer at a club door who lets in one person out of every ten, no matter who they are. Tail sampling is like a talent scout who watches the whole show first and then decides which performances deserve to be remembered.
If you are just beginning, start by recording everything in a small test environment so you can learn what traces look like. Introduce sampling only when volume or cost makes it necessary.
13Reading a Trace: The Waterfall View
Most tracing tools draw a trace as a waterfall, and learning to read it is a superpower.
In a waterfall view, each span is drawn as a horizontal bar. The bar starts where the span started and ends where it ended, along a shared time axis. Child spans are indented beneath their parents. The result looks like a staircase of bars flowing down and to the right, which is where the name comes from.
What to look for
Find the longest bars
The widest bars show the spans that took the most time. Start there, but remember that a parent’s width includes its children.
Look for gaps
A blank stretch between a parent bar and its children means time was spent in the parent’s own work, or that something is missing from your instrumentation.
Spot sequential patterns
A long staircase of similar bars one after another often reveals the same call being repeated many times, a famous problem sometimes called the “N plus one” pattern.
Check for overlap
Bars that sit side by side in time show parallel work. If work that could be parallel is drawn one after another, you may have found an easy speed-up.
Notice red marks
Spans with an error status are usually highlighted. Click them to read their attributes and events and learn what went wrong.
A worked example in words
Suppose the “Buy” click took eight seconds. You open the trace and see the root span spanning eight seconds. Below it, the login check took 50 milliseconds, the cart read took 100 milliseconds, the stock check took 200 milliseconds, and the payment took 7.5 seconds. Inside the payment span, you see that the call to the bank lasted 7.3 seconds. Within one minute, you know that the slow step is the bank call, not your own code, and you can have a sensible conversation with the right people.
Always compare a slow trace with a normal trace for the same operation. Seeing the two side by side often makes the difference jump out immediately.
14Traces Together with Metrics and Logs
Each signal is strongest when it can point to the others.
OpenTelemetry treats traces, metrics and logs as members of one family, and it is designed so they can reference each other. This is called correlation.
Traces to Logs
When a log message is written during a span, the trace ID and span ID can be stamped on it. Then from a slow span you can jump straight to the exact log lines written during it.
Metrics to Traces
Metrics can carry small samples called exemplars that point at a specific trace. From a spike on a chart, you can open an example trace that contributed to it.
Traces to Metrics
Spans can be counted and timed to produce request-rate, error-rate and duration metrics, giving you dashboards without extra work in applications.
Shared Resources
All three signals carry the same resource description, so everything from one service lines up under the same name.
Think of a hospital again. The heart-rate chart (metrics) shows that something went wrong at 3 p.m. The patient’s journey sheet (trace) shows which departments they visited around that time. The nurse’s notes (logs) explain what was said and done in each room. With a shared patient number on all three, the doctor can move between them in seconds.
A typical investigation flow
Many teams follow the same path. An alert fires because a metric crossed a limit. The engineer clicks an example trace attached to that metric. In the trace waterfall, they find the slow or failing span. From that span, they open the correlated logs to read the exact error message. Three signals, one smooth journey, all connected by identifiers.
flowchart LR
AL["Alert fires from a metric"] --> EX["Open an example trace"]
EX --> SP["Find the slow or failing span"]
SP --> LG["Read logs linked to that span"]
LG --> FX["Understand and fix the cause"]
15Security and Privacy in Tracing
Traces describe real user activity, so they must be handled with care.
Because a trace follows real requests, it can accidentally capture sensitive information. Attributes might contain email addresses, session tokens, full web addresses with secret values in them, or even fragments of database queries with personal data. Treat trace data like any other data that could identify a person or reveal secrets.
What to protect
Personal information
Avoid putting names, emails, phone numbers or government identifiers into span names or attributes. If you need to group by user, consider a harmless internal identifier instead.
Secrets and credentials
Never record passwords, access tokens, keys or authorization headers. Automatic instrumentation can sometimes capture more than you expect, so review what is collected.
Query contents
Database spans may include query text. Prefer recording the shape of a query rather than the real values filled into it.
Where to enforce safety
The best defence is layered. Developers should avoid adding sensitive attributes in the first place. A central OpenTelemetry Collector can then act as a safety net, removing or masking sensitive values before data leaves your network. Data in transit should use encrypted connections, access to the tracing tool should be limited to the people who need it, and retention should be set to keep data only as long as it is useful.
Be careful with propagated context
Trace context sent to third-party services reveals that a trace exists and gives them your identifiers. This is usually harmless, but you should be thoughtful about whether to propagate context to outside parties, and also about whether to trust incoming context from untrusted callers, such as the public internet, which could be used to force expensive recording decisions.
Problem
A developer adds the customer’s email address and full payment details as span attributes to make debugging easier.
Why It’s Harmful
Sensitive data is now stored in a tracing tool that many people can access, possibly breaking privacy rules and increasing the damage of any leak.
Correct Approach
Record a harmless internal identifier or a coarse category instead, and add a redaction rule in the Collector as a second line of defence.
16Best Practices and Anti-patterns
Habits that keep traces useful instead of noisy.
Good habits
Name services and spans clearly
Use stable, readable service names and short span names that describe the kind of work, not the specific data involved.
Propagate context everywhere
Make sure every hop, including queues, gateways and background jobs, passes the context along. One missing hop breaks the story.
Set status and record errors
Mark failed spans as errors and attach the failure details as an event so problems stand out.
Follow semantic conventions
Use the standard attribute names so tools and teammates understand your data without a dictionary.
Send traces through a Collector
It lets you change destinations, redact data and sample intelligently without touching application code.
Stamp logs with trace identifiers
This single habit makes jumping between traces and logs effortless and saves hours during incidents.
Mistakes to avoid
Problem
Span names contain real data, such as a full web address with an order number inside it.
Why It’s Harmful
Every request creates a brand-new span name, which makes grouping, searching and statistics nearly impossible and can inflate storage costs.
Correct Approach
Use a generic name for the route or operation, and put the specific value into an attribute where it can be handled sensibly.
Problem
Every small function in the code becomes its own span.
Why It’s Harmful
Traces become enormous and slow to render, and the important steps disappear in the noise. Overhead on the application also rises.
Correct Approach
Create spans for meaningful units of work such as network calls, database operations and key business steps.
Problem
Only some services are instrumented, and the rest are left dark.
Why It’s Harmful
Traces show a long gap where an uninstrumented service sat, and engineers cannot tell what happened inside it. The tool shows a mystery rather than an answer.
Correct Approach
Roll out automatic instrumentation across every service in the critical paths first, then deepen coverage over time.
Problem
Each service makes its own independent sampling choice, with different percentages and no shared decision.
Why It’s Harmful
Traces arrive with some services present and others missing, producing broken and misleading pictures.
Correct Approach
Make the decision once at the start and let every service respect it through the trace flags, or use tail sampling in a central place.
17Advantages and Trade-offs of Tracing
Tracing is powerful, but it is not free.
Advantages
- Shows exactly where time is spent across many services
- Makes the critical path of a request visible
- Reveals hidden dependencies and surprising call patterns
- Speeds up root-cause investigation during incidents
- Works across languages and tools thanks to open standards
- Connects naturally to logs and metrics for full context
Disadvantages / Trade-offs
- Adds a small processing and network cost to applications
- Produces a lot of data, which can become expensive to store
- Needs every service to pass context correctly to stay complete
- Sampling means some individual requests will not have a trace
- Poor naming and noisy spans can make traces confusing
Traces, metrics and logs compared
| Signal | Best Question It Answers | Granularity | Typical Weakness |
|---|---|---|---|
| Trace | Where did this request spend its time? | One request at a time | Can be costly at high volume |
| Metric | How healthy is the system overall? | Aggregated over time | Hides individual requests |
| Log | What exactly happened at this moment? | One event at a time | No built-in link between services |
“Tracing replaces metrics and logs.” It does not. Metrics tell you that something is wrong, traces tell you where, and logs often tell you why. The three work best together.
18Real-World Scenarios
Picture how different teams use traces to solve everyday problems.
An online shop finds a slow checkout
Customers complain that paying sometimes takes ten seconds. Engineers search for slow checkout traces and notice that in nearly all of them, one external fraud-check call is the long bar. They discuss the issue with the provider and add a sensible time limit with a fallback. Complaints disappear.
A streaming service spots repeated database calls
A page that should load quickly is slow. The waterfall shows a staircase of one hundred nearly identical database spans in a row. The team realises the code asks the database one question per item instead of asking once for all items. After the fix, the staircase collapses into a single bar.
A bank follows a message through a queue
A transfer request drops into a queue and is processed minutes later. By carrying trace context inside the message, the team can see the producer span, the waiting time in the queue and the consumer span, all linked. They discover that messages sit idle for long periods when workers are too few.
A platform team maps its own architecture
A large company is unsure which services depend on which. Because traces record who called whom, the tracing tool automatically draws a service map. The team discovers several forgotten dependencies and removes unused ones, simplifying their system.
A support engineer reproduces a customer’s problem
A customer reports an error and gives a request reference. Because that reference is the trace ID, the engineer opens the exact trace, sees the failing span, and reads the linked logs. No guesswork and no long email chains asking for more details.
19Getting Started: A Gentle Path
You do not need to master everything at once.
Pick one important user journey
Choose something visible, such as checkout or sign-in, so you can see the value quickly.
Turn on automatic instrumentation
Enable the OpenTelemetry option for your language and framework, and give each service a clear name.
Send data to a Collector and a viewer
Use a free open-source tracing tool or your existing vendor to display the traces.
Read your first traces
Open a few, find the root span, follow the longest branch, and notice anything surprising.
Add a few meaningful manual spans
Mark key business steps and add attributes you would genuinely search by.
Add sampling, redaction and correlation
Once value is proven, control cost, protect privacy and link traces with logs and metrics.
Your first goal is not perfection. It is to see one real trace crossing at least two services. That moment, when the pieces join into one picture, is when tracing finally “clicks”.
20Frequently Asked Questions
A span is one piece of work with a start and an end. A trace is the whole collection of spans that belong to one request, all sharing the same trace ID. Spans are the parts, and the trace is the whole story.
It is a long random number, generally written as a string of letters and digits. It is long enough that the chance of two different traces receiving the same one is extremely small. You do not need to memorise it, only to copy it when you want to find a trace.
No. Traces can describe any unit of work, including background jobs, scheduled tasks, message processing and command-line programs. Anything with a clear start, end and sequence of steps can be traced.
Almost always because context was not passed along at some hop. A gateway, queue or library in the middle may have dropped or ignored the propagation label, so the next service started a new trace. Check each boundary one by one.
No. Profiling looks deep inside one program to see which code lines use the most processor time. Tracing looks across many services at the path of one request. They answer different questions and can be used together.
It adds a small cost, because creating spans and sending data takes some effort. With sensible instrumentation, batching and sampling, this cost is usually small compared to the benefit. Creating far too many spans is the main way people make it noticeable.
No, applications can send traces straight to a compatible backend. A Collector is recommended as systems grow because it centralises redaction, sampling and destination changes.
It is the first span in a trace, the one with no parent. It usually represents the entry point of the request, such as the moment a web server received a call, and its duration normally covers the whole journey.
A span has a start, an end and a duration. An event is a single moment recorded inside a span, with just one timestamp. Use spans for pieces of work and events for noteworthy instants within that work.
If the later work is a direct continuation and you want one connected tree, a parent and child relationship works well. If the relationship is looser, such as one batch job handling many separate messages, links are a better fit because they connect traces without forcing a single tree.
Yes. By showing which services are called most often and which calls take the longest, traces highlight where optimisation will pay off. Teams use them to remove wasted calls, parallelise work and decide where to add capacity.
It can be, provided you avoid recording sensitive values, apply redaction in a central Collector, encrypt data in transit, limit who can view traces, and keep data only as long as needed. Plan for privacy from the start rather than adding it later.
21Summary and Key Takeaways
A trace in OpenTelemetry is the story of one request as it travels through your system. It is built from spans, each describing a single piece of work with a name, a start and end time, a status and helpful attributes. Spans are tied together by a shared trace ID and arranged into a tree by parent identifiers. Context propagation carries the small span context across every boundary, so separate services can join the same story. The application creates spans with a tracer, a sampler decides which traces to keep, and exporters deliver them to a Collector and a backend, which rebuilds the tree and draws the waterfall. When traces are linked with logs and metrics, you can move from an alert to a root cause in a few clicks. Treat them with care: name things clearly, avoid sensitive data, keep spans meaningful, and sample wisely.
Key Takeaways
- A trace is one request’s journey — it is the collection of all spans that share the same trace ID, showing who called whom and how long each step took.
- Spans are the building blocks — each has a name, span ID, parent, start and end time, kind, status, attributes, events and optional links.
- Parents and children form a tree — the root span sits at the top, and the longest chain of work is the critical path you should optimise first.
- Context propagation holds it together — services inject the span context into outgoing calls and extract it from incoming ones, using the standard W3C Trace Context by default.
- Backends assemble the trace — each service sends its own spans when they finish, and the backend groups them by trace ID and rebuilds the tree.
- Sampling controls cost — head sampling is simple and cheap, tail sampling is smarter but needs more resources and careful design in the Collector.
- Correlate and protect — link traces with logs and metrics through identifiers, follow semantic conventions, and keep sensitive data out of spans.