The idea is simple, and once you see it, you will notice it everywhere. A span can start other spans while it is still running, and the span that started the work is called the parent.

Start with the word “span”

A span is one timed piece of work. It has a name, a start time, an end time, and some labels. “Handle the checkout request” is a span. “Run this database query” is a span. “Call the payment service” is a span.

Now add the relationship

When one piece of work starts another piece of work, the first span becomes the parent and the new span becomes its child. The child is a smaller step done on behalf of the parent. The parent usually waits for its children to finish before it can finish itself, although there are important exceptions that we will cover later.

A span with no parent is called the root span. It is the very first piece of work in the story, often the arrival of a user request. All the spans that belong to the same story, linked by parent-child connections, form a trace. Because every span has at most one parent, the trace is shaped like a family tree turned upside down, with the root at the top.

A small tree you can picture

Look at the diagram below. A checkout request arrives, and the service does three things: it validates the cart, it charges the card, and it saves the order. Charging the card means calling another service, which itself does a fraud check.

flowchart TB
    R["POST checkout (root span, SERVER, Order Service)"]
    V["validate cart (INTERNAL, Order Service)"]
    C["charge card (CLIENT, Order Service)"]
    P["POST charge (SERVER, Payment Service)"]
    F["fraud check (INTERNAL, Payment Service)"]
    S["save order (CLIENT, Order Service)"]
    D["INSERT order (database call)"]
    R --> V
    R --> C
    C --> P
    P --> F
    R --> S
    S --> D

Read the arrows as “started”. The root started three children. The “charge card” span started the span in the payment service, which started a fraud check. Two spans that share the same parent, such as “validate cart” and “save order”, are called siblings.

Timing numbers alone are almost useless. The relationship between spans is what gives those numbers meaning.

A flat list tells you very little

Suppose a tool gave you only a plain list: “validate cart, 20 ms”, “charge card, 900 ms”, “fraud check, 850 ms”, “save order, 40 ms”. You would see big numbers, but you could not tell whether they add up, overlap, or belong to different requests. Is the fraud check part of the card charge, or a separate slow step? You would be guessing.

With the parent-child links, the answer is obvious. The fraud check is a child of the card charge. The card charge took 900 milliseconds, and 850 of those were spent inside the fraud check. The place to look is no longer a mystery.

Questions the relationship lets you answer

A short history

The tree-of-spans idea did not start with OpenTelemetry. It was popularized by a 2010 paper from Google about an internal tracing system called Dapper. Open-source tools such as Zipkin were inspired by it. Later, the OpenTracing project described relationships with two reference types, “child of” and “follows from”. When OpenTracing and OpenCensus merged into OpenTelemetry in 2019, the model kept the parent-child tree and replaced “follows from” with a more flexible feature called span links, which you will meet in Chapter 9.

A few simple terms come up again and again. Learn them now, and the rest of the guide will feel easy.

Parent is not the same as caller, exactly

In everyday speech we say “service A calls service B”. In tracing, we get more precise. The client span inside service A, which represents the outgoing call, is the parent of the server span inside service B, which represents handling that call. The parent-child link is between spans, not between whole services.

Depth and breadth

The depth of a tree is how many levels of parents and children it has, and the breadth is how many children a single parent has. A deep tree usually means a long chain of services calling services. A wide tree often means one span doing many small things, or fanning out to many parallel calls. Both shapes are worth noticing because each points to different performance risks.

There is no magic here. The relationship is stored as a few plain IDs, and a backend uses them to rebuild the tree.

Three IDs do all the work

That is the entire mechanism. A child span copies the trace ID from its parent, makes a new span ID for itself, and records its parent’s span ID in the parent field. A backend that receives thousands of spans can group them by trace ID, then connect them by matching each parent span ID to some other span’s own ID.

The span context: a tiny passport

The part of a span that is safe to pass around is called its span context. It is small and unchangeable. It contains:

• the trace ID,
• the span ID,
• trace flags, including a bit that says whether the trace is being sampled,
• optional trace state, a place for vendor-specific hints,
• a marker that says whether the context came from a remote service.

The parent’s span context is exactly what a child needs to be born. Nothing else about the parent, such as its attributes or its end time, has to be known by the child. This is why the relationship can cross network borders so easily: only the small passport needs to travel.

Local parent and remote parent

If the parent runs in the same process as the child, it is a local parent. If the parent’s context arrived through an incoming message, such as an HTTP header, it is a remote parent. The span context records this difference with its remote marker. Remote parents are what stitch many services into one trace.

Your code rarely says “this span’s parent is span number 7”. Instead, the SDK works out the parent for you by looking at something called the context. Understanding this is the key to fixing most broken traces.

Context: who is the “current” span?

Context is a small invisible package that follows your code as it runs. One of the things it holds is the active span, the span that is currently “in progress” for this line of execution. When your code starts a new span and does not say otherwise, the SDK looks into the current context, finds the active span, and uses it as the parent.

The decision, step by step

flowchart TB
    A["Code asks the tracer to start a span"] --> B{"Was a parent context given explicitly?"}
    B -- Yes --> C["Use that context to find the parent"]
    B -- No --> D{"Is there an active span in the current context?"}
    D -- Yes --> E["Make the active span the parent"]
    D -- No --> F["Create a root span with a new trace ID"]
    C --> G["Child copies trace ID and records parent span ID"]
    E --> G

There are three outcomes. If you pass a context explicitly, the SDK uses it. If you do not, it uses the active span from the current context. If neither exists, the new span becomes a root and starts a brand-new trace. You can also deliberately force a root span by starting it with an empty context, which is useful when a background job should not be tied to the request that happened to trigger it.

Starting a span is not the same as making it active

This is the most common surprise for beginners. In OpenTelemetry’s design, simply creating a span does not automatically make it the active span. A second step, often called “make current” or “activate”, puts it into the context. Most languages give you a helper that does both steps at once, such as a “start as current span” style function. If you create a span but never activate it, the next span you create will not see it, and will attach to some older parent or become a root.

How each language carries the context

Languages store the current context in different ways, and this explains many language-specific bugs.

Inside one program, context lives in memory. Between two services, it must be sent over the network. This is where a trace either becomes a beautiful end-to-end story or falls into pieces.

Inject and extract

When service A calls service B, a component called a propagator writes the current span context into the outgoing request, usually as HTTP headers. This step is called inject. When service B receives the request, its propagator reads those headers and rebuilds the span context. This step is called extract. The extracted context becomes the remote parent of the first span that service B creates.

The default format is the W3C Trace Context standard, which uses a header named traceparent. It has four dash-separated parts: a version, the trace ID, the parent ID, and flags. That third part is the span ID of the caller’s span. Its name in the standard, “parent ID”, tells the whole story: it is the identity of the parent that the receiver should use.

One request, three services

sequenceDiagram
    participant U as User Browser
    participant G as API Gateway
    participant O as Order Service
    participant P as Payment Service
    U->>G: HTTP request
    Note over G: Creates root SERVER span S1
    G->>O: Call with trace T1 and parent ID S2 (client span)
    Note over O: Creates SERVER span S3 with remote parent S2
    O->>P: Call with trace T1 and parent ID S4 (client span)
    Note over P: Creates SERVER span S5 with remote parent S4
    P-->>O: Response
    O-->>G: Response
    G-->>U: Response

Follow the IDs. The trace ID, T1, never changes. At every hop, the caller creates a client span, and its span ID is placed in the header. The receiver creates a server span whose parent is that client span. So the chain is: gateway root span, gateway client span, order service server span, order service client span, payment service server span. Each service writes its own spans and sends them to the backend on its own. The backend later joins them through the IDs.

Why parent-child works across languages and vendors

Because the header format is a public standard, the gateway can be written in one language, the order service in a second, and the payment service in a third. As long as each one reads and writes the same header, the tree stays connected. This is one of the biggest practical gifts of OpenTelemetry to teams with mixed technology stacks.

When the chain breaks

Every span carries a label called its kind. The kind does not change the parent-child rule, but it tells tools what role the span plays, and that helps them draw correct pictures.

The natural pairs

Kinds usually come in pairs across a border. A CLIENT span in one service is the parent of a SERVER span in the next. A PRODUCER span is paired with a CONSUMER span. INTERNAL spans fill the space in between. When tools see these pairs, they can safely say “service A called service B”, and they can measure network time as the gap between the end of the server span and the end of the client span.

flowchart LR
    CL["CLIENT span (Order Service)"] -- parent of --> SV["SERVER span (Payment Service)"]
    SV -- parent of --> IN["INTERNAL span (fraud check)"]
    PR["PRODUCER span (Order Service)"] -. message carries context .-> CO["CONSUMER span (Email Service)"]

Most tracing tools show a trace as a waterfall, a stack of horizontal bars. Parent-child relationships decide how those bars are indented and how you should read them.

The shape of a waterfall

Each bar is one span. Its left edge is the start time and its length is the duration. Children appear indented under their parent and usually sit inside the parent’s time range. Sibling bars that sit one after another show sequential work. Sibling bars that overlap show parallel work.

An example with real numbers

Say the “POST checkout” root span takes 300 milliseconds. Here is how its children could be laid out.

Self time: what the parent did by itself

A parent’s own work is called its self time. It is the parent’s duration minus the time covered by its children. For “charge card”, the duration is 200 ms and the fraud check child covers 150 ms, so its self time is about 50 ms. That 50 ms is spent in the card-charging span itself, for example preparing the request and waiting on the network. For the root, the children cover 20 + 200 + 40 = 260 ms of the 300 ms, leaving about 40 ms of self time spent between steps.

If children overlap, you should not add their durations. Count only the time covered by at least one child. Tools calculate this for you, but knowing the rule helps you avoid wrong conclusions.

The critical path

The critical path is the chain of spans that determines the total length of the trace. In our example, “validate cart”, then “charge card” with its fraud check, then “save order” run one after another, so all of them sit on the critical path. If two siblings run in parallel, only the longer one is on the critical path, because shortening the shorter one would not make the request faster. Always optimize spans on the critical path first.

Surprises you may see

• A child that starts before its parent. Different machines have slightly different clocks. A small clock difference can make a child look as if it began before its parent. This is called clock skew, and it is normal across services.
• A child that ends after its parent. In asynchronous work, a parent may finish and return a response while a child keeps going in the background. This is allowed.
• A parent much longer than its children. This means time was spent in the parent itself or in uninstrumented code. It is a hint to add more spans.

Real systems are not always shaped like clean trees. Sometimes one piece of work is caused by several earlier pieces, or by something from a completely different trace. For these cases OpenTelemetry offers links.

What is a link?

A span link is a pointer from one span to another span’s context, without making it the parent. The link says “I am related to that span, but I was not started by it”. A link may point to a span in the same trace or in a different trace, and it can carry its own small set of attributes to explain the relationship.

When to use a link instead of a parent

flowchart TB
    M1["Message 1 (trace A)"] -. link .-> B["Batch job span (new trace)"]
    M2["Message 2 (trace B)"] -. link .-> B
    M3["Message 3 (trace C)"] -. link .-> B
    B --> W["write results to database (child of batch job)"]

Imagine a job that collects one hundred messages from a queue and processes them together as a batch. Each message came from a different original request, so each belongs to a different trace. A batch span cannot have one hundred parents. Instead, the batch span starts its own trace and holds one hundred links, each pointing to the message that contributed to it. Someone investigating a slow message can follow the link to the batch, and someone investigating the batch can follow the links back to the messages.

Modern programs do many things at once. When work moves between threads or tasks, the context that holds the active span has to move too. If it does not, parent-child links quietly vanish.

The problem in plain words

Think of the context as a sticky note placed on a worker’s desk. When the worker hands part of the job to a colleague at another desk, the sticky note stays behind unless someone copies it. The colleague then starts a new span, looks at the desk, finds no note, and creates a root span. The trace splits in two, and nobody knows why.

How to keep the link

• Prefer the language’s built-in async features that carry context automatically, where they exist.
• Use the SDK’s context-wrapping helpers for thread pools and task runners, or the automatic agent that does it for you.
• In languages that pass context explicitly, make passing the context object a habit in every function that does traced work.
• When work will clearly outlive the request and should not be a child, start a new root span and add a link back to the originating span.

Message queues break the usual rule that the caller waits. The sender drops a message and moves on, and the receiver may pick it up seconds or hours later. This makes the parent-child question more interesting.

How context travels through a message

HTTP has headers. Messages have their own header-like fields, often called properties or attributes. When a service sends a message, the propagator injects the span context into those fields. When another service receives the message, it extracts the context from them. The idea is the same as with HTTP, but the time gap between sender and receiver can be large.

Two common shapes

The semantic conventions for messaging describe which spans to create and how to connect them, and they have been refined over time. Check the current version for your messaging system before choosing, and remember that the choice is partly a design decision about how long you want one trace to last.

Now let us follow what actually happens, step by step, from the moment a child span is created to the moment a dashboard draws the tree.

flowchart LR
    A["Gateway SDK (spans S1, S2)"] --> CO["Collector"]
    B["Order Service SDK (spans S3, S4)"] --> CO
    C["Payment Service SDK (span S5)"] --> CO
    CO --> BE["Backend: group by trace ID"]
    BE --> TR["Rebuild tree by matching parent span IDs"]
    TR --> UI["Waterfall view for engineers"]

Why no service needs the whole tree

This design is clever because every service knows only two things: its own span, and the ID of the span that started it. No service has to know how deep the tree is, how many services are involved, or who sits above its parent. This keeps the work small and the system scalable. The full picture only exists at the end, in the backend.

Orphans: spans whose parent never arrived

If a parent span is lost, because it was dropped, never exported, or still waiting in a queue, its children become orphans. Tools handle orphans in different ways. Some show a placeholder where the parent should be, and some show the orphan as a top-level span. A growing number of orphans is a clear sign of lost data or broken export.

The best way to trust an idea is to do it yourself on paper. Here is a tiny set of span records, and you will rebuild the tree exactly the way a backend does.

The raw records

Five spans arrive at a backend, in no special order. To keep the numbers short, we write the IDs as simple names instead of long hexadecimal strings. Every record has the same trace ID, T1, so we know they belong together.

The rebuilding steps

Notice what just happened. The records arrived in the opposite order from the tree, with the deepest span first and the root fourth. It did not matter. The parent IDs alone were enough to rebuild the structure. Notice also what is missing: no single record knew the whole tree. Each knew only its own ID and its parent’s ID.

What if one record never arrives?

Remove the record for “charge card” (S3) from the table. Now “POST charge” (S4) points to a parent, S3, that does not exist. A backend would treat S4 as an orphan, and the payment branch would float disconnected from the rest. The tree does not crash, but it shows a gap, and the gap itself is a clue that data was lost or is late.

Traces are not alone. Logs and metrics describe the same system, and the span relationship is the thread that can connect them all.

Logs that know their span

When your code writes a log message while a span is active, the SDK can copy the current trace ID and span ID onto the log record. The log line is then pinned to one exact box in the tree. If a payment fails, you can open the failing child span and read only the log lines written inside it, instead of searching through a million unrelated lines. In the other direction, you can start from an error log and jump up to the whole trace that produced it, seeing the parents that led to the failure.

Choosing the right signal for each question

At high traffic, you cannot keep every trace. Sampling decides which ones to keep, and it must respect parent-child relationships, or you end up with trees that have missing branches.

Why parents and children must agree

Suppose the gateway decides to keep a trace, but the order service independently decides to drop its part. The saved trace would show a gateway span with a missing branch where the order service should be. The picture lies by leaving things out. To prevent this, the sampling decision is made once, at the root, and carried to every child through the sampled flag inside the trace context. Children that see the flag simply follow it.

The standard tool for this is the parent-based sampler. For a root span, it asks a normal sampler, such as a ratio sampler, to decide. For a child span, it copies the parent’s decision. The result is that a trace is either fully kept or fully dropped, and the tree stays whole.

What dropped parents mean for children

Even when a span is not sampled, the SDK still creates a lightweight placeholder that carries the span context. This lets child spans and outgoing calls inherit the same trace ID and “not sampled” flag, so downstream services also stay quiet. Dropped spans cost almost nothing and still keep the family consistent.

A trace header is just text that anyone can write. Services that accept parent information from strangers need to think about what could go wrong.

Practical protections

• At the public edge, do not trust incoming trace context. Start a new root span and record the incoming context as a link instead of a parent, or accept it only from known partners.
• Configure which outgoing destinations receive propagation headers, so internal IDs and baggage do not reach unknown third parties.
• Cap how much the incoming sampled flag can raise your volume, using rate limits in the Collector.
• Scrub sensitive attributes centrally, before data is stored.

Parent-child spans are cheap one at a time, but trees can grow big. A few habits keep them healthy.

The cost of a bigger tree

Every child span costs a little CPU to create, a little memory to hold, and some bandwidth and storage to ship and keep. A trace with 20 spans is cheap. A trace with 20,000 spans, produced by a loop that creates a span per item, is painful to produce, send, store, and read. The SDK also enforces limits on how many events, links, and attributes one span may hold, so a single span cannot grow without bound.

Signs that your tree needs attention

Measuring the health of relationships

Treat the quality of your traces as something to monitor. Useful signals include the share of traces that have exactly one root, the number of orphan spans per hour, the average and maximum number of spans per trace, and the share of cross-service calls whose server span has a remote parent. If these numbers worsen after a release, a change probably broke propagation.

The parent-child model is the foundation of distributed tracing, but it is a model, and every model has limits. Here is an honest balance.

Decisions to make on purpose

Teams who read traces every day keep arriving at the same advice. Here are the habits that work, and the traps to avoid.

Patterns worth copying

Anti-patterns to avoid

If you remember only this chapter, you will still avoid most of the pain beginners meet when working with span relationships.

Common mistakes and their fixes

First, a made-up story that shows the idea at work. Then a look at how real tools and platforms use the parent-child model.

Quick answers to the questions people ask most often about span relationships.

You started with a slow pizza and a vague complaint. You now have the tool that turns vague complaints into precise answers.