If you only have two minutes, this chapter gives you the whole idea. The remaining chapters explain why each part exists and how to use it well.

A span link is a reference from one span to another span, held inside the first span. The reference consists of the identifiers that name the target span, which are its trace ID and span ID, plus a few optional attributes that describe the relationship. A span can hold zero, one, or many links.

The key difference from a parent is that a link does not form the tree. A parent says “I am part of this span’s work, nested inside it”. A link says “I am related to that span, but not nested in it”. The linked span may even live in a different trace, which a parent never can.

Before learning the details, it helps to feel the problem. A tree works beautifully until one situation appears, and then it breaks down.

A normal trace is a tree. A request comes in and creates a root span. The root calls other steps, which create child spans, which may call further steps. Each span has exactly one parent, except the root, which has none. This shape is natural for calls that wait for answers, and it makes it easy to see where time went.

The situation that breaks the tree

Imagine a service that reads messages from a queue, collects a hundred of them, and processes them together in one go. Each of the hundred messages was sent by a different request, each with its own trace. The processing job has not one cause but a hundred. Which of the hundred traces should the job’s span become part of? Choosing one would be unfair to the other ninety-nine. Choosing none loses the connection altogether.

Other situations create the same puzzle. A request arrives from an outside company with a trace identifier that you cannot trust. You want your own fresh trace, yet you want to remember the caller’s trace. A task fails and is retried an hour later, and you want the new attempt to point back at the old one. A scheduled job runs because of an earlier event, and you want to see which one. A cache computes one result for many waiting requests, and you want the computation to show who it served.

The elegant answer

OpenTelemetry does not bend the tree to fit these cases. It keeps the tree simple, with one parent per span and one trace per tree, and adds a separate, lightweight mechanism for everything else: the link. The link carries the identifiers of any number of other spans, in any traces, and leaves the tree alone.

A short list of words appears in every conversation about span links. Learn these once and everything else reads easily.

Two ideas hiding in the table

First, a link points to a span context, not to a whole span. The span context is just the identifying bundle. This matters because a link can be created from nothing more than the headers that arrived with a message, without having the other span in memory.

Second, a link is held by the linking span, and it points to the linked span. The linked span does not know it is being pointed to. Nothing is added to it. This one-way nature has consequences for how tools find links, which later chapters explain.

A link is small and simple. Looking at its parts makes it clear what it can and cannot do.

flowchart TB
    SPAN["Linking span"] --> LINKS["Links list: zero or more links"]
    LINKS --> LINK["One link"]
    LINK --> CTX["Span context of the linked span"]
    LINK --> LATTR["Link attributes: optional key-value notes"]
    CTX --> TID["Trace ID"]
    CTX --> SID["Span ID"]
    CTX --> FLG["Trace flags such as sampled"]
    CTX --> TST["Trace state: vendor-specific values"]

Reading the picture

The linking span holds a list of links, which may be empty. Each link has two parts. The first part is the span context of the span being pointed to. It contains the trace ID, which names the whole trace, and the span ID, which names the exact span in it. Together these two numbers form an address, like a street and a house number. The context also carries trace flags, such as whether that trace was marked for recording, and a trace state, which is a small bag of values that vendors may use.

The second part is a set of link attributes. These are optional key-value notes that describe the link itself, not the spans. They can say why the link exists, for example that the linked span is a previous attempt, or that it is one member of a batch.

What a link does not contain

A link holds no timing information about the relationship, no copy of the linked span’s data, and no promise that the linked span even exists in your backend. It is an address, and the address might lead to a span that was dropped by sampling, or that has not arrived yet. This is a feature, since it keeps links tiny, and also a limit to remember.

The most important idea in this guide is the difference between these two relationships. Almost every design decision comes down to choosing between them.

flowchart LR
    subgraph T1["Trace A"]
      A1["Span A1"] --> A2["Span A2 child of A1"]
    end
    subgraph T2["Trace B"]
      B1["Span B1 root"] --> B2["Span B2 child of B1"]
    end
    B1 -. "link" .-> A2

What the picture shows

Trace A and Trace B are two separate trees. Inside each tree, solid arrows show parent-child relationships. The dotted arrow from span B1 to span A2 is a link. It tells you that B1 is related to A2, perhaps because A2 caused B1 to happen later, but B1 is not nested inside A2. B1 is the root of its own trace. If you opened Trace B alone, you would see B1 at the top with no parent, and you would find the link in B1’s details, pointing at the other trace.

Why you cannot just use a parent

A parent relationship carries a promise about time and structure: the child ran as part of the parent’s work and typically finished within it. Duration charts and critical path analysis in trace viewers rely on this promise. If you used a parent for a relationship that is not nesting, the viewer would draw a bar for B1 inside A2’s time range, when B1 actually ran an hour later. The picture would be false. A link makes no such promise, so it makes no such error.

The two parts of a link each have their own job. Understanding both helps you design useful links.

The span context part

The span context identifies the target. The trace ID is a 128-bit number, usually written as thirty-two hexadecimal characters. The span ID is a 64-bit number, written as sixteen. Together, they let a tool find the exact span in a backend, if it was stored. The trace flags say whether the linked trace was sampled, which hints whether it is likely to exist in storage. The trace state is an optional list of vendor values and is rarely touched by ordinary users.

The attributes part

Link attributes are key-value pairs, just like span attributes, but they describe the connection. The OpenTelemetry project does not define a long list of standard link attributes. Teams usually create a small set of their own, such as an attribute that names the kind of relationship. Examples of useful notes include “retry of earlier attempt”, “member of batch”, “triggered by schedule”, and “caller from external system”. The text should be short and stable, with a small set of values, so that tools can group and filter by it.

Keep link attributes simple

Link attributes count against limits and add size to every span that carries a link. If a batch has hundreds of links, a long note on each one multiplies quickly. Choose short, stable values, and avoid putting large or sensitive data there.

A link goes one way. This simple fact shapes how you create links, how you read them, and how you search for them.

The span that holds the link is the source, and the span it points to is the target. The target span is created without knowing that anyone will link to it. When a message producer sends a message, it does not know how many consumers will later link back to its span. The link is created later, by the span that wants the reference, using the identifying information that traveled to it.

sequenceDiagram
    participant P as Producer service
    participant Q as Message queue
    participant C as Consumer service
    P->>P: Create send span, span context S1
    P->>Q: Publish message with span context in headers
    Q->>C: Deliver message after some time
    C->>C: Read span context from headers
    C->>C: Create processing span with a link to S1
    Note over C: The producer span S1 is never changed

Reading the sequence

The producer creates a span for sending a message, and that span has a span context. The producer puts the context into the message headers, using the same propagation method used for ordinary calls. The queue holds the message for however long it takes, perhaps seconds or hours. When the consumer receives it, the consumer reads the context from the headers and creates its own span for processing the message. Into that new span, it adds a link carrying the producer’s span context. The producer’s span is already finished and untouched.

Consequences of the one-way design

• Finding forward links is easy. Open the consumer’s span, and its links are right there.
• Finding backward links is harder. To learn which spans link to the producer’s span, a tool must search all spans for links that mention it. Some backends index links for this, and others do not.
• Cheap for the target. Millions of spans can link to one popular span without changing it or slowing it.

You can attach links when you start a span, and in newer versions of the API also afterward. The first way is better, and the reason is worth knowing.

Links at span creation

When code starts a span, it can pass a list of links as part of the starting options, together with the name, the kind, the parent, and initial attributes. The span is born already holding its links. This is the recommended way, because everything the span needs for its first decisions is in place from the beginning.

Why the timing matters

The sampler, the component that decides whether to record a trace, runs at the moment a span starts. The sampler is given the links as part of its inputs. A custom sampler might use them, for example to keep any trace that links to a sampled trace. If you add links after the span has started, the sampler has already decided, and it never saw them. Adding links at creation lets the sampler and any early processing use them.

Links added later

Recent versions of the OpenTelemetry API have introduced a way to add a link to a span after it starts. This is useful when you learn about a related span only partway through the work, for example when a batch grows as messages arrive. Support depends on your language and version, and the sampler will not have seen these late links, so use them only when creating them at the start is impossible.

The classic reason for links is a step that gathers work from many sources. Here is the idea in full.

Suppose a service called “Receipts” reads from a queue where each message represents one order. To save effort, it reads fifty messages at a time and generates fifty receipts in a single batch step. Each message was published by a different request with a different trace.

flowchart LR
    T1["Trace 1: order span"] --> M1["Message 1"]
    T2["Trace 2: order span"] --> M2["Message 2"]
    T3["Trace 3: order span"] --> M3["Message 3"]
    M1 --> Q["Queue"]
    M2 --> Q
    M3 --> Q
    Q --> BS["Batch processing span in a new trace"]
    BS -. "link" .-> T1
    BS -. "link" .-> T2
    BS -. "link" .-> T3

What you would do without links

Without links, you have only bad choices. You could make the batch span the child of just one of the fifty messages, which hides the other forty-nine. You could make fifty copies of the batch span, one per trace, which multiplies cost and lies about how many batches ran. Or you could start a fresh trace with no connection, which loses the information about where the work came from.

What links give you

With links, the batch gets its own span, in its own trace, with fifty links, one per message. An engineer investigating a slow receipt for one order can go from that order’s trace to the batch that handled it, by looking for the batch span that links to it. An engineer looking at the batch can see all fifty orders it served, and can jump to any of them. The tree stays honest, and the connections are preserved.

The same pattern in other shapes

The pattern is called fan-in, meaning many sources join into one step. Other examples include a nightly report that aggregates many events, a database write that combines many small updates into one, and a stream processor that handles a window of events together.

Message queues are the most common home of links. Yet even here there are two ways to connect the sending and receiving sides, and the choice depends on the shape of the work.

When one service sends a message and another processes it later, the two sides are separated by time. The queue may hold the message for milliseconds or for days. The question is whether the processing span should be a child of the sending span, putting both in one trace, or a new trace that links back to it.

Option one: one message, one trace, parent-child

For a simple flow where each message is handled by exactly one consumer step, many teams and instrumentation libraries make the processing span a child of the producer’s span. The whole journey, from request to send to processing, appears as one trace. This is easy to read. The cost is that a trace can become very long in time, and the waiting time in the queue appears as a gap between parent and child.

Option two: a new trace with a link

When processing is batched, or when the work should be seen as its own unit, the consumer starts its own trace and adds a link to the producer. This keeps traces short and bounded, and it supports many-to-one relationships. The messaging semantic conventions describe both approaches and recommend links in cases where one span relates to many messages, such as when a consumer receives a batch. Details of the conventions have been evolving, so check the current version for the exact rules.

Sometimes you do not want to continue a trace that arrived from outside. A link lets you start fresh while remembering where the request came from.

The problem with trusting incoming context

When your service receives a request from another system, the request may carry a trace context in its headers. By default, your service continues that trace: its first span becomes a child of the caller’s span, and both share the same trace ID and the same sampling decision. This is perfect inside your own company, where all services cooperate. It is risky at the border with outsiders.

An outside caller could send a trace ID that collides with an existing one, could mark every request as sampled to inflate your volume, or could simply use a different convention for IDs. If you continue their trace, their choices spill into your system. Your trace storage, your sampling, and even your dashboards are influenced by someone you do not control.

The pattern: start new, link to the old

At the public edge, many teams start a new trace for each incoming request, ignoring the caller’s trace as a parent. To avoid losing the connection, they add a link from the new root span to the caller’s span context. Engineers can still follow the connection in either direction, but your trace follows your own rules for IDs and sampling. This pattern is mentioned in the OpenTelemetry guidance for instrumentation at service boundaries, and it is the reason the API allows creating a root span with links.

sequenceDiagram
    participant X as External caller
    participant G as Your gateway
    participant S as Internal services
    X->>G: Request with foreign trace context
    G->>G: Start a new root span in a new trace
    G->>G: Add a link to the foreign span context
    G->>S: Continue with the new trace context
    Note over G,S: Internal sampling and IDs follow your rules

Reading the sequence

The external caller sends a request that carries its own trace context. The gateway creates a brand new root span with a new trace ID that the gateway chose. It adds a link that holds the caller’s span context, so the relationship is recorded. Internal services then continue the new trace, which means they follow your sampling decision and your ID format. The foreign context is remembered as a link, and not trusted as a parent.

What you gain

• Control of sampling. Outsiders cannot force you to record everything by setting a flag.
• Clean identifiers. Your trace IDs come from your own generator, which protects ratio-based sampling from non-random IDs.
• A preserved connection. If the partner shares their trace ID in a support case, you can search for links that mention it.

When work fails and is tried again later, you often want each attempt to tell its own story while pointing at the earlier ones.

A quick retry inside one call, such as three tries within one second, is usually shown as sibling spans under one parent. The parent covers the whole operation, and each attempt is a child. This is natural and needs no links.

The picture changes when the retry happens much later. A message that fails processing is put back on the queue and delivered again an hour later. A payment that failed is retried by a scheduled job the next morning. A webhook that was rejected is sent again tomorrow. In these cases, the second attempt happens in a different time and often a different trace. Making it a child of the first would stretch the trace over hours or days. Starting a completely fresh trace would lose the connection.

flowchart LR
    A1["Attempt 1 span in trace X, status Error"] --> Q["Retry queue, waits one hour"]
    Q --> A2["Attempt 2 span in trace Y, status Error"]
    A2 -. "link: previous attempt" .-> A1
    A2 --> Q2["Retry queue, waits one day"]
    Q2 --> A3["Attempt 3 span in trace Z, status Unset"]
    A3 -. "link: previous attempt" .-> A2

What the chain shows

Each attempt has its own trace, with its own timing and its own status. The first two attempts ended in Error, and the third succeeded. Each later attempt carries a link to the one before, with an attribute noting that the reason for the link is “previous attempt”. An engineer who finds the failed third attempt, or the successful one, can follow the chain backward to see how many tries it took and why the earlier ones failed.

Why this beats the alternatives

Nesting all attempts under one parent would create a trace that lasts a day and a half and holds spans from unrelated times. Tools would show a huge gap, and the trace would be hard to read and hard to store. Separate unlinked traces would make the chain invisible. Links keep each trace small, and still let you walk the chain.

Work that starts because of something earlier, but not as a direct call, is a natural fit for a link.

Scheduled jobs

A cron job runs every night with no incoming request. It starts its own trace, because there is no parent. But if the schedule was created by an earlier request, for example a user who asked for a monthly report, it can be helpful to link the nightly job’s span to the span of the request that created the schedule. This explains why the job exists.

Event-driven reactions

In event-driven systems, one service publishes an event, and several others react at different times. A reaction can be a child of the event publication, a link to it, or both, depending on design. Links work well when the reaction happens long after the event, or when the reaction considers many events together.

Webhooks and callbacks

A service asks an outside provider to do something, such as process a payment, and the provider calls back later with the result. The callback arrives as a new request, with no trace context, or with a context the provider chose. A link from the callback span to the original request span, found by looking up a stored span context for that transaction, ties the two together.

Some business processes last for hours or days. Keeping them all in one trace creates a monster. Links let you cut the monster into readable pieces.

Consider an online order. The customer places it, it waits for payment confirmation, then for stock allocation, then for shipping, and finally for delivery. The whole life of the order may take a week. If every step were a child in one trace, the trace would be open for a week, the root span would last a week, and the trace would contain thousands of spans over several services. Many backends hold traces in memory for a limited time while waiting for late spans, and such a trace would be cut, incomplete, or very costly.

Cutting by phase

A better design starts a new trace for each phase, and links each phase to the previous one. Phase one, “place order”, is a normal short trace. When payment is confirmed, a new trace begins for “fulfill order”, with a link to the order placement span. When shipping starts, another trace begins with a link to the fulfillment span. Each trace is small, bounded, and easy to read. The chain of links preserves the story.

The cost of the design

Cutting a story into pieces means that nobody sees the entire week in one picture. To get the whole story, a tool or a person must follow the chain of links. Some backends can do this, others show only the links as references. Decide whether the loss of the single picture is worth the gain in readable, bounded traces. Most teams find that it is, for processes lasting longer than a few minutes.

When one piece of work serves many requests at once, the requests all wait for it, and none of them owns it. Links tell the truth about this sharing.

Imagine a cache that protects a slow database. When a popular item expires, a hundred requests arrive within a second, all wanting it. A smart cache lets only one of them fetch the value from the database, and makes the other ninety-nine wait for that single result. This pattern is called request coalescing or sometimes single flight.

The tracing question

Which request’s trace should contain the database call? Giving it to the first request makes that request look slow for reasons that also affect the others. The other ninety-nine requests show a mysterious wait with no cause. Duplicating the database span into all hundred traces would multiply the apparent database load by a hundred.

The link solution

The single database fetch gets its own span. In the trace of the request that triggered the fetch, it can be a child. In each of the waiting requests, a short “waiting for shared fetch” span carries a link to the fetch span. An engineer looking at a slow request sees “waiting for shared fetch”, clicks the link, and finds the database call and its real duration. One true span exists, and many requests point to it.

With three ways to relate things, a short decision path keeps your choices consistent.

flowchart TB
    START["Two pieces of work are related"] --> Q1{"Did the second run inside the first and wait for it?"}
    Q1 -- "yes" --> PARENT["Use parent-child in the same trace"]
    Q1 -- "no" --> Q2{"Does the second span relate to more than one span?"}
    Q2 -- "yes" --> LINK["Use links, one per related span"]
    Q2 -- "no" --> Q3{"Is the gap long, or the trust low, or the trace getting huge?"}
    Q3 -- "yes" --> LINK
    Q3 -- "no" --> Q4{"Is a business ID enough to find the relation?"}
    Q4 -- "yes" --> ATTR["Use a shared attribute such as order ID"]
    Q4 -- "no" --> LINK

Explaining the path

First ask whether the second piece of work ran inside the first and waited for it, which is the definition of a call. If so, use a parent. If not, ask whether the second relates to more than one span. A span can have only one parent, so many relations require links. If it relates to one span only, consider the practical issues: a long gap between the two, an untrusted source, or a trace growing unmanageable. Those favor links. If none of these applies, a simple shared attribute, such as an order ID on both spans, may already be enough and costs less.

Why attributes sometimes win

A shared business identifier is the simplest connection. Every span for order 8472 carries the attribute order.id = 8472. A search finds them all, across traces, with no special link support in any tool. Links are better when you need the exact span-to-span relationship, such as which batch handled which message, or when the relationship has no natural business identifier.

A link needs the span context of the target. That context has to travel from where it was created to where the link is made.

Inside one program, getting a span context is easy: you hold the span object, and you can ask it for its context. Across services, the context must be carried with the data. In messaging, this happens by placing the context into the message headers or properties, using the same propagator that carries trace context in ordinary HTTP calls. In the standard format, the context travels as a traceparent header that holds a version, the trace ID, the span ID, and the flags.

flowchart LR
    SPAN["Producer span"] --> INJ["Propagator injects context into message headers"]
    INJ --> MSG["Message with traceparent header"]
    MSG --> Q["Queue or store"]
    Q --> EXT["Consumer extracts context from headers"]
    EXT --> LNK["New span is created with a link to the extracted context"]

Inject and extract

The two verbs to remember are inject and extract. The producer injects the current span context into the carrier, which is the message headers. The consumer extracts the context from the carrier and gets back a context object. From it, the consumer reads the span context and uses it for either a parent or a link. Most messaging instrumentation libraries do both steps for you, and also let you choose between parent and link for the consumer side.

When there is no carrier

Not every system has headers. A scheduled job that needs a link to the request that created it must store the span context itself, for example as two text fields in a database table: the trace ID and the span ID, perhaps with the flags. Later, the job reads these fields and builds a span context for the link. Some SDKs allow constructing a span context directly from stored identifiers for this purpose.

What if the context is missing or invalid?

If a message arrives without a context, because the producer was not instrumented or the broker stripped headers, there is nothing to link to. A well-behaved consumer starts a new trace without a link. A link built from an invalid context, with all zeros for the IDs, is meaningless and is typically ignored by the SDK or the backend. Handle this case calmly, and perhaps count how often it happens.

A link by itself says “related”. Link attributes say how, and that single word makes the difference between a mystery and a helpful clue.

Why you need them

A span that carries many links, such as a batch, or a span that carries links of different kinds, such as one retry link and several batch links, needs a way to tell them apart. Without a note, a reader sees a list of identifiers and has to guess what each means. A short attribute removes the guess.

A small, consistent vocabulary

There is no big official list of link attribute names, so teams invent their own. Keep the vocabulary tiny and consistent across services. A single attribute, with a handful of possible values, is usually enough.

Good habits for link attributes

• Keep values few and fixed. A small fixed set allows grouping and counting.
• Keep them short. Each link carries them, and a batch may have hundreds of links.
• Keep them private. No personal data or secrets in link attributes.
• Document them. Put the vocabulary in the team’s instrumentation guide.

Links are small, but a span with thousands of them would be a problem. The SDK enforces limits, and ignoring them leads to silent loss.

The OpenTelemetry specification defines span limits, which cap how much a span may carry. Among them are a limit on the number of links per span, and a limit on the number of attributes per link. The common default for the link count is 128, and for attributes per link it is also 128, though SDKs may differ, so check yours. When a span tries to hold more links than the limit, the extra ones are dropped, and the span keeps a count of how many were dropped, which is exported with the span as a dropped links count.

A worked example with made-up numbers

A batch consumer reads 500 messages at a time and creates one span with a link for each message. With a limit of 128, only the first 128 links are kept, and 372 are dropped. The span shows a dropped links count of 372. An engineer investigating one of the dropped messages will never find the batch through its link, and may conclude that the batch never happened.

flowchart TB
    BATCH["Batch of 500 messages"] --> Q{"Is the batch larger than the link limit?"}
    Q -- "no" --> OK["Add one link per message"]
    Q -- "yes" --> CH{"Choose a strategy"}
    CH --> S1["Raise the link limit carefully"]
    CH --> S2["Split into several smaller batch spans"]
    CH --> S3["Link only a sample or the most important messages"]
    CH --> S4["Use a shared batch ID attribute on message and batch spans"]

Choosing a strategy

Raising the limit works but costs memory and size for every span, and some backends cap the size of a span or message. Splitting the batch into smaller processing spans, each within the limit, keeps every link and keeps spans small, at the price of more spans. Linking only a subset is acceptable when you need only examples, but it means that some messages cannot be traced forward. A shared batch identifier as an attribute on both the message spans and the batch span is a cheap fallback that works with any tool.

Where to set the limits

SDKs allow setting span limits in code or through environment variables, such as OTEL_SPAN_LINK_COUNT_LIMIT for the link count, and another for attributes per link. Names and defaults can differ by language and version, so confirm in the documentation.

Sampling decides which traces are kept. Since links cross traces, sampling can leave a link pointing at nothing.

Each trace is sampled or dropped as a whole. A link from a span in trace B to a span in trace A is just an address. If trace A was dropped by sampling, the address leads nowhere. The trace flags inside the link tell you whether trace A was marked sampled, which is a hint about whether it exists in storage.

Four cases

The sampler sees links at start

As discussed earlier, a sampler receives the links of a span when it starts. A custom sampler can use this to keep what matters. For example, a rule might say “if any linked span was sampled, sample this span too”. This raises the chance that both sides of a link are kept, and avoids dead links. Built-in samplers such as the ratio-based one ignore links, so by default each trace is sampled independently.

Tail sampling and links

Tail sampling, which decides after a trace is complete, looks at one trace at a time. It does not follow links to keep the linked traces. If you want to keep a failing batch and the traces of all its messages, you need custom logic, because each message’s trace was a different trace and might already be gone.

flowchart LR
    TB["Trace B: batch span with link"] -. "link" .-> TA["Trace A: producer span"]
    TB --> SB{"Was B sampled?"}
    TA --> SA{"Was A sampled?"}
    SB -- "yes" --> LIVE["Link is visible in B"]
    SA -- "yes" --> TARGET["Target exists and can be opened"]
    SA -- "no" --> DEAD["Dead link: target not stored"]

Practical advice

Accept that some links will be dead, and design the experience around it. Show the sampled flag from the link in your tooling so that people do not click on links that cannot work. Where the relationship matters a lot, such as for failed batches, use a custom sampler or a higher sampling rate on the paths that use links.

A link is only as useful as the tool that displays it. Support varies, so it is wise to know what to expect.

What most viewers do

Most modern trace viewers show the links of a span in its detail panel. Typically, you see a list of the linked spans’ trace IDs and span IDs, and a button or a hyperlink that opens the linked trace if it exists in the backend. Link attributes appear beside each entry. A few viewers draw a small marker on spans that hold links, so you notice them without opening every span.

What many viewers do not do

• They do not draw links as part of the tree. A link does not change the nesting, so the waterfall remains a normal tree.
• They may not show reverse links. Showing “spans that link to this one” needs an index over links, which not every backend maintains.
• They may not follow chains automatically. You may need to click through each hop to follow a chain of retries.
• They may show dead links without a warning. A click may lead to a “trace not found” page.

Test before you rely

Before designing your tracing around links, create a few example spans with links and look at how your own viewer presents them. If it is clumsy, compensate with shared attributes and documentation that tells people how to find related traces. Do not assume that the experience you imagine is the experience your team will get.

Links are only worth creating if people use them during real investigations. Here is how a typical investigation moves across them.

The two directions of reading

Going forward, from a batch span to the messages it handled, is easy because the links are inside the batch span. Going backward, from a message to the batch that handled it, requires a reverse search. In practice, investigations often need the backward direction, since the symptom starts at the individual request. This is why a shared attribute, such as a batch ID that appears on both the message span and the batch span, is such a valuable companion to a link.

A span can carry several kinds of extra information. Knowing which to use for which purpose keeps spans clean.

Mixing them well

A batch processing span could carry many links, one per message. It could carry an attribute with the batch size, an event noting that one message was skipped as invalid, and a status of Error if the batch failed. Each feature does its own job. A common mistake is to put a list of trace IDs into an attribute as text. That hides the relationship from tools that know how to read links, and it breaks attribute size limits quickly.

Use the right tool for the right data

If you need the reader to jump to another span, use a link. If you need to describe the span, use an attribute. If you need to mark a moment, use an event. Following this simple rule keeps spans understandable and tool-friendly.

Links are not only for messages between services. Inside one program, asynchronous code raises the same question: parent or link?

Modern programs run work on thread pools, event loops, and futures. A request handler might start background tasks and carry on without waiting for them. A parent span ends when the handler returns, but the background task continues afterward. If the task’s span is a child, it outlives its parent, which many viewers draw in a confusing way, with a child bar extending beyond its parent’s end.

Fire-and-forget tasks

For a task that the handler starts and does not wait for, there are two reasonable choices. One is to make the task’s span a child of the handler span anyway, accepting that it ends later. This is simple and keeps the work in one trace, and it is the usual default of context propagation in many libraries. The other is to start a new trace for the task, with a link to the handler span. This gives clean bounded traces, and it is a good choice when the task is long, or when many requests feed one shared background worker.

Fan-out and join

If the handler starts ten parallel tasks and waits for all of them, the tasks are true children. The handler’s span covers them in time, and a parent relationship is right. Links are not needed. The difference is whether the parent waits. Waiting means nesting, and no waiting means you have a choice.

Links are lightweight, but they are not free. Their cost grows with count and with attributes, and a careless design can surprise you.

Where the cost lives

A worked example with made-up numbers

A consumer processes 20 batches per second, each with 100 messages. Each link takes about 60 bytes with a short attribute. A batch span therefore carries about 6,000 bytes of links, and across 20 batches that is about 120,000 bytes per second, or roughly 10 gigabytes per day. That is a noticeable but manageable amount. If the same system raised its batches to 1,000 messages and kept every link, the volume would multiply tenfold, and the team would need to reconsider.

Keeping it cheap

• Keep link attributes short or omit them for large batches, and use a single attribute on the span instead.
• Link only what you need. Examples may suffice for very large batches.
• Prefer splitting batches into several spans over raising the link limit far above its default.
• Use sampling, so that only a fraction of these spans is stored.

A link is only an address, so it carries little risk by itself. The risks are in what comes along with it and in whom you choose to trust.

Links reveal relationships

A link says that two pieces of work are related. In a system that spans organizations, that can reveal who talks to whom, how often, and when. A link to an external partner’s span context shows that the partner called you. If trace data is shared with a vendor or exposed widely, these relationships are visible too. Apply the same access controls to link data as to the rest of your traces.

Untrusted contexts

As described in the trust boundary chapter, a context from an outside source should be treated as untrusted. Using it as a parent lets the outsider influence your sampling and your trace identity. Using it only as a link keeps the influence minimal, since the link does not affect which trace you belong to or whether it is sampled. Still, the content of an untrusted context is untrusted data. It might contain a trace state with strange values, so do not interpret the trace state for decisions.

Link attributes are a leak path

Link attributes can carry text. If developers put customer names, email addresses, or tokens there, the data reaches every tool that stores traces. Keep link attributes to a short fixed vocabulary, as recommended earlier. Review a sample of real links periodically.

Link floods

Consider a service that builds links from incoming contexts, for example one link per received message. An attacker who can send many messages may inflate the link counts and make spans large. The span limits protect you by capping the stored links, which is one more reason to keep the limits in place and not set them to unlimited.

Links break quietly. A few simple measurements tell you whether the connections you designed are really there.

How to measure

Many backends let you query spans by whether they have links, and by the dropped links count. A weekly report that counts consumer spans with zero links, spans with dropped links, and links to unknown traces gives a good picture. If your backend cannot do this, a Collector component that turns span data into metrics can be configured to count spans with and without links, though the details depend on the components you run.

What to do with the numbers

A sudden drop in link coverage after a deployment is a strong clue that a library upgrade or a configuration change broke propagation. A rising dropped links count after a traffic change means that batches grew past the limit. A high dead link rate beyond what sampling explains means that the targets are not being stored for another reason, such as a pipeline problem.

Most link problems are silent. The system keeps running and the traces keep arriving, but the connections you expected are missing or misleading.

The wrong-target failure deserves attention

A link that points to the wrong span is worse than no link, because it appears trustworthy. A common cause is a cached or stored context that is reused for many different jobs. For example, a service keeps the span context of “the latest request” in a shared variable and uses it later for a background job. Under concurrency, the variable may hold another request’s context at the moment the job runs. Always pass the context along with the specific work item, never through shared mutable state.

Teams that get value from links tend to repeat the same few habits.

The value of the written policy

Links are easy to misuse because the choice between parent and link is a matter of judgment. A short written policy gives people a shared answer for the common cases. It should list each type of asynchronous flow in your system, say which relationship to use, and give the standard attribute values. A new developer then follows the policy instead of guessing.

These habits look harmless and slowly make traces less useful.

This is the checklist to keep next to your instrumentation guide.

Best practices

• Use a parent when one span contains and waits for another, and a link when work is related but not nested.
• Use links for batches, for work with many causes, and for work that spans long gaps in time.
• Create links when starting the span, so samplers and early processors see them.
• Carry the span context with each work item, in headers or in stored records.
• Add one short relationship attribute to each link, drawn from a small fixed vocabulary.
• Respect the link limit, and plan for batches larger than it.
• Add a shared business identifier beside links so that people can search even without link tools.
• Start fresh traces at trust boundaries and link to the outside context.
• Watch dropped links, link coverage, and dead link rates.

Common mistakes and their cures

These stories are invented to illustrate common situations. They are not reports about real companies.

When links do not behave, follow the path of the context from the producer to the viewer. The first break you find is the cause.

flowchart TB
    START["Expected link is missing or wrong"] --> A{"Does the producer inject context into the carrier?"}
    A -- "no" --> A1["Enable propagation or instrumentation on the producer"]
    A -- "yes" --> B{"Does the context survive the broker or proxy?"}
    B -- "no" --> B1["Check header names, filters, and bridges"]
    B -- "yes" --> C{"Does the consumer extract a valid context?"}
    C -- "no" --> C1["Fix extraction or the propagator format"]
    C -- "yes" --> D{"Is the link added when the span starts?"}
    D -- "no" --> D1["Create the link at span start or before the span ends"]
    D -- "yes" --> E{"Is dropped links count above zero?"}
    E -- "yes" --> E1["Raise the limit or split the batch"]
    E -- "no" --> F{"Does the viewer show links?"}
    F -- "no" --> F1["Check exporter, backend support, and the viewer"]
    F -- "yes" --> F2["Check whether the target trace was sampled"]

Explaining the checklist

Begin at the producer, because without an injected context nothing else can work. Check that the sending side’s instrumentation is on, and that the propagator format matches what the consumer expects. Next, check the path between the two. Brokers, proxies, and bridges between messaging systems sometimes drop or rename headers, which silently removes the context. Then check the consumer: does it extract the context, and is the result valid, with non-zero identifiers?

If the context arrives, check how the link is created. It must be added when the span starts or before it ends. After that, check for dropped links, which shows when the batch is bigger than the limit. Finally, check the display: whether your exporter sends links, whether the backend stores them, and whether the viewer shows them. If all of that is fine and the link still leads nowhere, the target trace may have been dropped by sampling.

The idea of a link is the same everywhere. The way you reach it in code looks a little different in each language.

The shared shape

Every OpenTelemetry API lets you pass a list of links when you start a span. Each link is built from a span context, which you obtain either from a span object you hold, or from a context you extracted from a carrier, and an optional set of attributes. The API also provides helper functions to build a span context from stored identifiers, which is how links work for scheduled and retry cases. Newer API versions add a method to add a link to a span after it has started.

Where languages differ

How to check your language

Find the tracing page in your language’s documentation, and look for the section about span creation and links. Then write a small test that creates a producer span, extracts its context, starts a consumer span with a link, and prints the result using a console exporter. In a few minutes you will know how links behave in your SDK and how your viewer shows them.

A span link is a lightweight pointer from one span to another that records a relationship without nesting. It holds the target’s trace ID and span ID, plus optional attributes, and it can cross trace boundaries. Links exist because the tree of parent and child cannot express many-to-one relationships, long delays, untrusted callers, or shared work. They are used for batch processing, messaging, retries, scheduled work, trust boundaries, and very long processes. To use them well, create them when the span starts, carry the context with the work item, add a short relationship attribute, respect the limits, and back them up with shared business identifiers, since viewers vary in how well they support links.