Start with the simple idea. The rest of this guide is just that idea meeting difficult situations.

When a user clicks “buy,” many services may work together to complete the purchase. Engineers want to see the whole story of that one click as a single connected picture. That picture is called a trace. A trace is made of spans, and each span is one timed step of work, such as “check inventory took 40 milliseconds.” For the picture to hang together, every span must know which trace it belongs to and which span came just before it.

That knowledge is held in a small bundle of facts called the context. In OpenTelemetry, the important parts of the context are:

The act of moving this bundle from one place to another is propagation. It has two halves. Injection is writing the context into something that travels, such as a message header. Extraction is reading it back out on the other side. The small program piece that knows how to do both, using an agreed format, is called a propagator. The thing that carries the context, such as a web request header or a message property, is called the carrier.

The standard format

OpenTelemetry uses a standard from the World Wide Web Consortium called W3C Trace Context by default. It describes a header named traceparent that holds four parts separated by dashes: a version, the trace ID, the parent span ID, and the flags. A second header, tracestate, holds optional vendor notes. A separate standard header, baggage, holds the business key-value pairs. Because these formats are agreed on by everyone, services written in different languages by different teams can still pass the baton to each other.

Direct calls are like phone conversations. Events are like letters. The rules of tracking change completely.

In a classic request-response system, service A calls service B and waits for an answer. The call is live. The connection exists while the work happens. The context rides along in the call’s headers and lives only as long as the call does. The shape of the trace naturally matches the shape of the calls: A starts a span, B’s work is nested inside it, and when B answers, both spans end close together.

In an event-driven system, service A publishes a message to a broker, which is software that stores and delivers messages, such as a message queue or an event streaming platform. A does not know who will read the message, or when. It simply announces “an order was placed” and moves on. Later, zero, one, or many other services, called consumers, read that message and react. The services are decoupled, meaning they are independent of each other in time and in knowledge. That independence is the whole point of the design, and it is also exactly what makes tracing hard.

The main differences

Every challenge in this guide comes from one or more of these differences. Keep the table in mind as we go. When a problem appears, ask which row it comes from.

A quick map of the challenges

Here is a short list of what we will explore. Each gets its own chapter with causes, effects, and remedies.

• Carriers that cannot hold the context, or hold too little of it
• The long and unpredictable time gap between sending and receiving
• One message with many readers, and many messages read together
• Retries, redeliveries, dead-letter queues, and replays
• Middle steps that drop or replace the context
• Context getting lost inside a consumer’s own threads and callbacks
• Different teams using different propagation formats
• Sampling decisions that must survive the gap
• Request-reply patterns, workflows, and long-running processes
• Security, privacy, and trust of incoming context
• Clock differences, cost, and volume

Why this matters beyond the engineering team

Broken propagation is not only a technical annoyance. When an order is delayed or a payment is charged twice, support teams, product managers, and sometimes customers are waiting for an answer. If the trace is in pieces, the investigation turns into hours of searching logs in several systems and guessing how events relate. Connected traces shorten that search to minutes, because one click shows which message triggered which work, how long each step waited, and where the first failure happened.

There is also a planning benefit. Teams that can see the real shape of their event flows, including which consumers are slow, which topics fan out the most, and which steps retry constantly, make better decisions about capacity, ownership, and design. Poor propagation hides this shape, so improvements are guessed rather than measured. Investing in reliable context propagation therefore pays back twice: faster incident response today and clearer architecture decisions tomorrow.

First see the happy path. Then every later chapter will show where it breaks.

flowchart LR
  P["Producer service"] -->|"1. inject context into message headers"| M["Message with headers"]
  M --> B["Broker: queue or stream"]
  B -->|"2. message waits"| M2["Same message with same headers"]
  M2 -->|"3. extract context from headers"| C["Consumer service"]
  C --> S["Consumer span linked to producer"]
    

Follow the numbers in the figure. In step one, the producer’s instrumentation, which is the code or library that creates spans automatically, writes the current context into the message’s headers right before the message is sent. In step two, the broker stores the message, headers and all, and holds it for as long as needed. In step three, when a consumer receives the message, its instrumentation reads the headers, rebuilds the context, and uses it when creating the consumer’s spans. The result is that the consumer’s span knows which producer span it came from, even though the two services never spoke directly.

Where instrumentation comes from

In many languages, ready-made instrumentation exists for popular messaging clients. Some of it works automatically by attaching to the client library when the program starts, and some requires adding a small wrapper in code. Coverage differs by language and by broker, and it changes over time. A frequent source of broken traces is simply that one side has instrumentation and the other does not. Always confirm, by testing, that both the sending and receiving programs handle context for the specific client library they use.

Why the happy path breaks

Each of the three steps can fail in its own way. Step one can fail if the producer does not inject, or injects into a place the broker drops. Step two can fail if the broker or a middle program rewrites the message. Step three can fail if the consumer does not extract, extracts the wrong thing, or loses the context before using it. The following chapters work through these failures one at a time.

One more habit is worth building early: write a short, plain-language description of how a typical event flows through your system, naming each program that touches a message. This document, even if only one page, becomes the map that guides every later decision about where context might be lost, who owns each hop, and which tests are needed. Teams that keep such a map up to date find propagation problems faster and onboard new engineers with far less confusion.

Before solving problems, you need the vocabulary OpenTelemetry uses to describe messaging.

Span kinds

Every span has a kind, a label saying what role it plays. For messaging, two kinds matter most. A producer span describes the act of creating or sending a message. A consumer span describes the act of receiving or processing one. Other kinds, such as client and server, are for direct calls. Labeling spans this way lets analysis tools draw messaging steps differently from ordinary calls and compute messaging-specific numbers.

Operations

OpenTelemetry’s guidelines for messaging, called semantic conventions, describe a few typical operations. Creating a message, publishing it, receiving it from the broker, processing it, and settling it, which means telling the broker “I handled this, remove it” or “I could not handle this, deliver it again.” Different libraries emit different subsets of these. A beginner should simply know that one message’s life contains several distinct moments, and each may or may not have its own span.

Parent-child versus links

A parent-child relationship says “this step happened because of that step, and is part of it.” A span link is a looser connection that says “this step is related to that other trace, but not nested inside it.” Links can point to spans in the same trace or in an entirely different one.

flowchart TB
  subgraph T1["Trace of the producer"]
    A["Span: place order"] --> B["Span: publish message, kind producer"]
  end
  subgraph T2["Trace of the consumer"]
    C["Span: process message, kind consumer"] --> D["Span: update stock"]
  end
  C -. "link to producer span" .-> B
    

The figure shows one popular style. The consumer starts a fresh trace, which keeps traces short and self-contained, and adds a link back to the producer span so you can navigate between them. The other style makes the consumer’s span a direct child of the producer’s span, so everything appears in one big trace. Both are legitimate. The guidelines lean toward making processing a child of the producer’s context for a single message, and toward links when many messages are involved. Which one is best depends on your system, and we will see the trade-offs in later chapters.

The baton needs a hand to hold it. Not every broker offers one.

For HTTP calls, headers are always available. For messaging, the carrier is the message’s own metadata, and brokers differ widely in what they offer. Some brokers support flexible user-defined headers on each message. Others offer a small, fixed number of named attributes. A few offer nothing at all, and the message is just a block of bytes.

The common workaround: put it in the body

When the broker has no header support, the context can be written inside the message’s content, for example as an extra field in a structured message, next to the business data. This works anywhere, but it has costs. Every producer and consumer must agree on the field name and format. The field mixes technical information into business data, which can annoy teams who own the message schema. Tools that read the content, such as validators and data warehouses, will see an unfamiliar field. And a processing step that rebuilds the message from scratch may forget to copy it.

Wrapper envelopes

A cleaner variant is the envelope pattern: every message is wrapped in a standard outer layer that holds technical fields such as the message type, the creation time, and the trace context, while the inner layer holds business data. Envelope standards for events exist, and many organizations define their own. Once everyone uses the envelope, the carrier question is solved uniformly, no matter which broker is underneath.

Bridges between systems

Many companies chain brokers together: a message in one system is forwarded into another by a connector or bridge. If the bridge copies only the body and not the headers, the context is lost at the border. When you introduce any bridge, test that it preserves or translates context headers, and write down how.

Space on a message is limited and shared. Every extra byte of context competes with real data.

The basic trace context is tiny: the main header is only a few dozen characters. That is rarely a problem. The trouble comes from the optional extras, which can grow without anyone noticing.

Trace state growth

The tracestate header lets several vendors and tools add their own notes. In a long chain of services, each tool may add or modify an entry. The standard sets limits on how many entries and how long they may be, and well-behaved libraries trim entries when the limit is reached. Still, a busy environment with several tracing tools running side by side may fill it, and the notes of one tool can push out the notes of another.

Baggage growth

Baggage is the more dangerous one. It is a set of key-value pairs that travels with the context so that every downstream step can read them. A team might add a customer identifier, then another team adds a region, then another adds an experiment name. The standard recommends supporting at least a certain number of entries and a total size on the order of several kilobytes, but individual brokers may impose smaller limits on a single header or on the combined metadata of a message. When the limit is exceeded, the symptoms vary: the message may be rejected, the header may be silently cut, or the broker may drop all metadata.

Hidden multiplication

Message volume makes bloat expensive. If every message carries one extra kilobyte of baggage and you send fifty thousand messages a second, that is fifty megabytes every second of extra traffic, and it is stored in the broker and copied to every consumer group. Brokers charge for storage and traffic in many environments, so tiny habits become real bills.

Defensive habits

• Treat baggage as a scarce resource. Allow only a short, approved list of keys, owned by a named team.
• Prefer IDs over data. Carry a small identifier and let downstream services look up details, rather than copying large values around.
• Limit entry count and size in libraries. Configure the propagation code to enforce your own smaller limits before the broker’s.
• Monitor header sizes. Track the size of context headers on sampled messages, and alert when it grows.
• Do not copy baggage into every span by default. It is not automatically turned into span attributes, and copying it indiscriminately can leak data and increase cost. Choose specific keys deliberately.

Between “sent” and “received” lies a stretch of time that no direct call ever has to deal with.

In a direct call, the end of the work comes right after the start. In an event-driven flow, the gap between the producer finishing its span and the consumer starting its own can be anything: a few milliseconds when the system is healthy, minutes when consumers are behind, hours when a consumer was down, or days when a message sits in a dead-letter queue waiting for a human. That gap is the asynchronous gap, and it creates several quiet problems.

Problem 1: Traces that stretch forever

If the consumer’s span is a child of the producer’s, the trace now spans the whole gap. A trace that took forty milliseconds of real work might appear to last six hours. Tools that show traces as a timeline become hard to read, with a tiny bar at the start, an enormous empty stretch, and another tiny bar at the end. Summary numbers such as “average trace duration” become meaningless because they mix real work with waiting.

Problem 2: Backends and time windows

Some tracing backends assume that all spans of one trace arrive within a limited time window and may give up waiting, finalize the trace, or display it partially. Very late spans can arrive after the trace has been considered complete and then appear as strange stray fragments, or be missed entirely by searches that look only at a certain time range.

Problem 3: Tail sampling and waiting

As described in the companion guide on scaling the Collector, tail sampling means deciding whether to keep a trace only after seeing most of it. It holds spans in memory for a fixed waiting period before deciding. If your gap is longer than that period, the producer’s spans and the consumer’s spans arrive after the decision has already been made, and the trace is judged in two separate halves. A trace with an error in the consumer half may be thrown away because the producer half looked boring when it was decided.

Problem 4: Held resources

Holding spans in memory for a long time, whether in the SDK, in a Collector, or in the backend, uses real memory. Systems with long gaps and high volume can run into limits quickly if they try to keep traces “open” until the end.

Better ways to think about it

• Separate the waiting from the working. Many teams model the wait explicitly, either as a short span labeled “waiting in queue” whose duration is the gap, or as a metric measuring queue delay. This preserves truth without distorting the trace.
• Use links across long gaps. Starting a new trace in the consumer, with a link to the producer, keeps each trace short and meaningful while preserving the connection.
• Measure the gap as its own signal. Compute the delay as the consumer’s start time minus the message’s creation time, and record it as a metric. Alerts on this number detect slow consumers faster than trace inspection can.
• Choose sampling that does not depend on seeing everything. Combine decisions made at the start of the trace with a small number of rules at the end, and lengthen waiting periods only where truly needed.

flowchart LR
  A["Producer span ends at time 0"] --> G["Message waits in broker: seconds to days"]
  G --> B["Consumer span starts at time N"]
  A -. "link" .-> B
  G --> M["Metric: queue delay"]
    

A tree has one parent per child. Real event systems are not trees.

Fan-out: one message, many readers

In publish-subscribe designs, a single event is delivered to many independent consumers. An “order placed” event might go to billing, inventory, shipping, email, fraud checks, and analytics at the same time. All six consumers extract the same context, so all six spans point to the same producer span. This is perfectly valid, and the trace becomes a wide fork. The challenge is not that it is wrong, but that the trace may become very large and wide, and every branch is on its own timeline, because each consumer works at its own speed.

Fan-in: many messages, one worker

The opposite situation is batching. For efficiency, consumers often fetch many messages at once, for example a hundred records from a stream, and process them together in one go. Now one unit of work is the result of a hundred different messages, each with a different producer context, perhaps from a hundred different traces. A span can only have one parent. Which of the hundred should it be?

flowchart TB
  M1["Message 1 with context from trace A"] --> W["Batch processing span"]
  M2["Message 2 with context from trace B"] --> W
  M3["Message 3 with context from trace C"] --> W
  M4["Message N with context from trace N"] --> W
  W --> R["Result written to storage"]
    

The answer in OpenTelemetry’s guidelines is span links. The batch span gets a link to each message’s producer context, rather than choosing one as its parent. If the consumer then also processes each message individually inside the batch, each of those smaller spans can be a child of the corresponding message’s context, while the outer batch span carries the links. This keeps every relationship true.

Why links are tricky in practice

• Limited tool support. Some tracing tools show links poorly, or not at all, which makes batch processing look disconnected even though the data is correct.
• Link count limits. SDKs cap the number of links per span, so a batch of ten thousand messages cannot have ten thousand links. Instrumentation must choose which to keep, often a sample or the first few, and mark that others were dropped.
• Sampling interaction. A link points to a trace that might not have been sampled. The link then points to nothing you can open, though it still carries the identity.
• Query difficulty. Asking “which batches contained messages from this order’s trace?” requires searching links in reverse, which many backends do poorly.

Partial batches and mixed batches

Real batches mix messages from many sources, some with context and some without. The instrumentation must handle missing context gracefully, create links only for messages that have it, and never fail the batch because one message has a malformed header.

In event-driven systems, doing something twice is normal. Tracing must cope with it.

Most messaging systems promise that a message is delivered at least once. That means that if anything goes wrong, such as a consumer crash, a slow acknowledgement, or a network blip, the broker delivers the same message again. Consumers must therefore be built to handle duplicates, and tracing must make sense of them.

Redelivery and the same context

When the same message is delivered three times, it carries the same context three times. Three consumer spans then share the same parent. The picture looks as if one producer span caused three identical pieces of work, which is true but surprising to a reader. Without extra information, it is hard to tell a legitimate retry from a bug that processes the same event repeatedly.

Helpful extras include recording the delivery attempt number as an attribute on the consumer span, if the broker exposes it, and marking spans that ended in an error and led to a retry. This lets analysts filter to “first attempts only” or to “retries only.”

Dead-letter queues

A dead-letter queue is a holding area where messages go after repeated failures, so they do not block everything else. They might sit there for days until an engineer examines them and sends them back for another try. The context stored in such a message is old. The producer’s trace might have been deleted by retention rules, long since sampled out, or recorded in a system that has been changed since.

flowchart LR
  P["Producer publishes message"] --> Q["Main queue"]
  Q --> C1["Consumer attempt 1: fails"]
  C1 --> Q
  Q --> C2["Consumer attempt 2: fails"]
  C2 --> D["Dead-letter queue"]
  D -->|"days later"| H["Human review and replay"]
  H --> Q
    

Replays and reprocessing

Event streams are often replayed: a consumer is rewound to an earlier point and processes old events again, perhaps to rebuild a database, fix a bug, or backfill a new feature. Every replayed event still carries its original context, possibly from months ago. If replay spans become children of those old contexts, then months-old traces suddenly gain new branches, a confusing and often harmful result. A replay storm can produce an enormous number of spans attached to old traces, drowning real traffic, and sampling decisions inherited from the old context may be misleading.

Practical guidance

• Record attempt numbers. Capture delivery count as an attribute whenever the broker provides it.
• Start new traces for replays and dead-letter handling, with a link to the original context rather than a parent relationship. This keeps old traces unchanged and new work easy to find.
• Mark reprocessing explicitly. Add an attribute such as “replay” so that dashboards can exclude or isolate it.
• Consider turning down sampling during replays, since massive bulk reprocessing can overwhelm tracing systems.
• Make duplicates visible. If your consumers detect duplicates and skip work, record that on the span as well.

Messages do not always travel unchanged. Anywhere a message is rebuilt, the label can be lost.

In a simple picture, a message goes from producer to broker to consumer. In real systems, many programs read a message and write a new one. A stream-processing job reads from one topic, transforms the data, and writes to another. A connector copies data from a database into a stream. A scheduler holds a job until a certain time. Each of these creates a new message. Whether the context survives depends on whether the program explicitly copies it, and by default many do not.

Stream processing

Programs that read, filter, join, and aggregate event streams are the most common context breakers. A filter or a simple one-to-one transform may pass headers through, depending on the framework. A join or an aggregation that combines several input messages into one output message faces the fan-in problem from earlier: which input context should the output carry? Many frameworks choose one of them, silently, or none.

Change data capture

Change data capture turns database changes into events: whenever a row is inserted or updated, a program reads the database’s internal log and publishes an event. The database row has no trace context unless someone stored it there, so the event has none either. The trace ends at the database write and a new, unrelated story begins at the event.

The outbox pattern

A common design for reliable publishing is the transactional outbox. Instead of writing to the database and publishing a message as two separate actions that could fail separately, the service writes the business data and an “outbox” record in one database transaction. A separate relay program later reads the outbox and publishes the messages. This is reliable, but it separates the moment of decision from the moment of publishing, possibly by seconds. If the context is not saved in the outbox record, the relay cannot attach it.

flowchart LR
  S["Service handles request with context"] --> DB["Database transaction: business row and outbox row with saved context"]
  DB --> R["Relay reads outbox rows"]
  R -->|"restores saved context, injects into headers"| BR["Broker"]
  BR --> C["Consumer extracts context"]
    

The fix is to store the serialized context as a column in the outbox record at write time, and have the relay restore it before publishing. This simple habit rescues the trace across the database hop.

Schedulers, delayed jobs, and workflows

Delayed messages, scheduled jobs, and “run this tomorrow” tasks have the same difficulty. The context at scheduling time must be saved with the job, or it is gone by execution time. Whether to continue the original trace or start a new linked one is again a design choice, and for long delays, links are usually kinder.

Gateways and transformations at the edge

Integration products that translate between formats, such as one message standard to another, or add and remove fields, may rebuild the entire message. They can drop headers that they do not recognize. Include them in your propagation tests.

Even after a perfect extraction, the baton can slip from the runner’s hand inside their own house.

Extracting the context from a message is only the first half of the job on the consumer side. The context must then be made current, meaning it is placed in the spot where the rest of the program’s code will find it automatically when it creates spans, makes outgoing calls, or publishes new messages. In most languages, that spot is tied to the running thread or the running task. The trouble is that real consumers rarely do all their work on one thread.

Where the context slips away

• Thread pools. A consumer hands the message to a pool of worker threads. The context lives on the original thread, and the workers start with none. Spans created there become new, orphaned traces.
• Asynchronous frameworks. Callbacks, futures, promises, and reactive streams jump between execution contexts. Unless the framework carries the context along, or the instrumentation wraps the framework, the context is not passed on.
• Internal queues. A consumer may put the message into its own in-memory queue for another part of the program to handle later. This is a small messaging system inside one program, with the same problem and the same cure: store the context with the item.
• Batch handlers. As covered earlier, there is no single context to make current, so a decision about links is required.
• Scheduled and background tasks. Work triggered by timers has no incoming context at all.

The remedy: carry it explicitly

The robust approach is to treat the context as an ordinary piece of data when crossing any boundary where automatic carrying is uncertain. Capture the current context into a variable, pass it with the work item, and make it current again on the other side. Most languages provide small helper tools that wrap executors and callbacks to do this for you. Using them consistently, for every worker pool and async mechanism in the consumer, removes a whole class of broken traces.

Symptoms to watch for

• Consumer traces that start with spans of kind consumer but whose child work appears as separate, rootless traces.
• Outgoing calls from a consumer that do not appear in the trace of the message that caused them.
• Traces that look complete in tests with a single thread, and broken in production with many threads.

Not everyone speaks the same language, especially in an organization that has grown over many years.

The W3C format is the default in OpenTelemetry, but it is not the only one in the world. Older tracing systems and cloud providers created their own header formats before the standard existed. Examples include formats used by certain open-source tracers, by service meshes, and by cloud tracing services. Large organizations often run a mix: new services speak the standard, older ones speak a legacy format, and some cloud services add their own special identifiers.

How OpenTelemetry handles it

OpenTelemetry lets you configure a composite propagator, a list of several propagators that are all applied. On injection, the program writes the context in every configured format. On extraction, it tries each in turn until one finds a valid context. A commonly used setting is an environment variable that lists the formats to use, by name, for each service. By default it uses the W3C format for trace context and for baggage.

What still goes wrong

• Configuration drift. Team A configures the standard format only, team B adds a legacy format. When A sends to B, B can read it. When B sends to A, A ignores the legacy header and, if B did not also write the standard one, A starts a fresh trace.
• Conflicting information. If two formats are present in the same message and describe different traces, the order in which formats are tried decides which one wins, and the choice is easy to misconfigure.
• Different ID sizes and rules. Some older formats use shorter trace IDs or different flag meanings. Translating between them can lose information, such as the sampling decision.
• Cloud-specific mechanisms. Some managed messaging services add their own tracing field to messages on their own, which can coexist with, or overwrite, your custom headers, depending on the service and the client.
• Header name rules. Some carriers restrict characters or letter case in names, and some treat names as case-insensitive. A header written by one library in one style may not be found by another.

A sane strategy

1. Choose the standard format as the organization’s single target.
2. During migration, configure every service to read both the legacy and standard formats, and to write the standard one plus the legacy one for as long as needed.
3. Keep a registry of which services still depend on legacy formats, and retire them on a schedule.
4. Test the mixed-format paths specifically, in both directions.

Choosing which traces to keep is hard enough in a straight line. Queues make it harder.

Recording every trace is usually too expensive, so systems sample: they keep only some. The decision made at the very start of a trace is called head sampling, and its result is stored in the trace flags, so that every later step knows whether to record. This keeps whole traces together, since everyone follows the same decision. A step that respects the incoming decision is using what is often called a parent-based rule.

When the decision crosses a queue

If a producer’s trace was sampled, the flag travels in the message. A consumer using a parent-based rule will also record, keeping the trace whole. If the producer’s trace was not sampled, the consumer will also not record, even if something interesting, such as an error, happens in the consumer. That is a known limitation of head sampling: it cannot see the future.

Link-based starts and independent decisions

When the consumer starts a new trace with a link, it has the freedom to make its own sampling decision for the new trace. That could be more useful if you want, for example, to always keep consumer traces with errors. But it also means that the two traces may be sampled independently: you might see the consumer trace but its linked producer trace was dropped, or the reverse. The link then points to a trace that does not exist in your storage.

Amplification and storms

Fan-out multiplies sampling effects. One sampled producer trace with six consumers produces seven times the spans. Replays and retries multiply them further. An innocent-looking one percent sample rate can produce a much larger share of spans than expected when each sampled trace fans out into hundreds of consumer spans.

Tail sampling and the gap

As explained in the chapter on the asynchronous gap, tail sampling needs all parts of a trace within one waiting period at one Collector. In messaging systems, parts arrive from different services at very different times. This makes tail sampling for entire messaging traces expensive and sometimes impractical. A common compromise is to run tail sampling separately for each short, self-contained trace, producer side and consumer side, linked rather than joined.

flowchart TB
  H["Head decision at producer: sampled or not"] --> F["Flag travels in message headers"]
  F --> C1["Consumer follows parent flag"]
  F --> C2["Consumer starts new trace with link and decides independently"]
  C1 --> R1["Whole trace kept or dropped together"]
  C2 --> R2["Two traces may be kept or dropped separately"]
    

Consistency tools

The OpenTelemetry project has worked on consistent probability sampling, a method where services can use different sampling rates yet still make compatible choices, using extra information stored in the trace state. This is helpful when producers and consumers have different volumes and want different rates. Support varies by language and version, so check before relying on it.

Some business processes take many steps over a long time. Mapping them to a single trace is a design puzzle.

Request-reply over messaging

Sometimes a service sends a message and expects an answer back on another queue. This is called request-reply over messaging. The service that sent the request waits, or at least expects, a response. The tricky part is the return trip. The reply message must carry the context so that the original caller can connect the response to its waiting span. If the caller is waiting on a different thread or even a different copy of the service, the context must also be recovered from storage using a correlation identifier. This is easy to get half right: the request carries context, but the reply is built by a different program that forgets to include it.

Sagas and choreography

A saga is a business process spread over several services, each performing a step and publishing an event that triggers the next. If a step fails, earlier steps are undone with compensating actions. In the choreography style, no one is in charge. Each service reacts to the previous event. In the orchestration style, a central coordinator tells each service what to do. A saga might take seconds or many days, for example an order that waits for a payment confirmation, a warehouse pick, and a delivery.

The challenge: one trace is the wrong shape

Chaining every step as parent and child produces one enormous, extremely long trace. It strains tools, hides the real work in a sea of waiting, and makes sampling impractical. Yet breaking the process into many unconnected traces loses the overall picture. The practical answer combines two ideas:

• Many short traces with links, one for each step or each short burst of work, connected by span links to what triggered them.
• A business correlation identifier, such as an order number, recorded as a span attribute on every step, so that searching by that value gathers the whole process even if the traces are separate.

Workflow engines

Specialized workflow and orchestration engines keep long-running process state in a database and wake up the right code when events arrive. Their internal replay mechanisms, where a workflow function is re-run from the start to rebuild state, can confuse tracing because the same code path produces the same span over and over. Check how your engine’s own tracing integration handles replays, and prefer integrations that avoid duplicate spans on replay.

A message header is data written by someone else. Treating it as trustworthy can be a mistake.

Context headers are ordinary text. Anyone who can write a message to your broker, or call your API that results in a message, can set those headers to anything. That creates several risks that people rarely think about at first.

Forged or abused context

• Forcing sampling. An outside sender can set the “sampled” flag on every message. If your consumers follow the parent’s decision, the sender can make you record every trace, driving up cost and load.
• Polluting traces. A sender can reuse the trace ID of a real, important trace, adding unrelated spans to it, or attach noise to someone else’s investigation.
• Overloading with size. Very large baggage or trace state values may waste memory or slow parsing in every consumer.
• Injecting misleading business hints. If downstream logic reads baggage, such as a customer tier, to change behavior, an untrusted sender could manipulate it. Baggage must never be used for security decisions.

Leaks to outside parties

Baggage travels wherever the context travels. If a service publishes a message that is later delivered to an external partner, or the context is forwarded in a call to a third-party API, everything in the baggage goes along. Internal customer IDs, experiment names, and internal region codes can leak this way. Likewise, trace identifiers themselves reveal how many requests your system handles, or let outsiders correlate their own messages.

Protective measures

• Draw a trust boundary. At the edge where messages enter from outside, ignore incoming context headers, or start a new trace with a link to the foreign context instead of adopting it as a parent.
• Limit sampling influence. Use sampling rules that cap the rate no matter what incoming flags say, so a sender cannot force unlimited recording.
• Strip baggage at the boundary. Remove it before messages leave your organization, and filter incoming baggage to a short list of approved keys.
• Never put secrets or personal data in context. Not in baggage, not in trace state, not in span names made from headers.
• Validate sizes and formats. Reject or truncate absurdly large or malformed context values before using them.
• Check regulations. Identifiers that can be linked to a person may count as personal data in some legal settings. Involve your privacy team in the baggage allow-list.

Time is a shared illusion. Computers keep their own, and they never agree perfectly.

Every span records a start time and an end time taken from the clock of the machine that created it. Different machines have clocks that drift apart, even when synchronized by network time services, typically by small amounts like milliseconds, and sometimes by much more when a service is misconfigured or a machine has been paused. In direct calls, this matters little. In event-driven flows, it matters more, because the producer span and the consumer span run on different machines and are linked only by a header.

What it looks like

• Negative gaps. The consumer’s span appears to start before the producer’s span ended. Visual tools may draw it with a strange overlap, or compute a negative queue delay.
• Wrong order. Two events published in a certain order may appear reversed in a timeline because of clock differences.
• Jittery delay metrics. A delay measured by subtracting timestamps from two machines inherits the clock difference as error. Small delays can look wrong or even negative.

Mitigations

• Keep clocks synchronized using reliable time services, and monitor the clock offset on machines that run important services.
• Prefer the broker’s timestamp for measuring delay when possible, since the broker records a single, consistent timestamp when it receives each message, and the consumer can compare it with its own receive time on its own machine, which is more consistent than comparing across two services.
• Do not trust tiny differences. Treat delays below clock accuracy as noise, and clamp negative values to zero in dashboards, noting that a clamp was applied.
• Use sequence and offsets for ordering. Streaming platforms give each record a position number that is a more reliable order than a wall-clock time.

Event-driven systems move enormous numbers of small messages. Tracing each one fully can cost more than the work it observes.

A queue might handle a hundred thousand messages per second. Each message may produce several spans: one at the producer, one at the broker interaction, one at the consumer, and more inside the consumer’s logic. Without careful choices, tracing can double or triple the amount of data your system emits compared to the business data it processes.

Places where cost grows

• Per-message spans. One span per message for both publish and process multiplies quickly.
• Large fan-out. Every consumer group produces its own spans for the same message.
• Per-receive spans in polling consumers. Consumers that poll the broker constantly may produce spans for empty polls that return nothing, creating noise.
• High cardinality attributes. Cardinality means the number of distinct values. If every span carries a unique message ID or key, backends have more work to index them, and any metrics derived from those attributes become enormous.
• Header bytes. As covered earlier, extra headers cost storage and traffic on every message.

Practical controls

• Sample at the producer. A rate chosen to match volume, applied once so that downstream steps follow it.
• Use metrics for the bulk, traces for the detail. Counts of messages, delays, error rates, and consumer lag are best as metrics. Keep traces for investigation.
• Skip empty polls. Configure instrumentation to create spans only when messages are actually received.
• Limit links in batches. Cap the number of links and record how many were omitted.
• Choose attributes carefully. Prefer attributes with a bounded set of values, such as topic name and operation, over unique identifiers, and keep unique identifiers only on spans, not on metrics.
• Filter at the Collector. Drop spans from known noisy sources close to where they enter the pipeline.

The Collector sits after the spans are created. It can reshape and route, but it cannot invent a lost context.

It is important to understand which problems the Collector can address and which must be solved in the applications. Propagation happens inside the applications: the producer must inject, the consumer must extract. If either fails, no Collector setting can restore a missing parent, because the information was never recorded. What the Collector can do is process spans after the fact.

Tail sampling and the waiting window

Chapter 7 explained that long gaps can exceed the waiting window of a tail sampler. Two approaches soften this. First, treat producer and consumer sides as separate traces joined by links, each short enough to be judged on its own. Second, if you must keep them together, raise the waiting window for only the tier that handles messaging spans and size its memory accordingly, accepting the cost. Both choices should be made on purpose, with a clear explanation of why.

flowchart LR
  P["Producer services"] --> A["Agent Collectors"]
  C["Consumer services"] --> A
  A --> G["Gateway Collectors: filter, redact, enrich"]
  G --> M["Connector: metrics from spans, queue delay and orphan counts"]
  G --> S["Sampling tier: short windows per trace"]
  M --> MB["Metrics backend"]
  S --> TB["Tracing backend"]
    

Generate metrics before sampling

Because traces will be sampled, counts calculated from the surviving traces underestimate real volume. Create message counts, delay distributions, and error counts from the full stream of spans before sampling, then sample the traces afterward. This gives accurate dashboards and keeps trace storage affordable.

Each challenge has a small set of well-known answers. This chapter pulls them together into a decision guide.

A decision table

You cannot fix what you cannot see. Measure how well context actually flows.

Many teams discover broken propagation by accident, during an incident, when the trace they need turns out to be in pieces. A better approach is to measure propagation health continuously, the same way you measure the health of any other system.

Useful health measures

• Orphan consumer rate. The share of consumer spans that have no parent and no link. A high or rising rate means context is not being extracted, or is being dropped upstream.
• Header presence rate. The share of messages that carry a context header, per topic and per producer. Drops point to a specific producer, or a specific hop.
• Link validity. For sampled traces, how often a link points to a trace that actually exists in storage.
• Context size distribution. Percentiles of header and baggage sizes, to catch bloat early.
• Queue delay. Distribution of the time between message creation and processing, per consumer.
• Trace completeness. For a set of test flows, whether the expected spans all appear in the trace.

Synthetic test messages

Send a known test message through important flows on a regular schedule, tagged clearly as a test, and automatically check that the expected chain of spans appears. This catches regressions introduced by upgrades of client libraries, brokers, bridges, or stream jobs, long before real users are affected.

Tests during development

• Unit tests that inject and extract context through the actual carrier type you use, not a mock.
• Integration tests that run a real producer, broker, and consumer, and assert the relationship between spans.
• Concurrency tests that run the consumer with the same thread and worker settings as production.
• Failure tests: redelivery, dead-letter handling, malformed headers, missing headers, and oversized headers.
• Upgrade tests: rerun the whole set whenever a messaging library, instrumentation library, or broker version changes.

Documenting the contract

Write down, for each topic or queue, the propagation contract: which format is used, whether headers or envelopes carry the context, whether consumers use parent or link, which baggage keys are allowed, and who owns the contract. Treat it like any other interface agreement. When a new team joins, this document saves them from rediscovering all of these lessons the hard way.

The same errors appear in team after team. Spot them early.

More mistakes worth avoiding

• Forgetting the outbox. Publishing through an outbox or change capture without storing the context, so the trace always ends at the database.
• Ignoring delivery attempts. Treating retries as separate normal work, which hides bugs that cause constant redelivery.
• Recording empty polls. Creating spans for receive calls that return nothing, filling storage with noise.
• Unbounded links. Linking thousands of messages from a huge batch and hitting SDK limits or backend slowdowns.
• Using baggage for decisions. Letting business logic or security rely on a value that any sender could have written.
• Skipping correlation IDs. Relying only on trace IDs for long processes that will never fit one trace.
• Silent upgrades. Upgrading messaging or instrumentation libraries without rerunning propagation tests.

Best practices in one place

Put the pieces together with an imaginary online shop. Every step is described in words, with no code.

A customer places an order on the shop’s website. The web service saves the order and needs to tell the rest of the company. Let us follow the story and see where the baton is at risk.

Lessons from the example

• Every hop was considered separately: web to database, database to relay, relay to broker, broker to consumers.
• The choice between parent and link depended on timing, count, and risk, not on a single rule.
• Replays, retries, and batches each had a specific rule that kept old traces clean.
• Trust boundary and health measures were designed in, not added after the first incident.

Four composite illustrations of the kinds of problems teams meet. They are not accounts of specific companies.

When a trace looks broken, ask the questions in order. Most problems fall to the first few.

A simple investigation routine

1. Capture one real message that should have context, and read its headers directly from the broker.
2. If the headers are present, the problem is on the consumer side: extraction, making context current, or crossing threads.
3. If the headers are absent, work backwards through each program that touched the message until you find the one that dropped them.
4. Check that both sides use the same propagation format and header names.
5. Check sampling: was the original trace even recorded?
6. Fix the cause, then add a test or a dashboard check so the same failure cannot return unseen.

Context propagation connects the pieces of a request into one story. In direct calls, the connection is simple and short-lived. In event-driven systems, messages wait, fan out, merge, retry, and pass through programs that rebuild them. Each of those events is a place where the baton can be dropped, duplicated, or misread. The solutions are not exotic, but they must be applied with care: agree on one contract, test every hop, use links for batches and long gaps, keep traces short, protect trust boundaries, store context with outbox records and hand it across threads, add business correlation identifiers, and measure the health of propagation itself.