Message queues are everywhere in modern systems, and they are exactly where observability tends to go dark.

What a message queue is

A message queue is a middleman that holds messages until someone is ready to handle them. A service that sends a message is called a producer. A service that receives and handles it is called a consumer. The middleman itself is called a broker. Well-known brokers include Apache Kafka, RabbitMQ, Amazon SQS and SNS, Google Pub/Sub, and Azure Service Bus.

Teams use queues for good reasons. A queue lets the producer finish quickly without waiting for slow work. It absorbs sudden bursts of traffic. It lets consumers fail and recover without losing work. And it lets many separate services react to the same event. An online shop, for example, may place an “order created” message on a queue, and then the payment, stock, email, and analytics services each handle it in their own time.

What a trace needs

A trace is the full story of one request as it passes through many services, and each step of the story is a span. For a trace to join spans from different services, they must share a trace ID, a long random number, and each span must know the ID of its parent span. That identity data is the span context. When two services talk directly over the web, the context travels inside the web request. Queues remove that direct conversation. The producer never talks to the consumer. It only talks to the broker, and the consumer only talks to the broker.

What goes wrong without it

If the producer and consumer do not share context, each creates its own separate trace. A slow order in the shop shows up as a fast “create order” trace and a separate slow “handle order” trace, with nothing to connect them. Engineers cannot see that one caused the other. They cannot measure how long the message waited in the queue. They cannot follow a failure back to its source. Dashboards that count end-to-end journeys become impossible to build.

A handful of words carry the whole topic. Learn these and the rest becomes much easier.

The W3C Trace Context format

The most common format is defined by the W3C, the organization that sets web standards. It uses a header named traceparent, which has four parts separated by dashes: a version, the 32-character trace ID, the 16-character ID of the parent span, and two characters of flags. One flag bit says whether the trace is being sampled, meaning kept for storage. A second header, tracestate, can carry extra vendor-specific values. The total traceparent is always 55 characters long, which is why it adds so little weight to a message.

Other formats

Older and vendor-specific formats still exist: B3 from Zipkin, the Jaeger format, and the Amazon X-Ray format. OpenTelemetry supports them through separate propagators and lets you combine several in a composite propagator, which writes all of them and reads whichever it finds. This is vital during migrations, when half of your services speak one format and half another.

Two kinds of travel

Context moves in two different ways, and beginners often confuse them. In-process propagation carries context inside one program, across function calls, threads, and async tasks, using thread-local storage or similar tools provided by the language. Inter-process propagation carries context between separate programs, using inject and extract through a carrier. This article is mostly about the second, but the first matters too, because context lost inside a consumer is just as harmful as context lost on the wire.

Before adding a queue, look at the simple case. It shows the pattern that every messaging system copies.

The five-step dance

When service A calls service B over HTTP, five things happen. First, A starts a span for the outgoing call. Second, A asks the propagator to inject the current span context into the request headers. Third, the request travels to B. Fourth, B asks the propagator to extract the context from the incoming headers. Fifth, B starts its own span as a child of the extracted context. The trace ID stays the same throughout, and the parent link ties the two spans together.

sequenceDiagram
    participant A as Service A
    participant P as Propagator
    participant B as Service B
    A->>A: Start client span
    A->>P: Inject current context into request headers
    P-->>A: Headers now hold traceparent
    A->>B: HTTP request with traceparent header
    B->>P: Extract context from request headers
    P-->>B: Parent context rebuilt
    B->>B: Start server span as child of extracted context
    B-->>A: HTTP response

Why this works so smoothly

The web request is a natural carrier. It has headers, it is sent immediately, it gets exactly one response, and there is exactly one receiver. Because the call is synchronous, the parent span is still open when the child starts, so the relationship is clear. Instrumentation libraries for web frameworks do all five steps automatically, which is why many teams never think about propagation until the first queue appears.

Configuring propagators in practice

Propagators are normally chosen once, at application start-up, either in code or through a standard environment setting that lists the formats to use, separated by commas. A typical choice is the W3C trace context together with baggage. During a migration, the list may include a second format, so that both old and new services understand each other. Because the setting is global, a change affects every call the program makes, web and messaging alike. Keep the setting identical across all services in a system, or at least document every difference, since a mismatch is the single most common reason traces stop at a boundary.

Queues are not just slower web calls. They change the shape of the relationship between sender and receiver, and propagation must adapt.

Three gaps to bridge

The first gap is a time gap. The producer’s span has ended long before the consumer’s span begins. This raises a design question: should the consumer’s span still be called a child, or only related? Later chapters explore this choice in depth.

The second gap is a cardinality gap. One message may feed many consumers, and one consumer may handle many messages at once. A tree-shaped trace, in which every span has a single parent, cannot describe these many-to-many relationships without extra tools such as links.

The third gap is a carrier gap. Every broker stores metadata differently, and some do not allow any metadata. The inject and extract logic must be adapted to each broker’s way of attaching extra data to a message.

flowchart LR
    subgraph NOPROP["Without propagation"]
        P1["Producer trace A"] --> Q1["Queue"]
        Q1 --> C1["Consumer trace B, unrelated"]
    end
    subgraph WITHPROP["With propagation"]
        P2["Producer span in trace A"] -->|"traceparent in message headers"| Q2["Queue"]
        Q2 -->|"headers delivered with message"| C2["Consumer span in trace A or linked to it"]
    end

Every message has a body and some metadata. Context belongs in the metadata, never in the body.

Why not put it in the body

The message body is the business content, such as the order details. Mixing tracing data into it forces every producer and consumer to agree on a changed schema, breaks strict validation, pollutes data that may later be stored for years, and tempts developers to start using the trace ID as a business key. Metadata, by contrast, is separate and can be added or ignored without touching the business content.

Metadata styles by broker

The getter and setter idea

Because every carrier looks different, OpenTelemetry’s propagators never touch a carrier directly. Instead, they use two tiny helpers. A setter knows how to write one key and value into the carrier. A getter knows how to read a key, and how to list all keys. When you build messaging instrumentation, you write a getter and setter for your broker’s header type, and the standard propagators do the rest. This is why one propagator works across web requests, Kafka headers, and AMQP properties alike.

The producer has one job when it comes to propagation: make sure the message leaves with the right identity attached.

What the producer does

sequenceDiagram
    participant PR as Producer Service
    participant BR as Broker
    participant CO as Consumer Service
    PR->>PR: Start producer span, make it current
    PR->>PR: Inject traceparent into message headers
    PR->>BR: Send message with headers
    BR-->>PR: Acknowledge receipt
    PR->>PR: End producer span
    Note over BR: Message waits in the queue for seconds, minutes, or hours
    BR->>CO: Deliver message with the same headers
    CO->>CO: Extract context from headers
    CO->>CO: Start consumer span with extracted context as parent or link
    CO->>CO: Handle the message and end span
    CO->>BR: Acknowledge message

A subtle point: which span’s identity is written?

The context written into the message is the one that is active at the moment of injection. If you inject before starting the producer span, the message carries the identity of whatever span was active earlier, perhaps the incoming web request’s span. If you inject after starting the producer span, the consumer will point back to the send operation itself. The second choice is usually better, because it describes the actual handoff, and it is what the semantic conventions (the shared naming and structure rules for telemetry) recommend.

What about messages created now and sent later?

Some systems separate creation from sending. A message may be built in one place, placed in a buffer, and sent in a batch by a background thread. The context must be captured at creation time, not at send time, because by the time the background thread runs, the original request’s context is gone. Good instrumentation writes the context into the message when it is created and treats the send as a separate step.

The consumer must rebuild context from the message and then make sure that context stays alive while the work is done.

If extraction finds nothing

Not every message carries context. Messages sent before instrumentation was added, messages from third-party producers, and messages from older services will arrive with empty headers. Extraction must never fail loudly in that case. It should return an empty context, and the consumer span starts a new trace. Counting how often this happens, as a metric, tells you how much of your system still lacks propagation.

If extraction finds garbage

Headers can also be malformed, truncated, or malicious. A well-written propagator validates the format and ignores anything invalid, so that a bad header cannot crash the consumer or poison the trace with nonsense identifiers. If you write a custom propagator, validate the lengths and characters of every field.

Keeping context alive while working

Extracting context is only the start. The consumer must make that context active for the duration of the work. If the framework hands the message to a worker thread, or if the handler is asynchronous, the active context may not travel with it. Child spans created inside the handler then attach to the wrong parent or to none. Section 16 of this guide covers in-process propagation in more detail, but remember now that “extract” and “make current” are two separate steps.

After extracting context, you must decide how the consumer span relates to the producer span. There are two main answers, and each has a price.

Option one: make the consumer a child

In a parent-child relationship, the consumer span uses the producer’s span as its parent. Both share the same trace ID, so the trace viewer draws one connected tree: the web request, the send, the long wait, the handling, and everything the handler did. This gives the most natural picture of a single business transaction and makes end-to-end timing easy to read.

Option two: start a new trace and add a link

In a link relationship, the consumer starts a new trace with its own trace ID, and the consumer span carries a link pointing to the producer’s span context. The two traces are separate, yet each can find the other. This keeps traces short and independent, while still preserving the connection.

flowchart TD
    S{"How many messages does one consumer span handle?"}
    S -- "Exactly one" --> D{"Is the delay between send and handling short and bounded?"}
    S -- "Many, as a batch" --> L1["Use links from the consumer span to every message context"]
    D -- "Yes, seconds to minutes" --> P1["Make the consumer span a child of the producer context"]
    D -- "No, hours or unbounded" --> L2["Start a new trace and add one link to the producer context"]
    P1 --> F{"Does the message fan out to many consumers?"}
    F -- "No" --> OK["Keep parent-child"]
    F -- "Yes, many" --> L3["Consider links to keep each consumer trace small"]

What the conventions say

The OpenTelemetry messaging conventions suggest that for a span handling a single message, the context extracted from the message is used as the parent, and for spans that handle several messages at once, links are the right tool. The conventions leave room for judgment, and many teams adopt a hybrid: parent-child for normal short delays, and links when delays become long or when a message goes to many receivers. Whatever you choose, apply it consistently across services, so that engineers know what to expect when they open a trace.

A worked example of the choice

Consider an online shop. When a customer places an order, the web service publishes an “order placed” message, and the payment service handles it within a second or two. These services form a tight conversation, so the payment span is a child of the publishing span, and the customer’s checkout trace shows the whole journey. The same message also reaches a reporting service that reads it in batches every ten minutes. For that service, a batch span with links is the honest answer, because each batch holds hundreds of orders from hundreds of checkouts. Finally, a reminder service receives a “send review request” message scheduled for three days later. It starts a fresh trace with a single link back to the order, because a trace that stays open for three days would be unusable. One message, three consumers, three different relationships, each chosen to fit the way that consumer works.

Propagation tells spans how to connect. The messaging conventions tell them what to look like, so that every tool can read them the same way.

The operations of a message

The conventions describe a message’s life as a handful of operations. Create means building the message. Send (or publish) means handing it to the broker. Receive means asking the broker for messages and getting some back. Process means actually handling a message’s content. Settle means telling the broker the outcome, such as acknowledging success or rejecting it. Instrumentation produces spans for the operations that make sense for the broker in use.

Naming and attributes

Span names follow a simple pattern: the operation followed by the destination, such as “send orders” or “process orders”. Using the destination (the queue or topic name) rather than the message content keeps names stable and low in variety, which matters for grouping and for storage. The span also carries attributes, which are labels that describe it.

Real consumers almost never fetch one message at a time. Batches are where simple propagation breaks and links earn their keep.

The batch problem

A Kafka consumer calls a method that returns a batch of records, perhaps five hundred of them, coming from many different producers and many different traces. A span that covers the fetching of that batch has five hundred possible parents. A tree allows only one. Choosing a single message’s context as the parent would be arbitrary and would hide the other four hundred ninety-nine.

The solution: links

The receive span for a batch uses links, one for every message context in the batch. The span itself starts a fresh trace or continues the consumer’s own context. Each link says “this work was triggered by that message”. A trace viewer can then show, for any message’s trace, which batch span picked it up, and for any batch span, every message that fed it.

flowchart LR
    M1["Message 1 from trace T1"] --> R["Receive span for one batch"]
    M2["Message 2 from trace T2"] --> R
    M3["Message 3 from trace T3"] --> R
    R --> PR1["Process span for message 1, link to T1"]
    R --> PR2["Process span for message 2, link to T2"]
    R --> PR3["Process span for message 3, link to T3"]

Per-message spans inside a batch

Many teams combine two levels. A single receive span covers the fetch, with links to all messages. Then, for each message, a process span is started with the message’s own context as the parent, or with a single link, depending on the choice from the previous chapter. This gives two benefits: the fetch is timed once, and each message’s handling appears in its own producer trace, which is exactly where an engineer investigating a particular order wants to see it.

Limits of links

Links cost memory and storage. A span with five hundred links is heavy. SDKs also apply limits to the number of links on a span, silently dropping extras. For very large batches, consider linking only a sample of the messages, or only the ones that were sampled, and record the batch size as an attribute so that the count remains visible. The goal is useful navigation, not a perfect index.

Queues allow communication shapes that web calls cannot. Each shape asks something different of propagation.

Fan-out: one message, many consumers

In a publish-subscribe system, a single message is delivered to several consumers, each in its own group. The producer wrote its context once, and every consumer extracts the same context. If all consumers choose parent-child, they all become children of the same producer span, and the trace grows several branches, one per consumer. That is a true picture of what happened, and it is often exactly what you want for a moderate number of consumers.

The picture can become unwieldy when the number of consumers is large, or when consumers run at wildly different times, so that the trace stays “open” for days as late consumers join. In such cases, links keep each consumer’s trace short and independent.

flowchart TB
    PRD["Producer span: publish order created"] --> BRK["Topic on the broker"]
    BRK --> C1["Consumer group A: payment service"]
    BRK --> C2["Consumer group B: stock service"]
    BRK --> C3["Consumer group C: email service"]
    BRK --> C4["Consumer group D: analytics service"]
    C1 --> S1["Process span, child or link of producer span"]
    C2 --> S2["Process span, child or link of producer span"]
    C3 --> S3["Process span, child or link of producer span"]
    C4 --> S4["Process span, child or link of producer span"]

Fan-in: many messages, one result

Fan-in is the reverse. An aggregator may wait for ten messages, such as parts of an order from ten services, and then produce a single combined message. The new message has ten possible ancestors. The honest way to record this is for the aggregator’s span to link to all ten contexts, then inject its own context into the combined message. Downstream consumers then continue from the aggregator’s span, and the ten source traces remain discoverable through links.

Request-reply over queues

Some systems use queues for remote procedure calls. The caller sends a request message with a “reply to” queue name and a correlation identifier, then waits for the answer on the reply queue. Propagation must work in both directions. The request carries the caller’s context. The reply carries the responder’s context. The caller, on receiving the reply, can use the extracted context as the parent of the span that continues its work, or simply keep working inside its original span if it blocked while waiting.

It is worth separating two ideas here. A correlation ID is a business-level identifier that ties a request to its reply in the application’s logic. A trace ID is an observability identifier that ties spans together for monitoring. They often serve different audiences and should not be merged. Using the trace ID as the correlation ID would make your application logic depend on your tracing setup, which can be sampled, reconfigured, or switched off.

Kafka is the most common streaming platform, and it has features that shape how propagation behaves.

How Kafka stores messages

Kafka keeps messages, called records, in ordered logs called topics, which are split into partitions. Records are not deleted when they are read. They remain for a configured retention period, and each consumer group tracks its own position, called an offset. A consumer can even go backwards and read old records again. This durable log design has several effects on tracing.

Headers carry the context

Each Kafka record can carry headers: a list of names with byte values. The producer instrumentation injects traceparent and optional tracestate and baggage headers into this list, and the consumer instrumentation reads them from each record after a poll. Headers have been supported by Kafka since version 0.11, so modern deployments handle them. Very old clients or message formats do not.

Kafka-specific behaviors that surprise people

Ordering and partitions

Kafka guarantees order only within a partition. Spans from records in different partitions can run in parallel on different consumers. A trace viewer will show these as concurrent siblings. Do not expect the span order to match the order of production across partitions. Adding the partition number and the offset as attributes helps engineers locate a specific record when debugging.

Consumer groups and trace shape

A Kafka consumer group is a set of consumers that share the work of reading a topic, so that each partition is read by one member at a time. If two different groups read the same topic, each sees every record, which is the fan-out case. A trace viewer will therefore show one producer span with one branch per group. When a group is scaled from three consumers to thirty, the trace shape does not change, because the branch belongs to the group, not to the individual consumer. What does change is the spread of spans over hosts, so add the client or host identity as an attribute when you want to see which member handled which record.

Queue-style brokers behave differently from log-style brokers, and cloud services add their own conventions.

RabbitMQ and AMQP

RabbitMQ receives messages at an exchange, which routes them to one or more queues. Consumers pull messages from queues and send back an acknowledgement when done. Context travels in the message’s headers table. Because messages are removed after acknowledgement, replay is rare, and the delay between send and handling is usually short, which makes parent-child a natural choice. The consumer’s span should cover the handling and end around the time of the acknowledgement. Watch for header name case: AMQP headers are case sensitive, so a producer writing Traceparent and a consumer reading traceparent will not find each other.

Amazon SQS

SQS stores messages in a queue until a consumer deletes them. Consumers poll for batches, and messages that are not deleted become visible again after a timeout, which is how redelivery works. Custom context can ride in message attributes, but SQS limits a message to a small number of them, so writing several tracing headers and also your own business attributes can hit the limit. AWS also provides the AWSTraceHeader system attribute, which carries the X-Ray format header and is populated when an X-Ray-aware sender writes to the queue. Many teams use both: the X-Ray attribute for AWS-native tools, and a W3C attribute for OpenTelemetry-based tools.

Amazon SNS and fan-out to queues

SNS publishes one message to many subscribers, often several SQS queues. Message attributes can travel through SNS to the subscribing queues, but only if the subscription is configured to deliver the original attributes as attributes rather than wrapping the whole message in an envelope. Wrapped delivery hides the attributes inside the body, and the consumer must then unwrap it before extraction. This is a frequent reason why traces stop at SNS.

AWS Lambda as a consumer

When Lambda reads from SQS, the service itself delivers batches of messages to the function. The platform passes the trace header from the message to the function’s environment so that X-Ray traces connect. For OpenTelemetry, the function’s instrumentation must extract the W3C context from the message attributes of each record, and decide between parent-child for single-record handling and links for batches. The function’s short life means spans must be flushed before it freezes, and a missed flush looks like missing context but is actually missing data.

Some brokers give you nowhere to put metadata. You can still propagate context, but you must build the carrier yourself.

The envelope pattern

If the broker offers only a payload, wrap the payload in a small envelope that holds two sections: a metadata section and a data section. The metadata section carries the trace context fields, and the data section carries the original business content. The producer builds the envelope, and the consumer unwraps it. The pattern works with any broker, including Kinesis, plain Redis lists, files, and older protocol versions.

Costs of the envelope

An envelope changes the message format, so every consumer must understand it, including tools that peek into the stream for debugging or analytics. It also tempts people to read tracing fields as business data. Keep the metadata block clearly named, document it, and never rely on it for application logic.

Real systems are full of patterns that move a message through extra stages. Each stage is a chance for context to fall off.

The outbox pattern

Many services need to update a database and publish a message as one reliable action. The outbox pattern solves this by writing the message into a table in the same database transaction as the business change. A separate relay process later reads the table and publishes the messages to the broker. The relay runs long after the original request ended, so it has no active context. If the outbox row does not store the trace context, the published message will have none.

flowchart LR
    A["Web request span"] --> B["Service writes order and outbox row in one transaction"]
    B --> C["Outbox row stores traceparent column"]
    C --> D["Relay or change capture tool reads the row later"]
    D --> E["Relay restores context from the stored traceparent"]
    E --> F["Relay publishes message with headers"]
    F --> G["Consumer extracts and continues the trace"]

The fix is to store the current traceparent value in a column of the outbox row at write time, and have the relay restore it as the parent or link when publishing. Some change-data-capture tools, such as Debezium’s outbox event router, support carrying tracing information through this path. Check the documentation of the tool you use.

Retries and redelivery

When a consumer fails and the broker delivers the message again, the second attempt extracts the same context and creates a second process span. Under parent-child, both attempts are siblings under the producer span. Mark each attempt with a retry count attribute, and record the failure of the first on its span, so the trace explains why there are two. Without that detail, engineers see duplicate work and suspect a bug in the producer.

Dead letter queues

After too many failures, a broker may move a message to a dead letter queue, a holding area for problems. The message and its headers usually travel along. A tool or a person later inspects it and perhaps replays it. Good practice is to preserve the original trace context, and add attributes about why the message was moved and how many times it was tried. A replay should then start a new trace with a link to the original, because the gap may be days, and the original trace may have expired.

Delayed and scheduled messages

Messages meant for the future, such as a reminder in three days, make a planned time gap. Parent-child would keep a trace “open” for three days, which many backends handle poorly. A link is the clean choice: the scheduled consumer starts a new trace and links to the scheduling span.

Extracting context from a message is only half the work. The other half is making sure it stays attached to the work that follows.

How a program remembers the current context

Inside one program, OpenTelemetry stores the current context in a place that depends on the language. In some languages it is thread-local storage, a variable that has a separate value for each thread. In others it is an object passed explicitly through function calls, or a special variable that follows asynchronous tasks. Instrumentation reads this “current context” whenever it creates a child span or makes an outgoing call.

Where context gets lost

The fixes

The standard fixes are to wrap thread pools and executors so that they capture the current context when a task is submitted and restore it when the task runs, to use the context-aware versions of async helpers that your language provides, and to keep the message and its extracted context together in the buffer, activating the right one when each item is handled. Most OpenTelemetry SDKs and agents offer these wrappers. Check which ones exist for your stack.

Sampling decides which traces are kept. Queues make this tricky, because the two halves of a story may be judged at different times and by different services.

A quick reminder

Sampling means keeping only a fraction of traces to control cost. With head sampling, the decision is made when the trace starts and is carried in the sampled flag of the trace header. With tail sampling, a collector waits until most of the trace has arrived and decides then, based on what happened, such as errors or slowness. Both interact with queues.

Head sampling and parent-child

When the consumer is a child of the producer, the consumer’s parent-based sampler reads the sampled flag from the extracted context and follows it. Both sides are kept or dropped together. This is the cleanest outcome. A message whose producer was dropped will produce a consumer span that is also dropped, saving cost on both sides.

Head sampling and links

When the consumer starts a new trace with a link, its root sampler makes a fresh decision, independent of the producer’s. The result may be that the producer trace is kept and the consumer trace dropped, or the reverse. Engineers then follow a link into a trace that does not exist. A link-aware sampler fixes this by looking at the links when the span starts and choosing to keep the new trace whenever a linked trace was sampled. The sampler interface in OpenTelemetry passes the links to the sampler for this purpose.

flowchart TD
    A["Consumer receives message with traceparent"] --> B{"Relationship chosen?"}
    B -- "Child of producer" --> C["Parent based sampler reads the sampled flag"]
    C --> D["Same decision as producer: kept together or dropped together"]
    B -- "New trace with link" --> E{"Sampler inspects links?"}
    E -- "Yes, link aware" --> F["Keep new trace if the linked trace was sampled"]
    E -- "No, independent root" --> G["Random decision: halves may be mismatched"]

Tail sampling and long delays

A tail sampling collector holds spans for a fixed decision wait, often tens of seconds. A message that waits in a queue for ten minutes will produce consumer spans long after the producer’s spans were judged. Under parent-child, those late spans arrive for a trace that has already been decided. Some collectors remember recent decisions, but only for a limited time and number of traces, so late spans may be dropped or treated as orphans. This is a strong reason to prefer links when queue delays are long, and to give the consumer side its own tail sampling policies, such as keeping all traces with errors.

Baggage lets you carry your own labels along with the trace. Over queues, it becomes both more useful and more dangerous.

What baggage is for

Baggage is a set of key-value pairs that travels through the same propagation path as the trace identity, in a header named baggage. A service early in the chain can put in a value such as the customer tier, and every service later in the chain can read it and, if it wishes, copy it onto its own spans. This lets you group and filter all spans of a request by a business attribute that only the first service knew.

Why queues change the picture

On a web call, baggage reaches one receiver. With a queue, baggage is written into a message that may be read by many consumers, some of which belong to other teams, or even other companies, and the message may sit in storage for days. Anything you place in baggage is then visible to every consumer and visible in any tool that inspects message headers.

Size and limits

The W3C baggage format sets minimum limits that implementations should support, around eight kilobytes in total, and many systems enforce smaller ones. Baggage that grows without discipline can overflow header limits in a broker, particularly in systems like SQS that allow only a small number of attributes. Keep baggage to a handful of short values, and review it regularly.

Copying baggage onto spans

By default, baggage does not automatically appear as span attributes. A span processor can copy chosen keys onto each new span, so that trace searches can use them. Copy only an approved list of keys. Copying everything would turn an accidental secret in baggage into a permanent attribute in your trace storage.

Context data is input from another program, and input from another program must never be trusted blindly.

Forged or manipulated context

Anyone who can write to your queue can write any traceparent header. A malicious or buggy producer might set the sampled flag on every message, which forces your consumers to record and pay for every trace. It might copy a real trace ID from another team to pollute their traces with unrelated spans. It might send absurdly long headers to waste memory. For queues that accept messages from outside your own organization, treat extracted context as a hint and not as a command.

Practical defenses

• Validate format strictly. Ignore headers that are the wrong length or contain illegal characters.
• Do not obey the sampled flag from untrusted sources. Make your own sampling decision and add a link to the foreign context instead of becoming its child.
• Limit baggage. Cap the number of entries and bytes you accept, and ignore the rest.
• Strip at boundaries. When a message leaves your trust zone, such as going to a partner, remove internal baggage and decide whether to share trace identity at all.
• Do not derive security decisions from tracing data. A trace ID or baggage value must never decide who can see or do anything.

Privacy of message headers

Headers are visible to broker administrators, monitoring tools, and sometimes to many consumers. Placing trace IDs there is normally harmless, because they are random, but combining them with other metadata may allow a person with broker access to reconstruct the path of a request. If your data is sensitive, consider who has read access to the queues and whether trace identities should be shared with them.

Data retention

Because messages in a log-style broker live for days or weeks, the trace context in their headers can outlive the traces in your tracing system, and may outlive your retention rules. That is not usually a problem, since an ID alone reveals nothing. It does mean that replayed messages point to traces that no longer exist, which is a nuisance rather than a risk.

If propagation silently breaks, traces just look shorter than they should. You need numbers that tell you when this happens.

Metrics worth publishing

The messaging conventions also define standard metrics, such as the duration of publish, receive, and process operations and counts of sent and consumed messages. Using the standard names lets ready-made dashboards work with your data. As with attributes, metric names have evolved, so confirm what your libraries emit.

The queue latency number

Queue latency deserves special attention. It is computed by subtracting the time recorded when the message was created from the time the consumer started handling it. It tells you directly whether consumers are keeping up. Because it depends on clocks in two different machines, be aware of clock differences. A few milliseconds of skew is harmless when delays are seconds. It misleads when delays are tiny.

Canary messages

A canary is a synthetic message sent every minute through each important queue, with a known destination and a recognizable name. An alert fires if the consumer’s span with the same trace link does not appear within a set time. Canaries test the complete path, including managed pieces such as topic subscriptions, that your instrumentation cannot see.

Reading the numbers together

The metrics are most useful in combination. Rising queue latency with steady coverage means consumers are falling behind, so look at capacity. Falling coverage with steady latency means a producer lost its instrumentation, so look at recent deployments. Rising extraction failures after a broker upgrade suggest that the new version rewrites or truncates headers. A sudden surge in orphan consumer spans on one topic points to a new, uninstrumented producer. Teaching on-call engineers these small patterns turns a pile of charts into a quick diagnosis, and saves the long guesswork that otherwise follows every broken trace.

Propagation is cheap per message, but cheap things multiplied by billions deserve a calculation.

The header weight

The standard trace header is 55 bytes. With the header name and the encoding that brokers add around each header, call it about 70 bytes per message. Consider a platform that sends one million messages per second. The added data is about 70 megabytes per second, or around 6 terabytes per day. For most platforms this is a tiny portion of total traffic, because the payloads themselves are far larger. But if your messages are tiny, such as short status updates of a few dozen bytes, the header can double their size, and the cost is no longer negligible.

CPU and memory

Injecting and extracting are small operations, but they run on every message. They involve string building, parsing, and validating, so avoid creating new objects needlessly and reuse propagator instances. Spans created for each message cost more than the headers do. On a very hot path, sampling the span creation, or creating spans only for a fraction of messages, can save CPU, while still propagating the header so that downstream services can decide for themselves.

The trade-off between richness and cost

A good compromise is to start lean for the busiest destinations, and enable richer detail on lower-volume, high-importance queues such as payments or orders. Revisit the choice as traffic changes.

Propagation bugs are quiet. Nothing crashes. Traces just come out shorter than they should. Only deliberate tests catch them.

Use an in-memory exporter

Most SDKs include an exporter that keeps finished spans in memory, which lets automated tests read them back and check parents, links, kinds, and attributes. Combine it with a containerized broker for realistic integration tests, and you can run the whole pipeline on a developer’s laptop or in a build server.

Test the sad paths

Write tests where the broker rejects the message, the headers are removed by a proxy, the consumer crashes mid-processing, and a message is replayed a day later. Each of these exercises an error path that real systems will meet, and each tends to reveal a place where context is lost or an exception escapes.

These are the situations that most often break connected traces across queues. Knowing them in advance shortens every investigation.

Scenario 1: Traces end at the broker

Producer spans exist, consumer spans exist, but they never connect. The usual causes are a producer without messaging instrumentation, a consumer extracting from a differently named header, a managed hop that strips attributes, or a propagator that was never configured. Recovery: look at the raw message on the broker. If the header is absent, the problem is on the producer side or in a hop. If it is present, the problem is in the consumer’s extraction or configuration.

Scenario 2: Two formats, no overlap

The producer writes the W3C format while the consumer reads only the older B3 format, or the reverse. Neither complains. Recovery: use a composite propagator that writes and reads every format in use during the migration, then retire the old format once all services have moved.

Scenario 3: Huge traces from long delays

A scheduled task parented to its creator makes a trace that stays open for days. Trace viewers struggle, tail sampling policies misjudge, and storage costs rise. Recovery: switch long-delay paths to links, set a maximum acceptable delay for parent-child, and make that rule visible in the team’s guidelines.

Scenario 4: Missing context after a retry or dead letter hop

A message moved to a dead letter queue arrives without headers because the relay copies only the payload. Recovery: make sure every component that moves messages copies headers, and add a test that follows a message through the dead letter path.

Scenario 5: Wrong parent after a thread hop

Database spans appear in separate traces rather than under the consumer span, because the handler’s work moved to another thread. Recovery: wrap executors, use context-aware helpers, and add a thread hop test.

Scenario 6: Sampling mismatch

Engineers follow links from a kept producer trace and find nothing, because the consumer trace was dropped. Recovery: use link-aware sampling on the consumer side, or choose parent-child so the decision is shared.

Scenario 7: Header size limit exceeded

Baggage grows, a broker rejects messages with too many attributes, and the producer starts failing. Recovery: cap baggage at the producer, and treat tracing headers as optional so that failure to attach them never blocks the send.

Scenario 8: Clock skew distorts queue latency

Producer and consumer clocks disagree, so queue latency appears negative or inflated. Recovery: keep clocks synchronized, treat very small or negative values as measurement noise, and when precise timing matters, measure waiting time using broker-provided timestamps.

Teams that have connected traces across many queues tend to land on the same good habits, and to regret the same shortcuts.

Patterns that work

Anti-patterns to avoid

Different platforms solved the same problem in slightly different ways. Their choices reveal the trade-offs in practice.

Connecting traces across every queue in a large organization takes time. A phased plan reduces risk and shows progress early.

A message queue separates a producer and a consumer in time and in knowledge. Context propagation repairs the gap by writing a small identity record into the message’s metadata when it is created, and reading it back when the message is handled. Everything else is a set of decisions about how to use that record well: whether the consumer becomes a child or a linked new trace, how batches and fan-out are described, how sampling stays consistent, how untrusted senders are handled, and how you will notice when the chain breaks.