Before learning the tool, let us feel the problem it solves.

A quick reminder about traces and sampling

A trace is the complete story of one request as it travels through your system, made of timed steps called spans. In a large system, recording every trace is expensive, so most companies keep only a share of them. Choosing which traces to keep is called sampling.

The agreement problem

Suppose a customer’s request passes through five services: a gateway, an order service, a payment service, an inventory service and a notification service. If the gateway decides to keep this trace, the other four must also record their parts. Otherwise the trace will have holes, and holes make traces hard to trust. But the five services may run in different languages, on different machines, in different countries, owned by different teams. They cannot all pause to hold a meeting about each request.

The elegant solution is to make the decision once, at the start, and write it into a tiny message that travels along with the request. Every later service reads the message, learns the decision, and follows it. That tiny message is stored in the trace flags.

sequenceDiagram
    participant G as API Gateway
    participant O as Order Service
    participant P as Payment Service
    participant N as Notification Service
    G->>G: Decide to keep this trace and set the sampled flag
    G->>O: Request with trace identity and flags
    O->>O: Read flags and record the span
    O->>P: Request with the same flags
    P->>P: Read flags and record the span
    P->>N: Request with the same flags
    N->>N: Read flags and record the span
    

The short answer first, and then the pieces.

Trace flags are a small set of on and off switches stored as part of a span’s identity, called the span context. They are one byte in size, which means eight switches. Each switch is a single bit that is either zero or one. Trace flags describe facts about the whole trace, and they travel from service to service with the trace ID and span ID.

Where trace flags fit in the span context

A span context is the small, unchangeable identity card of a span. It holds five items. We introduce each one briefly, so that you can see where the flags sit.

What trace flags are not

• They are not span attributes. Attributes describe one span in detail. Flags describe the trace as a whole, and they are tiny.
• They are not the trace state. The trace state holds free-form vendor notes. Flags hold fixed, standard switches.
• They are not baggage. Baggage holds your own key and value notes. Flags hold only standard switches.
• They are not a command. As we will see, the main flag is more like a strong hint than an order.
• They are not about success or failure. That is the job of the span status.

Reading trace flags is easy once you know how to read a single byte.

A quick lesson on bits and hex

A bit is the smallest piece of computer data: it is either 0 or 1, like a light switch that is off or on. A byte is a group of eight bits. Because eight switches in a row are awkward to write down, people write a byte using two hexadecimal characters, which is a counting system that uses the digits 0 to 9 and the letters a to f. So the byte with all switches off is written “00”, and the byte with only the lowest switch on is written “01”.

flowchart LR
    A["Trace flags byte"] --> B["Bit 7: reserved"]
    A --> C["Bit 6: reserved"]
    A --> D["Bit 5: reserved"]
    A --> E["Bit 4: reserved"]
    A --> F["Bit 3: reserved"]
    A --> G["Bit 2: reserved"]
    A --> H["Bit 1: random trace ID flag in newer versions"]
    A --> I["Bit 0: sampled flag"]
    

What each position means

Reading some real values

Where you see them

In the standard web header called traceparent, there are four parts joined by dashes: a version, the trace ID, the parent span ID and the flags. The flags are the last two characters. A full header might end in “01”, which tells the receiver that the sampled switch is on.

The one flag every OpenTelemetry user meets, and the reason the others are rarely discussed.

The sampled flag says whether the trace is being kept. When it is on, the caller recorded its part of the trace, and downstream services are encouraged to record theirs as well. When it is off, the caller did not record, and downstream services should normally not record either.

A hint, not a command

The standard describes the sampled flag carefully. It says that the flag indicates that the caller may have recorded trace data, and that when the flag is off, the caller did not record. It is a statement of what the caller did, which the receiver may use as strong guidance. The receiver is still free to make its own decision, within the rules of its own sampler.

How the flag is first created

At the start of a new trace, there is no parent. The sampler, which is the component in the SDK that decides whether to keep a span, makes the first decision. If it says “record and sample,” the sampled switch is turned on in the new span’s flags. If it says “drop,” the switch is left off. Every child span and every call made onward inherits that value, unless a sampler somewhere along the chain deliberately chooses otherwise.

Why the flag matters even when a trace is dropped

When a trace is not sampled, the span context is still created and still passed on. The next service needs to learn that the decision was “not sampled.” If the context were not passed, the next service would think it was the start of a brand new journey, make its own decision, and perhaps keep a trace that the front door had dropped. That would produce a partial trace, with a root that appears in the middle of the story. So the rule is simple: always pass the context on, even when the flag is off.

The most common confusion about trace flags, cleared up.

The words “recording” and “sampled” sound alike, but they mean different things. Understanding the difference makes many odd behaviours make sense.

flowchart TD
    A["Sampler is asked about a new span"] --> B{"Which decision?"}
    B -->|Drop| C["Not recording. Flag stays off."]
    B -->|Record only| D["Recording. Flag stays off."]
    B -->|Record and sample| E["Recording. Flag is turned on."]
    C --> F["Context with flag off passes to next service"]
    D --> F
    E --> G["Context with flag on passes to next service"]
    

A second switch that helps fairness and counting in advanced sampling.

Many sampling methods use the trace ID as a fair lottery number. For this to be fair, the trace ID must be truly random. If some system builds trace IDs from a counter or from a pattern, such as starting with the date, then a sampler that uses part of the ID as a random number could pick traces unevenly: all the traces from one morning might be kept, or none.

To address this, newer revisions of the W3C trace context standard define a second flag, often called the random trace ID flag. When it is on, the creator of the trace promises that the trace ID, or at least the part of it used for sampling, was generated with proper randomness. Receivers and tools can then trust it for sampling decisions and for estimating totals.

flowchart TD
    A["Trace ID arrives with flags"] --> B{"Is the random flag on?"}
    B -->|Yes| C["Safe to use the trace ID as a lottery number for consistent sampling"]
    B -->|No| D["Do not assume randomness. Use another method or generate a fresh ID"]
    

Why six switches have no labels, and what to do with them.

The remaining bits in the trace flags byte are reserved. This means the standard keeps them for future uses that have not been defined yet. A good standard plans for the future, and the reserved bits are the room left for it.

The rules for reserved bits

• Do not give them your own meaning. If every vendor invented its own use for the spare switches, two systems could read the same bit in different ways, and trace data would be misread.
• Set them to zero when you create flags. A new trace should begin with the reserved bits off.
• Ignore bits you do not understand. A program that reads a flag byte with an unfamiliar bit turned on should still read the parts it knows, and not reject the whole header.
• Be careful when passing on. Libraries differ in whether they keep or clear unknown bits. Check the behaviour of your propagator, particularly if you rely on a newer flag that your older components do not know.

Why forward compatibility matters

Because the flags byte is part of a standard used by thousands of products, a new switch can only be added if old software keeps working when it sees the new switch turned on. By ignoring unknown bits, old and new software can share the same traces, and the standard can grow slowly, without a day when everything must change at once.

The same switches appear in several places along a telemetry journey.

The exported span flags

In the standard protocol used to send telemetry, called OTLP, each span has a field for flags. The lowest eight bits hold the same trace flags we have been discussing. Two further bits record whether the span’s parent was remote, meaning it came from another program, and whether that information is known at all. This extra detail helps tools tell the difference between a span whose parent is missing because it was lost and a span whose parent is simply in another service. The exact fields can differ between protocol versions, so check the version you use.

Flags on log records

The log data model includes a trace flags field next to the trace ID and span ID. This lets a logging pipeline see whether the trace that a log line belongs to was sampled. A pipeline might use this to keep log lines only for sampled traces, or to keep all error logs regardless, depending on policy.

flowchart LR
    A["Span context in memory"] --> B["traceparent header on outgoing requests"]
    A --> C["Message headers for queues"]
    A --> D["Exported span flags in OTLP"]
    A --> E["Log record trace flags"]
    B --> F["Next service reads the flags"]
    C --> F
    D --> G["Collector and backend"]
    E --> G
    

Let us follow the flags through the life of a request.

sequenceDiagram
    participant U as User
    participant A as Service A with sampler
    participant B as Service B parent based
    participant C as Service C parent based
    U->>A: Request with no trace header
    A->>A: New trace. Sampler rolls. Flags become 01
    A->>B: traceparent ends with 01
    B->>B: Parent sampled so keep. Flags stay 01
    B->>C: traceparent ends with 01
    C->>C: Parent sampled so keep. Flags stay 01
    U->>A: Another request with no trace header
    A->>A: New trace. Sampler rolls. Flags become 00
    A->>B: traceparent ends with 00
    B->>B: Parent not sampled so drop. Flags stay 00
    B->>C: traceparent ends with 00
    C->>C: Parent not sampled so drop. Flags stay 00
    

Immutability: changing a flag means making a new context

A span context cannot be edited after it is created. If something needs to change a flag, for example a sampler that wants to override the parent’s decision, it creates a new span context with the new value. This keeps data safe when many parts of a program read the same context at once, and avoids surprising changes in the middle of a request.

What happens in a program with no SDK

If only the API is present and no SDK is installed, the API still reads the incoming context and passes it on unchanged when calling other services. The flags travel through that program untouched, even though it records nothing. This is a clever safety feature: an uninstrumented hop in the middle of a journey does not break the trace or change its sampling decision.

The W3C format is the default, but older and cloud-specific formats carry the same idea in their own way.

Before the W3C standard, various tools invented their own headers. Many companies still run systems that speak these formats. OpenTelemetry’s propagators, the components that read and write headers, can translate between them and the common internal span context.

flowchart LR
    A["Incoming B3 header with sampled 1"] --> P["Propagator reads it"]
    B["Incoming W3C header with flags 01"] --> P
    C["Incoming Jaeger header with sampled bit"] --> P
    P --> S["Internal span context with sampled flag on"]
    S --> O1["Outgoing W3C header with flags 01"]
    S --> O2["Outgoing B3 header with sampled 1"]
    

The “debug” idea and why it gets lost

Some older formats have a special debug value, meaning “keep this trace no matter what the sampler would say.” Engineers use it to force a trace of a specific request while investigating a problem. The W3C standard has no debug flag. When a debug request is translated into the W3C format, the information that it was a forced trace is reduced to the plain sampled flag. Downstream services that follow the W3C format will see “sampled,” but they will not know it was forced. This can matter if a downstream sampler would override the parent’s decision.

Three different ways to carry information with a request. Each has its own purpose.

How they cooperate in advanced sampling

Some newer sampling designs use the trace state to record information about the sampling probability, such as a threshold value, and use the trace flags to indicate whether the trace ID is random. Together they allow later services to know both whether a trace was sampled and how likely it was to be sampled, so that counts can be scaled back up. The flags give the yes-or-no answer, and the state gives extra detail.

How the flag fits with head sampling, parent-based sampling and tail sampling.

Head sampling and the flag

In head sampling, the decision is made at the start of the trace by the sampler in the SDK. The flag is the vehicle that carries the decision. It is the core of how head sampling stays consistent across services.

Parent-based sampling and the flag

The parent-based sampler reads the incoming flag and chooses what to do for the new span. It has separate rules for whether the parent was sampled or not, and whether the parent was local or remote. This means you can control how much you trust incoming flags: follow them fully inside your own network, and treat them as hints at a public edge.

Tail sampling and the flag

Tail sampling is different. In tail sampling, the decision is made later in the Collector, after the trace is complete. The SDKs usually record and send everything, so they behave as if the sampled flag were on. The real decision is made in the Collector, which looks at the whole trace. In this design, the flag in the data may not reflect the final decision, since the decision came after. Do not use the flag on a stored span to find out whether a trace “was sampled by tail sampling”. By definition, any trace in storage was kept.

flowchart TD
    A["Application SDK"] --> B{"Which strategy?"}
    B -->|Head sampling| C["Sampler decides at start and sets the flag"]
    B -->|Tail sampling| D["SDK keeps everything and sends it, flag usually on"]
    C --> E["Dropped traces never leave the application"]
    C --> F["Kept traces go to the Collector"]
    D --> G["Collector holds each trace and decides at the end"]
    F --> H["Backend stores kept traces"]
    G --> H
    

The arithmetic across both layers

If you use both head sampling and tail sampling, remember that the head decision comes first. A trace that was dropped at the start, with the flag off, is never sent to the Collector, so the Collector cannot rescue it, even if it later turns out to contain an error. If you want the Collector to keep every error, the head sampler must not drop that trace before the error happens.

The flag is a small bridge between traces and the other signals.

Logs

When a log record is written while a span is active, it can carry the trace ID, the span ID and the trace flags. The flags tell a logging pipeline whether the linked trace was kept. This is useful in several ways.

• Link checking: a tool can show a “view trace” button only for logs whose trace was sampled, since the trace for other logs does not exist.
• Cost control: a pipeline can keep the logs of sampled traces in full, and keep fewer of the rest, matching the trace policy.
• Error protection: a pipeline can keep all error logs regardless of the flag, so that important evidence is not lost.

flowchart TD
    L["Log record with trace ID and flags"] --> Q{"Is the sampled flag on?"}
    Q -->|Yes| K["Keep and link to the stored trace"]
    Q -->|No| E{"Is it an error or warning?"}
    E -->|Yes| K2["Keep, but note that no trace exists"]
    E -->|No| D["Drop or keep at a low rate"]
    

Metrics and exemplars

Metrics recorded through the metrics API do not depend on the sampled flag, so counts stay accurate. Exemplars are different. An exemplar is an example trace ID attached to a metric measurement, so that you can jump from a graph spike to a real trace. By default, SDKs usually attach exemplars only from spans whose flag says “sampled,” because linking to a trace that was never stored would lead to nothing. So the sampled flag also decides which metric points can carry a useful link.

Follow one flag from the moment it is created to the moment an engineer sees its effect.

Anything in a request header was written by someone. At the public edge, that someone may not be friendly.

Inside your own network, trusting the incoming flags is usually fine. At the public edge, where requests come from the open internet, the trace header may have been written by a stranger. Strangers can write anything, including flags you did not expect.

Three kinds of problems

flowchart TD
    A["Request arrives at public edge"] --> B{"Is the traceparent header valid?"}
    B -->|No| C["Ignore it and start a new trace"]
    B -->|Yes| D{"Do we trust this caller?"}
    D -->|Yes| E["Follow the incoming flags"]
    D -->|No| F["Treat the sampled flag as a hint and apply our own policy"]
    F --> G["Optionally start a new trace and link to the caller"]
    

The practical controls

• Validate the format. The standard requires specific lengths and lowercase hexadecimal characters. Software should ignore headers that do not follow the rules, and begin a new trace instead.
• Use the remote-parent rules of the sampler. At a public edge, configure what happens when the remote parent is sampled, so a stranger’s “keep” is checked against a ratio or a rate limit rather than obeyed without question.
• Rate limit recording. Put a cap on the number of traces recorded per second at the edge, no matter what the flags say.
• Consider fresh traces with links. Start your own trace for requests from outsiders, and record the caller’s trace as a link, so that cross-company debugging is still possible.

An honest look at what trace flags give and what they cost.

Key trade-offs

The flags themselves hold no personal data, but how you treat them still matters.

When flags go wrong, the symptoms are visible if you look for them.

• Share of requests arriving with the flag on: compare with the share you expect. A jump may mean a partner is forcing recording.
• Share of spans recorded versus flag on: if many requests arrive with the flag on but few spans are exported, exporters or processors may be failing.
• Traces with only one or two spans: suggest that flags or headers are lost between services.
• Roots in the middle of the system: a trace whose first span is deep inside the system suggests a hop that drops the header.
• Invalid header counts: count malformed or unreadable headers at your gateways.
• Per-service keep rates: each service should keep about the same share. One that differs may not follow the parent.
• Log link success: measure how often a log’s trace link leads to a stored trace.
• Format mismatches: during migrations, watch for traces that break at the boundary between systems that use different formats.

Common symptoms and likely causes

Habits worth copying and traps worth avoiding.

Good design patterns

Anti-patterns

A practical checklist for flags you can trust.

Best practices

• Use the standard format. Make W3C Trace Context your default, and add other formats only where old systems need them.
• Check every hop. Gateways, proxies, load balancers and service meshes must pass the trace headers through, with the flags unchanged.
• Use parent-based sampling everywhere. Let the front door decide, and let everyone else follow.
• Treat the edge differently. Decide how much to trust incoming flags from outsiders, and apply limits.
• Let libraries handle headers. Do not build or parse trace headers by hand.
• Respect reserved bits. Ignore what you do not understand, and never assign your own meaning.
• Mind the debug gap. If you rely on forced traces, test that they survive across formats and hops.
• Use flags on logs. Align log keeping with trace sampling, but always keep errors.
• Watch the numbers. Track the share of flags on, the share of traces kept and the count of broken or invalid headers.
• Test propagation end to end. Add a simple check that the flag arrives unchanged at the last service.
• Read the standard when adopting newer flags. Check tool support before you rely on the random flag.

Common mistakes beginners make

• Thinking trace flags are the same as the trace state, or the baggage.
• Believing the sampled flag is a command that every service must obey.
• Confusing “sampled” with “recording”.
• Reading the two hex characters as a number rather than as switches. “01” means the lowest switch is on, not “one percent”.
• Dropping the headers for unsampled traces.
• Hard-coding flag values in manual headers.
• Expecting the flag on stored spans to show whether tail sampling kept them.
• Using flags for security decisions.

How trace flags appear in tools and platforms you may already know.

A typical journey for a growing company

Quick answers to questions beginners ask most.

If you remember only a few things, remember these.