The short answer first, then the words that make it clear.

Trace State is a small list of named notes that travels along with a trace from one service to the next. Each note is a key paired with a value, written as two short pieces of text joined by an equals sign. Tracing tools use these notes to carry extra information that the basic trace identity has no room for: a sampling hint, a routing label, a vendor-specific code, or a reference to another system.

Let us unpack the vocabulary. OpenTelemetry, often shortened to OTel, is a free, open source framework, hosted by the Cloud Native Computing Foundation, that gives developers one standard way to produce telemetry: the stream of traces, metrics, and logs that describes how software behaves. A trace follows one request through every service it touches, and it is built from spans, each span being one timed step of the work. Every span carries a small bundle of identity facts called the span context. Trace state is one of the pieces inside that bundle.

The span context holds five things. The trace identity is a long random number shared by every span of the trace. The span identity names this particular step. The trace flags hold tiny switches, the most important being the sampled flag, which says whether the trace is being kept. The remote marker tells whether the context arrived from another program. And the trace state is the optional list of vendor notes. The first three are essential. The trace state is optional and is the only part that is open to extension.

What trace state answers

• Can tools share extra facts? Yes. Each tool may add its own named note without disturbing the standard fields.
• Can one tool read another’s note? It can see it, but it should not change it, and it should pass it along unchanged.
• Is it required? No. Many requests carry an empty trace state, and everything works.
• Does it carry application data? No. It is meant for tracing-system information, not for business details.
• Does it travel automatically? Yes, when the standard propagation is configured, it rides in a header next to the main trace header.

What trace state is not

Trace state is not baggage. Baggage is a separate feature that lets your application code attach its own facts, such as a customer tier, so that other services can read them. Trace state is for tracing tools, not for your business logic. Trace state is also not a span attribute. A span attribute is a label stored with one span and sent to the backend for people to read. And trace state is not a place for large or private data. It travels on every network call, so it must stay small and harmless.

A Closer Look at Why It Is a List

You might wonder why trace state is a list of pairs and not one free-form text. The answer is safe sharing. Imagine three vendors that all want to leave a note. If the trace state were one blob of text, each vendor would have to understand the others’ formats, and any careless edit might damage someone else’s note. By making it a list with named keys, each vendor owns its own entry and can change that entry without touching the rest.

This design is a classic solution to a coordination problem. When many independent parties write into one shared place, the safest rule is “every party gets its own labelled slot, and nobody edits another party’s slot.” Trace state follows that rule, and the rest of this guide will show how the small details of the format support it.

Why You Rarely See It, Yet It Matters

Most engineers work for years with tracing and never look at the trace state. The libraries handle it quietly, and no dashboard highlights it. This invisibility is a sign of good design, since a feature that demands constant attention would be a burden. But invisibility also hides risk. When something goes wrong with a vendor feature or with sampling statistics, the trace state is often where the first clue lies. Knowing that it exists, and roughly what it looks like, turns a baffling problem into a short investigation.

A standard identity is not enough, and private extras must not break the standard.

In the early days of distributed tracing, every tracing product invented its own headers. A request that passed through a product from one vendor and then a product from another often lost its trace, because the second product did not understand the first one’s header. Companies that wanted to mix tools were forced to choose one, or to write awkward translation code.

The web standards organization, the W3C, solved the core problem by publishing the Trace Context standard. It defines a common header, traceparent, that carries the trace identity, the span identity, and the flags in one fixed format. With this, tools from any vendor can join the same trace. But the standard header was deliberately kept small and strict. It had no room for vendor-specific extras, and yet vendors still had real needs: to pass along their own sampling hints, their own routing information, and their own correlation codes.

The standard therefore added a second header, tracestate. It is the official place for vendor-specific extras, and it comes with rules that keep one vendor’s notes from damaging another’s. OpenTelemetry adopted both headers as its default way of passing trace context, and exposes the trace state as part of its span context.

A short history

The Problem Before Standards

Before a shared format existed, a company that wanted to mix two tracing products faced an uncomfortable choice. It could ask both vendors to build converters, which was slow and fragile. It could write its own translation layer, which was costly to maintain. Or it could standardise on one vendor and give up the benefits of the other. Each option locked the company in or slowed it down. A shared header for identity, plus a shared and safe place for private extras, removed this lock-in and let companies choose tools on their merits.

Eight small ideas hold up the whole topic. Learn these and the rest will feel natural.

How the ideas fit together

Here is the whole story in one breath. A request arrives at a service, carrying two headers. The propagator reads them and builds a span context that includes the trace state. The sampler may look at the trace state and may add or change a note. The new span carries the resulting trace state. When the service calls another service, the propagator writes the trace identity and the trace state into the outgoing headers, and the cycle repeats.

Why immutability matters

Programs often do many things at once. If trace state could be changed in place, two parts of a program might change it at the same time and corrupt each other’s work. By making trace state immutable, OpenTelemetry guarantees that any copy you hold is stable. When you add, update, or delete a note, you receive a brand-new trace state, and the old one is left alone. Everyone holding the old copy continues to see exactly what they saw before.

Who Owns What

Ownership is the key to every rule in this guide. The tracing library owns the trace identity, the span identity, and the flags. Each vendor tool owns the single entry that carries its own key. The sampler owns any sampling notes. Your application code usually owns nothing in the trace state at all. When every party sticks to what it owns, the whole system works without negotiation. When one party starts changing what it does not own, subtle bugs appear that are hard to trace to their source.

The format is simple on purpose. A few small rules make it safe for many tools to share.

When trace state is written into a network header, it becomes a single line of text. The line is a list of notes separated by commas. Each note is a key, an equals sign, and a value. A short example, from the W3C standard itself, looks like this in words: the key “congo” paired with the value “t61rcWkgMzE,” followed by a comma, followed by the key “rojo” paired with the value “00f067aa0ba902b7.” The two names are made-up vendors used in the standard’s examples.

The rules for keys

Keys must be written in lowercase letters, digits, and a few simple symbols, and they must begin with a letter or a digit. A key can also take a two-part form for platforms that host several customers: a tenant name, an at-sign, and a vendor name. For example, a hosting platform might use a key made from the customer’s name, an at-sign, and the platform’s own name. This way, one vendor can serve many customers without their notes colliding. Keys have a length limit of 256 characters in the simple form, and the standard sets tighter limits on the two halves of the tenant form.

The rules for values

Values are plain printable text. They may not contain commas, because commas separate entries, and they may not contain equals signs, because equals signs separate keys from values. They may not end with a space, and they have a limit of 256 characters. These rules keep the line easy to split and easy to read by any program, without special quoting or escaping.

What happens when the rules are broken

If a trace state header does not follow the rules, for example because it has duplicate keys, a bad character, or too many entries, the standard says that a receiving tool should discard the whole trace state and start with an empty one, rather than try to repair it. The trace identity, carried by the other header, is unaffected. This is a good example of “failing safe”: a damaged note is dropped, and the trace continues.

Reading a Real Example Slowly

Take the list with two entries: first rojo with the value 00f067aa0ba902b7, then congo with the value t61rcWkgMzE. Reading from the left, the first entry is the most recently changed, so the vendor named rojo touched the trace last. The second entry was written earlier by congo and has stayed the same since. Neither value means anything to a human reader, and that is fine: each is a private code that only its owner understands. A third party looking at this list can learn only the names of the tools involved and the order in which they acted.

Two headers travel together. One is essential, and one is a bonus.

The traceparent header holds four parts: a version number, the trace identity, the identity of the parent span, and the flags. It has a fixed length and a fixed format, and every standard-following tool must understand it. The tracestate header holds the list of vendor notes. It is optional and variable in length.

The most important row is the last one. Trace state has meaning only in the company of a valid traceparent. If the main header is missing or damaged, the receiver begins a new trace, and any trace state that came with the bad header is discarded as well. This is natural: notes about a trace make no sense if the trace itself cannot be identified.

A note on the sampled flag

The sampled flag lives in the traceparent header, not in the trace state. When people say “the sampling decision travels in the header,” they mean this flag. The trace state can carry additional sampling facts, such as the probability with which the trace was chosen, but the yes-or-no mark itself is part of the main header. We will see later how these two work together.

Versioning and Future Growth

The identity header begins with a version number, which gives the standard room to grow without breaking old tools. The trace state has no version, because it is already designed to be extensible: new vendors simply add new keys. A tool that does not recognise a key keeps it and passes it on. This forward-looking design means that new features can appear in some parts of a system without forcing every other part to be upgraded first, which is exactly what you want in a large organisation where teams move at different speeds.

The notes ride along on every network call, and each service has a clear set of duties.

When one service calls another, the tracing library takes the current span context and writes it into the headers of the outgoing request. This is called injection. When a service receives a request, the library reads the headers and rebuilds the span context. This is called extraction. Both steps are performed by the propagator. The default propagator in OpenTelemetry follows the W3C standard, so both headers are handled together without any extra work from the programmer.

sequenceDiagram
    participant G as Gateway
    participant C as Checkout
    participant P as Payment
    G->>C: traceparent flag 01 and tracestate congo=t61rcWkgMzE
    Note over C: Reads both headers and builds the span context
    Note over C: A vendor tool adds its own entry on the left
    C->>P: traceparent flag 01 and tracestate rojo=00f067aa0ba902b7,congo=t61rcWkgMzE
    Note over P: Passes the unknown entries along unchanged
    P-->>C: Reply
    C-->>G: Reply

Explaining Figure 1

Read the diagram from top to bottom. The gateway calls checkout and writes two headers: the identity header with its sampled flag, and the trace state, which here holds one note from the vendor named congo. Checkout extracts both. A tracing tool from a second vendor, named rojo, adds its own note, and the library places it at the left of the list. When checkout calls payment, it writes the longer list. Payment does not recognise either vendor, but it knows the rule: pass unknown notes along unchanged. The notes therefore reach every later hop intact.

The three duties of a service

Services that do not understand trace state

A service that does not use any vendor tool still has an important job. If it uses the standard propagator, it will extract the trace state and inject it again on outgoing calls, even though it never looks inside. This “pass-through” behaviour is essential. If a middle service dropped the trace state, vendors on either side of it would lose their notes and might behave incorrectly. Make sure every service in a chain, including proxies and gateways, forwards both headers.

Asynchronous Work and Queues

Not every call is a direct request and reply. A service may place a message on a queue, and another service may pick it up minutes later. The headers of a network call do not exist in this case, so the trace context, including the trace state, must be copied into the message’s own properties and read back by the consumer. Libraries for popular queues often do this automatically, but older or custom code may not. When the context is not copied, the consumer starts a fresh trace and the notes are lost, which is why queues appear again and again in troubleshooting stories.

Because trace state is immutable, every change produces a new copy. Four simple operations cover everything.

OpenTelemetry offers a small set of operations on the trace state. You can get the value of a key, add a new entry, update the value of an existing entry, and delete an entry. The three that change something, add, update, and delete, never alter the original. Each one returns a new trace state with the change applied, and the original stays exactly as it was.

flowchart TD
    A["Existing trace state: congo=t61rcWkgMzE"] --> B{"Which operation?"}
    B -- "Get" --> C["Read the value; nothing changes"]
    B -- "Add a new key" --> D["New copy: new entry placed at the left"]
    B -- "Update an existing key" --> E["New copy: entry changed and moved to the left"]
    B -- "Delete a key" --> F["New copy: entry removed"]
    D --> G["Original is left untouched"]
    E --> G
    F --> G

Explaining Figure 2

The diagram starts with an existing trace state holding one note. If the program only wants to read a value, the Get operation returns it and nothing changes. The other three operations each lead to a new copy. Adding places the new entry at the left of the list. Updating changes an existing entry and also moves it to the left. Deleting removes the entry. In every case, the original trace state remains untouched, shown at the bottom of the diagram.

Why the newest goes to the left

The standard asks that the most recently modified entry sit at the left end of the list. This small habit tells a reader which vendor touched the trace last. It also helps when the list grows too long: if a tool must shorten the list to fit the limits, the entries at the right, which have been unchanged the longest, are the ones that can be dropped first with the least damage.

The golden rule: only touch your own entries

Each vendor should add, update, or delete only the entries whose keys it owns. It should leave every other entry exactly as it found it, including the order of the remaining entries. This rule is what makes the shared list safe. A vendor that rewrites everything, or that deletes entries it does not recognise, can break features in other tools without ever knowing it.

There are a few cases where a vendor must make room. If the list would exceed 32 entries, the standard allows a tool to remove entries from the right end, preferably the longest ones first, since they cost the most space. Even then, a tool should avoid removing entries unless absolutely necessary, and it should never reorder entries it does not own.

What a Typical Day of Changes Looks Like

On a normal day, in a normal system, the trace state is changed far less often than you might fear. Most requests arrive with an empty list or a list that a vendor wrote at the entrance. Most services forward it without touching it. One or two components, a monitoring tool or a sampler, may add or update a single entry. By the time the request completes, the list has changed once or twice, and each change was made by the owner of the entry. This calm pattern is what the rules are designed to produce.

The sampler can read the notes and add its own, which lets sampling facts travel with the trace.

Sampling means keeping only a portion of the traces, because recording everything is expensive. A sampler is the small piece of code that decides, each time a span is about to start, whether the trace will be kept. If you have read about ParentBased sampling, you know that the yes-or-no decision travels in the sampled flag of the main header. Trace state adds a second channel, one that can carry more detail about how and why the decision was made.

When a sampler is called, it receives the parent’s span context, which includes the incoming trace state. When it answers, it returns three things: the decision, any extra attributes to add to the span, and a trace state. The trace state it returns becomes the trace state of the new span, and from there it is injected into outgoing calls. A sampler may return the trace state unchanged, which is the most common case, or it may add or update an entry of its own.

flowchart TD
    A["Request arrives with traceparent and tracestate"] --> B["Propagator extracts the span context"]
    B --> C["Span context includes the incoming trace state"]
    C --> D["Sampler is asked: keep this span?"]
    D --> E["Sampler returns decision and a trace state"]
    E --> F{"Did the sampler change the trace state?"}
    F -- "No" --> G["New span carries the incoming trace state"]
    F -- "Yes" --> H["New span carries the updated trace state"]
    G --> I["Propagator injects it into outgoing calls"]
    H --> I

Explaining Figure 3

The diagram follows a request through one service. The propagator rebuilds the span context, including the trace state, from the incoming headers. The sampler is then asked whether to keep the span, and it hands back both its decision and a trace state. If it left the trace state alone, the new span simply carries what arrived. If it added or changed a note, the new span carries the updated version. Either way, the result is written into the headers of outgoing calls, so downstream services see the sampler’s notes.

Why sampling facts are worth carrying

Consider a backend that wants to estimate how many requests really happened by looking only at the traces that were kept. If one trace in ten is kept, each kept trace stands for about ten real requests. But the backend can only multiply correctly if it knows the probability with which each trace was chosen. If different services use different rates, or if the rate changes during the day, the backend needs that information attached to each trace. The trace state is a natural place to carry it.

Here is a small example. Suppose 1,000 traces were kept at a probability of ten percent. The backend estimates 1,000 ÷ 0.10 = 10,000 real requests. If the probability had been five percent, the same 1,000 traces would stand for 1,000 ÷ 0.05 = 20,000 requests. Without the probability, the backend cannot tell these two cases apart, and its counts will be wrong by a factor of two.

Consistent probability sampling

OpenTelemetry has been developing a newer approach, often called consistent probability sampling, that uses the trace state for exactly this purpose. It reserves a short vendor key for OpenTelemetry itself, the two-letter key “ot,” and places inside it small sub-values, such as a compact threshold that says how likely the trace was to be chosen, and, when needed, an explicit randomness value that all services can use to make the same decision from the same random number. The benefit is that services using different probabilities can still make decisions that fit together: a trace chosen at a lower probability is always a subset of what a higher probability would have chosen. The details of the encoding are still evolving, so check the current specification before building anything that depends on them.

What ParentBased does with the trace state

The ParentBased sampler, which follows the parent’s decision, normally leaves the incoming trace state alone and passes it on. This is exactly what you want in the common case: the root of the trace made the decision and wrote any notes, and every later service simply honours the decision and forwards the notes.

Two Channels, Two Jobs

It helps to remember that the sampled flag and the sampling notes in the trace state do different jobs. The flag answers a simple question that every service must be able to answer instantly: should I keep this? The notes answer a more subtle question that only the backend needs to ask later: with what probability was this chosen? Keeping the two apart means that even a tool that knows nothing about trace state can still respect the sampling decision correctly, while advanced tools gain extra information when it is available.

Four different places can hold a fact. Choosing the right one avoids confusion and cost.

Beginners often ask why OpenTelemetry has so many ways to attach information. The answer is that each one has a different audience and a different lifetime. Putting a fact in the wrong place either hides it from the people who need it or sends it to people who should not see it.

flowchart LR
    R["One request between services"] --> H1["traceparent header"]
    R --> H2["tracestate header"]
    R --> H3["baggage header"]
    H1 --> A["Every tracing tool: identity and sampled flag"]
    H2 --> B["Vendor tools and samplers: private notes"]
    H3 --> C["Application code: business facts"]
    S["Span attributes and resource attributes"] --> D["Sent only to the backend, never to the next service"]

Explaining Figure 4

The diagram shows one request leaving a service with three headers attached. The identity header is read by every tracing tool. The trace state header is read by vendor tools and samplers. The baggage header is read by application code. Notice the separate box at the bottom: span attributes and resource attributes never ride along on the request. They go straight to the backend with the exported data. This is why an attribute is the right choice for anything that only people need to read, and trace state or baggage is the right choice only when another service must see the fact during the request.

A simple decision guide

• Does a tracing tool need it during the request? Use trace state, and keep it tiny.
• Does your own application code in another service need it? Use baggage, with care about what you reveal.
• Do only people reading the trace need it? Use a span attribute.
• Is it a fact about the program or machine? Use a resource attribute.

Mixing Them Up Is a Common Beginner Mistake

New users often put a customer identifier in trace state because it travels across services, and then discover that their application code cannot easily read it, because the trace state is meant for tracing tools. Others put technical notes in baggage, where every service’s code can see them, and then discover that the notes leak to outside companies. The decision guide above exists to prevent both of these. Spend a minute with it each time you are tempted to attach a new fact to a request.

The notes do not vanish when a span ends. They are exported with the span so that backends can use them.

When a span finishes, the SDK hands it to an exporter, which sends it to a Collector or directly to a backend. The standard wire format used by OpenTelemetry, called OTLP, includes a field for the span’s trace state. The field holds the same text that would appear in the header: a comma-separated list of entries. The spans of a trace therefore carry the trace state they had at the moment they were created.

Links, which connect one trace to another, also carry a trace state of their own, because a link points to another span context, which may have its own notes. This lets a backend understand not only which trace a link refers to, but also what the sender knew about it.

What backends do with it

Most tracing screens do not show the trace state to ordinary users. It is an engineering detail, something you inspect when something looks odd. Yet it can be a precious clue. If a trace appears broken or a vendor feature does not work, the trace state of consecutive spans often reveals which hop lost or damaged the notes.

Collectors and trace state

The OpenTelemetry Collector passes the trace state along with the spans it receives. Processors can read it, and some can change it, though most do not. If you build rules in a Collector that depend on a note, make sure that every upstream service writes it consistently, or the rule will behave unpredictably on traces that lack the note.

Why Backends Like Having It

For the people who build tracing backends, the trace state in exported spans is a gift. Without it, a backend that wanted to know how a trace was sampled, or which tool had touched it, would have to guess or to ask the application through some separate channel. With the notes stored on each span, the backend can answer these questions directly from the data it already has. This is the reason vendor tools bother to write notes at all: the notes are a bridge between what happens inside a request and what the backend can learn afterwards.

A few extra bytes on every call add up faster than you would think.

Trace state travels in a header on every network call, so its size is multiplied by the number of calls. Suppose a system makes 50,000,000 service-to-service calls per day, and each call carries a trace state of 200 bytes. The total is 50,000,000 × 200 = 10,000,000,000 bytes, which is 10 gigabytes of extra network traffic per day, just for notes. If a careless tool doubles the size to 400 bytes, the cost doubles to 20 gigabytes.

The standard’s limits allow a trace state to become very large in theory: up to 32 entries, each with a key of up to 256 characters and a value of up to 256 characters. That is more than 16,000 bytes. Many web servers and proxies commonly limit the size of a single header to around 8 kilobytes, so a trace state near the theoretical maximum could cause requests to be rejected. In practice, nobody should come close to the limits. A budget of a few hundred bytes for the whole list is a healthy target.

Performance of parsing

Parsing the header takes a small amount of time on every request, in every service. A short, well-formed list costs almost nothing. A long list with many entries takes more time and memory, and a malformed one is parsed and then thrown away. These costs are small for each call but add up for services handling thousands of requests per second. Keeping the list short keeps this cost invisible.

Estimating Before You Adopt a New Tool

Before adding a new vendor tool, ask two simple questions. How many bytes will its entry add to the header, and how many calls does your system make per day? Multiply the two numbers to learn the daily cost. A tool whose entry is 40 bytes on 10,000,000 calls adds 400,000,000 bytes, which is about 0.4 gigabytes per day. A tool whose entry is 400 bytes adds ten times as much. This quick arithmetic often reveals which tools are cheap to adopt and which deserve a closer look.

The header travels across networks, sometimes to other companies. Treat it like a postcard, not a sealed letter.

Because trace state rides on every call, it also travels wherever your calls go. When your service calls a third-party application programming interface, the same headers can be sent to that outside company. When a customer’s browser or a partner’s program calls you, it can send any headers it likes. Both directions deserve a thought.

flowchart TD
    X["Request arrives from outside your network"] --> S{"Is the header size within your limit?"}
    S -- "No" --> R["Reject or drop the trace state"]
    S -- "Yes" --> V{"Does it follow the format rules?"}
    V -- "No" --> R
    V -- "Yes" --> T{"Is the caller a trusted partner?"}
    T -- "Yes" --> K["Keep the entries you recognise"]
    T -- "No" --> C["Clear the trace state and start fresh"]
    K --> P["Process the request"]
    C --> P

Explaining Figure 5

The diagram shows a checkpoint at the edge of your network. A request arrives, and the first question is about size. If the header is too big, the trace state is rejected or dropped. If the size is acceptable, the next question is whether the format rules are followed. If not, the trace state is dropped again. Only well-formed, reasonably sized trace state moves on to the third question: is the caller trusted? A trusted partner may keep its notes, which lets partners cooperate. An unknown caller has its notes cleared, and the system starts with a fresh, empty list.

Do not use trace state as an access control

Trace state is not a secure channel. Anyone who can send a request can set any entries they like, and anyone who can see the network can read them. Never decide who may do what based on a value in the trace state, and never place a secret there. It is a place for convenience notes, not for authorisation.

Partners and Trust

Not every outside caller is an enemy. Many companies integrate closely with partners and want traces to continue across the boundary. In that case, agree in advance on which keys each side may write and what the values mean, and treat the partner’s notes as trusted only for those keys. Everything else is cleared at the checkpoint. A written agreement of two or three lines is enough, and it turns an informal habit into a safe, documented arrangement that survives staff changes on both sides.

Trace state fails quietly. A lost note rarely causes an error; it simply stops a feature from working.

Because the trace still works without trace state, problems are invisible to ordinary health checks. A vendor feature may silently degrade, or sampling arithmetic may drift, and nobody sees a failure message. You must look for problems on purpose, using a few simple checks.

Check after every change

Keep a Known-Good Sample

One of the most useful troubleshooting tools is a saved example of a healthy trace state, captured when everything was working. When a problem appears, compare the current header with the saved example, and any difference jumps out: a missing entry, a changed order, an unusual size. Store the sample next to your documentation and refresh it after intentional changes. Teams that do this fix mysterious header problems in minutes instead of days.

Every design has a price. An honest view helps you decide how much to rely on trace state.

Common trade-offs in plain language

When trace state is the wrong tool

If you find yourself wanting to carry customer details, feature flags, or business decisions from service to service, trace state is not the right place. Those are application facts, and they belong in baggage or in explicit request parameters. If you want to record details only for people to read later, use span attributes. Trace state is reserved for the narrow job of letting tracing tools share notes about the trace itself.

Counting the Real Benefit

When deciding whether to use vendor notes, count the benefit as carefully as the cost. A note that lets a vendor link a trace to its own record may save engineers ten minutes per investigation, and a team that investigates twenty incidents a month saves more than three hours. A note that lets a backend scale sampled counts correctly may prevent a wrong capacity decision worth far more than any header cost. Write the benefit down next to the byte cost, and let the comparison guide the decision, not habit or enthusiasm.

Patterns are tried-and-true recipes. Habits are the small routines that keep trace state small, safe, and useful.

Habits that pay off

• Know which tools write notes. Keep a short list of every tool and the key it owns.
• Use standard libraries. Well-tested propagators handle the format rules correctly, including ordering and limits.
• Never hand-build the header. Manual string editing is how invalid entries are born.
• Keep values short. A value of a few dozen characters is plenty for almost every use.
• Test the whole path. Send a test request end to end and check the notes at the far end.
• Measure the size. Track the typical and maximum size of the header, and watch for growth after releases.
• Document your keys. If your own team uses a key, write down what it means, who owns it, and when it can be removed.
• Remove what you no longer use. Retire notes when the tool or feature that needed them is gone.

A gentle path to taking control

Keeping Documentation Close to the Code

A pattern that works well is to keep a short document describing every trace state key in the same repository that configures the propagators. The document names the key, its owner, the meaning of its value in plain words, its expected size, and the date it should be reviewed. Because the document lives beside the configuration, it is updated in the same change, and reviewers notice when a new key appears without an explanation. This small habit prevents the slow accumulation of mysterious entries that nobody dares to remove.

Most trace state problems come from the same few mistakes. Knowing them in advance saves a lot of confusion.

Myths worth clearing up

A Review Checklist Before You Add a Key

• Is it really needed? Could an attribute or baggage item do the job instead?
• Is the value small and opaque? It should be a short code, not a description or a name.
• Does it follow the format? Lowercase key, no commas or equals signs in the value, within the length limits.
• Is there one entry per tool? Updated in place, never appended at each hop.
• Is the boundary guarded? Incoming notes checked at the edge, outgoing notes stripped for outsiders.
• Has the whole path been tested? One request, end to end, with the notes checked at the far end.

Five minutes with this list before each change prevents nearly all of the problems described above.

Let us follow one imaginary online bookshop from a tangle of headers to a tidy, safe setup, using simple numbers.

Riverside Books has a gateway, a checkout service, and a payment service, and it calls an outside delivery partner. Two vendor tools are in use. A performance monitoring tool, whose key we will call rojo, writes a short note at the gateway. A routing tool, whose key we will call congo, writes a note at the checkout service. The shop handles about 10,000,000 internal calls per day.

Step 1: see what actually travels

The team captures the trace state at each hop of a test request.

The last row alarms the team. The delivery partner, an outside company, receives internal notes that are meaningless to it and might reveal something about the shop’s tools.

Step 2: count the cost

The two notes together are about 60 bytes. Over 10,000,000 calls a day, that is 60 × 10,000,000 = 600,000,000 bytes, about 0.6 gigabytes per day. That is acceptable. But the team also discovers that a misconfigured tool adds an 80-byte entry at each of six hops, appending a new one each time instead of updating its single entry. That adds 6 × 80 = 480 bytes, and the total grows to 540 bytes per call, or 540 × 10,000,000 = 5,400,000,000 bytes, about 5.4 gigabytes per day. The trace state has become nine times larger for no benefit.

Step 3: fix the growth

The team configures the misbehaving tool to update its own single entry in place. The header returns to about 60 bytes, and a new alert fires whenever any trace state exceeds 200 bytes. The savings are immediate: from 5.4 gigabytes per day back to 0.6.

Step 4: guard the boundaries

At the public gateway, the team adds a checkpoint: headers above a size limit are rejected, malformed trace state is dropped, and notes from callers that are not known partners are cleared. At the payment service’s outgoing call to the delivery partner, the team strips the trace headers, because the partner has no use for the shop’s internal notes. The partner still receives a fresh trace identity if both sides agree to cooperate, but no private notes.

Step 5: verify

The team sends a test request through the complete path and checks the trace state at every hop. The notes appear in the right order, the size is about 60 bytes, and the delivery partner receives nothing it should not. A hostile test with an oversized header from outside is rejected at the gateway. After a week of production traffic, the size alert has never fired, and vendor features that depended on the notes are working normally.

Three short stories show how trace state behaves in everyday situations.

Choosing Where to Begin

If your organisation is only starting out, do not try to master everything at once. Begin by capturing real trace state from a few requests and writing down what you see. Confirm that every hop forwards both headers. Set a size budget and an alert. Then add the edge checkpoint and the outgoing strip for outside parties. Each step is small, brings a clear benefit, and can be tested alone. Trace state is a humble feature, but handled with care it gives every tool a safe place for its notes, and gives every engineer a clear clue when something goes wrong.

Trace state is a small idea with a quiet but important role: a safe place for tools to leave notes.

We began with a parcel that carries a standard label and a few private stickers, and we ended with a clear picture of trace state. We saw where it sits in the span context, how it appears in the tracestate header, and which rules keep it safe to share. We followed it from service to service, watched how it is added, updated, and deleted, and saw how samplers use it to carry probability. We compared it with baggage and attributes, counted its cost, guarded its boundaries, and learned to check it. The cure for almost every trace state problem is the same: forward everything, change only your own entry, and keep it small.