When one request touches many programs, finding the slow or broken part becomes a detective story. Tracing gives the detective a map.

From one big program to many small ones

Years ago, most applications were one large program. A request came in, the program did all the work, and an answer went out. If something was slow, you looked inside that one program. Today, many teams split their software into microservices. A microservice is a small program that does one job, such as handling logins, calculating prices, or sending emails. Each one runs in its own container, and Kubernetes keeps them all running.

This split has real benefits. Teams can change one small part without touching the others, and parts can grow independently. But it also creates a new problem. A single click by a customer may cause the website to call the login service, which calls the user database, then the price service, then the stock service, then the payment service. Any one of these could be slow or broken.

What the other tools cannot tell you

You may already have metrics, which are numbers over time such as requests per second. You may also have logs, which are lines of text that programs write about what they did. Both are valuable. But metrics tell you that the checkout is slow on average, without saying which step is slow. Logs from twelve different programs are written in twelve different places, and it is hard to tell which lines belong to the same customer click.

Tracing fills this gap. It ties together everything that happened for one request, across all the programs, in order, with times. With a trace open, you can say, “this request spent eight hundred milliseconds waiting for the stock service, and the stock service spent most of that time waiting for its database.”

Questions a trace answers

Three simple words carry the whole idea. A trace is the story, a span is one step of the story, and context is the thread that ties the steps together.

The trace

A trace is the complete record of one request as it moves through your system. It starts when the request first enters, for example when a customer clicks a button, and ends when the last program has finished its part. Every trace has a unique identifier, called the trace ID, which is a long random number. All the steps that belong to the same request carry the same trace ID.

The span

A span is one unit of work inside a trace. It has a name, a start time, and an end time, so it knows how long it took. A span might be “handle the checkout request,” “ask the stock service,” or “run a database query.” Each span has its own span ID. It also remembers the ID of the span that caused it, which is called its parent. The first span in a trace has no parent and is called the root span.

Because each span knows its parent, spans form a tree. The root is the whole request, its children are the steps it took, and their children are the smaller steps inside those.

flowchart TD
    ROOT["Root span: checkout request, 900 ms"] --> A["Span: check login, 60 ms"]
    ROOT --> B["Span: get prices, 120 ms"]
    ROOT --> C["Span: reserve stock, 650 ms"]
    ROOT --> D["Span: take payment, 40 ms"]
    C --> C1["Span: database query, 600 ms"]
    B --> B1["Span: cache lookup, 5 ms"]
        

Look at the figure. The whole checkout took 900 milliseconds. Login, prices, and payment were quick. Reserving stock took 650 milliseconds, and nearly all of that was a single database query. Without the tree, you would only know that checkout was slow. With it, you know the database query inside the stock service is the place to look.

Context

Here is the tricky part. The programs are separate and do not share memory. How does the stock service know that its work belongs to a checkout that began in a different program? The answer is context. Context is a small package of identifiers, mainly the trace ID and the ID of the calling span, that travels along with the request. When one program calls another, it attaches this package to the call. The receiving program reads it, and creates its own span as a child of the sender’s span.

Passing this package along is called context propagation. It is the heart of distributed tracing. If any program in the chain forgets to pass it on, the trace breaks into two separate pieces.

Before tracing inside a cluster, here is a short guide to the Kubernetes words that will appear again and again.

Containers, pods, and nodes

A container is a small box that holds a program and everything it needs to run. A pod is the smallest thing Kubernetes runs, and it holds one or more containers that work closely together. A node is a computer, real or virtual, that runs pods. A cluster is a group of nodes managed together.

Deployments and services

A Deployment tells Kubernetes to keep a chosen number of identical pods running. If one crashes, a new one is started. A Service gives a group of pods a single stable name and address, so other programs can find them even when the pods behind the name change. When the checkout program wants the stock program, it calls the stock Service by name, and Kubernetes sends the call to one of the healthy stock pods.

Namespaces, ConfigMaps, and environment variables

A namespace is a named area inside the cluster that keeps related things together. A ConfigMap holds settings as text, and pods can read them when they start. An environment variable is a named setting handed to a program when it starts. These matter for tracing, because a program usually learns where to send its trace data, and what to call itself, through environment variables and settings supplied by Kubernetes.

DaemonSets and sidecars

A DaemonSet makes sure that one copy of a pod runs on every node. It is often used for helper programs that must be near everything else, such as collectors of data. A sidecar is a helper container that runs in the same pod as the main container, sharing its network. Sidecars are used to add features, such as traffic handling or data collection, without changing the main program.

Why Kubernetes makes tracing both harder and easier

It makes tracing harder because pods appear and disappear, so traces must be sent out quickly before a pod vanishes, and pod names change constantly. It makes tracing easier because Kubernetes gives every program a standard way to find its neighbours, to receive settings, and to run helper pods on every node. Those standard features are exactly what a tracing system needs.

A span is more than a name and two times. It carries extra details that make a trace useful when you are searching for a problem at three in the morning.

The parts of a span

Span kinds

When one program calls another, two spans are usually created. The caller makes a client span that measures how long it waited. The receiver makes a server span that measures how long it worked. The difference between the two is the time spent travelling over the network and waiting in queues. That gap can be surprisingly large and is often the clue to a hidden problem.

When work is passed through a message queue, the sender makes a producer span and the receiver makes a consumer span. Steps that stay inside one program are called internal spans.

Attributes: the labels you can search by

Attributes are pairs of a name and a value attached to a span. Examples are the web address called, the method used, the response code, the name of the database, or the name of the customer tier. They are what let you ask questions like “show me all slow requests to the payment service that failed for the premium tier.”

The tracing community has agreed on common names for many attributes, known as semantic conventions. Using the standard names means that tools from different makers understand your data, and your teams do not each invent their own words for the same idea.

Resource attributes: who made the span

Some details describe the program that made the span rather than the work itself. These are called resource attributes. They include the service name, the version, the pod name, the namespace, and the node. Kubernetes supplies most of them. They let you say, “this slow span came from version 2.3 of the stock service, running in pod number four, on this node.”

Everything in tracing depends on the trace ID reaching the next program. Here is how that happens, and why a standard matters.

How context travels

Programs usually talk to each other over the web, using a protocol called HTTP, or through a faster protocol called gRPC. Both allow small extra pieces of information, called headers, to be added to each call. Tracing uses a header to carry the context. The calling program adds it. The receiving program reads it, and creates its own span linked to the caller’s span. It then adds the header again on every call it makes to others.

A shared standard

In the past, different tracing tools used different header names and formats. A program using one tool could not pass context to a program using another. Today, most systems follow a standard from the World Wide Web Consortium, called W3C Trace Context. It defines a header named traceparent, which carries the trace ID, the parent span ID, and a flag telling whether this trace is being recorded. A second header can carry extra vendor-specific details.

Older formats still exist, such as the B3 headers from the Zipkin project. Modern tools can usually read and write several formats at once, which helps when a system is halfway through a move.

Baggage

Besides the identifiers, context can carry small extra facts called baggage, such as the customer tier or the region. Baggage travels to every program down the chain, so each can use it in its own spans. It is powerful, but it is also sent on every call, and it may leave your system when you call outside services. Use it sparingly and never for anything private.

Where context gets lost

Context is most often lost in a few predictable places. Message queues and background jobs are common, because the work does not run inside the original call. Programs that start their own worker threads may forget to carry context across. Proxies, gateways, and load balancers may drop unknown headers. And any program that has no tracing at all simply does not pass the header on.

Programs do not create spans by themselves. Someone, or something, must add the tracing. This is called instrumentation.

What instrumentation means

Instrumenting a program means adding the pieces that create spans, pass context, and send the results out. Think of it as fitting a program with sensors. There are two broad ways to do it.

Most teams begin with automatic instrumentation to get broad coverage, then add manual spans in the few places where the business logic matters most.

OpenTelemetry

OpenTelemetry, often written OTel, is the open standard and set of tools for collecting traces, metrics, and logs. It is a project of the Cloud Native Computing Foundation, and it is now the common way to instrument programs. It offers three main things. First, a specification that defines what spans, attributes, and context look like. Second, libraries called SDKs and APIs for many programming languages. Third, a program called the Collector that receives, processes, and forwards telemetry data.

The great benefit is freedom. If your programs speak OpenTelemetry, you can send the data to almost any tracing backend, and change the backend later without touching your applications. It replaced older projects called OpenTracing and OpenCensus, which were merged into it.

flowchart LR
    APP["Application with OpenTelemetry SDK"] -->|"OTLP"| COL["OpenTelemetry Collector"]
    COL --> BE1["Tracing backend, such as Jaeger or Tempo"]
    COL --> BE2["Another tool or vendor service"]
    BE1 --> UI["Trace viewer for people"]
        

Read the figure from left to right. The application uses the OpenTelemetry SDK to create spans. It sends them, using a standard protocol called OTLP, to the Collector. The Collector can process the data and forward it to one or several backends. People then use a trace viewer to read the results.

Instrumenting in Kubernetes

Kubernetes adds a helpful option here. An OpenTelemetry Operator can be installed in the cluster. An operator is a helper program that manages other programs by watching special Kubernetes objects. With it, you can add a simple note, called an annotation, to a Deployment, saying that its pods should be automatically instrumented. The operator then injects the instrumentation into new pods when they start, and sets the settings that tell them where to send data. This lets a platform team turn on tracing for many services without asking every developer to change code.

A working tracing setup has a handful of parts. Each has one clear job, and knowing them makes the whole picture simple.

The main parts

Why two layers of collectors

You could send spans straight from every application to storage. That works for a tiny system, but it has problems. Every application would need the storage address and any secret needed to reach it. Any slowdown in storage would slow the applications. And there would be nowhere to apply shared rules.

Adding a Collector in the middle solves these. Applications send to a nearby agent and move on quickly. The agent adds details such as the pod name and namespace, batches spans together, and forwards them. A central gateway can then make decisions that need a wider view, such as keeping all traces that contain an error. Only the gateway needs the credentials for storage.

The Collector’s own pieces

Inside every Collector are three kinds of building blocks that are joined into pipelines. Receivers accept data, for example in the OTLP format. Processors change data on the way through, for example by batching, adding labels, or dropping items. Exporters send data out to another system. A pipeline is simply a chosen list of receivers, processors, and exporters for one type of data.

Choosing a backend

Let us follow one customer click through three services and see exactly where each span and each header comes from.

The journey

A customer clicks “buy.” The click reaches the front door of the cluster, a gateway program. It has no trace yet, so it starts one and creates the root span with a new trace ID. It then calls the checkout service and adds the context header. The checkout service reads the header, creates its own server span as a child, and calls the stock service, again adding the header. The stock service creates its span, asks its database, and replies. Each program finishes its span when its work is done, and hands the finished span to the nearby agent.

sequenceDiagram
    participant U as Customer
    participant G as Gateway
    participant C as Checkout service
    participant S as Stock service
    participant A as Agent collector
    U->>G: Click buy
    G->>G: Start trace and root span
    G->>C: Call with context header
    C->>C: Create child span
    C->>S: Call with context header
    S->>S: Create child span and query database
    S-->>C: Reply
    C-->>G: Reply
    G-->>U: Answer
    S->>A: Send finished span
    C->>A: Send finished span
    G->>A: Send finished span
        

Notice two separate flows. The request itself moves along the chain, carrying the context. Meanwhile, each program sends its finished spans, quietly and separately, to the agent. The spans are not returned in the reply. This keeps tracing away from the customer’s response path, so a problem in the tracing system does not have to slow the customer.

What happens inside one program

Inside a program, the SDK keeps track of the current span, usually in a small holder that follows the running work. When the program starts a new piece of work, the SDK creates a new span whose parent is the current one. When that work ends, the SDK closes the span, records the end time, and places it in a queue. A background helper takes spans from the queue, groups them into batches, and sends them to the collector. Batching keeps the cost per span low.

What the collector does

When spans arrive at an agent, a processor adds Kubernetes details. It looks at the source of the connection, asks the Kubernetes API which pod that is, and attaches the pod name, namespace, node, and labels to the spans. Another processor batches them. The agent then forwards them to the gateway. The gateway may hold spans in memory for a short time, because it must wait for all the spans of a trace to arrive before it can decide whether to keep the trace.

How a trace is assembled

Spans reach the backend at different times and from different places. The backend groups them by trace ID and uses the parent IDs to rebuild the tree. A span whose parent has not yet arrived is temporarily an orphan, and it joins the tree when the parent shows up. If a parent never arrives, because a service was not instrumented or a span was lost, the viewer shows a gap or a missing parent. This is a useful clue that something upstream is not reporting.

The role of a service mesh

A service mesh is a layer of small network helpers, often sidecars, that handle traffic between services. Because they see every call, they can create spans for the network part of each hop, even for programs with no instrumentation. This gives a rough picture of who calls whom and how long each call took. However, a mesh cannot see inside a program, and it cannot pass context from an incoming call to an outgoing one unless the program itself copies the headers. So a mesh helps, but does not replace instrumentation.

A busy system produces far more traces than anyone can store or read. Sampling means deciding which ones to keep.

Why not keep everything

Imagine a service that handles ten thousand requests each second. Each request creates a trace with maybe twenty spans. That is two hundred thousand spans every second, around the clock. Storing and searching all of it would be very costly, and most of it would be nearly identical successful requests that nobody will ever look at. Sampling reduces the volume while keeping enough to learn from.

Head sampling

In head sampling, the decision is made at the very start of the trace, when the root span is created. For example, the system might decide to keep one trace in every hundred. The decision is written into the context flags, so every later program follows it. This is cheap and simple, because unwanted traces are never recorded. The weakness is that the decision is made before anyone knows how the trace will turn out. An interesting failure has only a one in a hundred chance of being kept.

Tail sampling

In tail sampling, the decision is made at the end, after the whole trace has been collected. The system can then keep traces that had errors, or that took longer than a chosen time, and keep only a small share of the rest. This keeps the most useful traces. The price is complexity. All the spans of a trace must reach the same collector, so it can see the whole trace, and that collector must hold spans in memory while it waits. This is usually done in the gateway layer.

flowchart TD
    START["Request arrives"] --> H{"Head sampling: keep this one?"}
    H -->|"No"| DROP1["Not recorded"]
    H -->|"Yes"| REC["Spans recorded and sent"]
    REC --> T{"Tail sampling at gateway"}
    T -->|"Trace has an error"| KEEP1["Keep"]
    T -->|"Trace is very slow"| KEEP2["Keep"]
    T -->|"Ordinary and fast"| PROB{"Random small share?"}
    PROB -->|"Chosen"| KEEP3["Keep"]
    PROB -->|"Not chosen"| DROP2["Discard"]
        

The figure shows both methods working together. Head sampling can cheaply throw away a large share at the start. Tail sampling then inspects what remains and always keeps the errors and the slow ones, while keeping only a small random share of ordinary traces.

Practical rules of thumb

Keep all traces in small systems and in test environments, because volume is low and the detail is valuable. In larger systems, use a low fixed share at the head, and add tail rules at the gateway that keep every trace with an error and every trace slower than a threshold. Always record the sampling rate, so that when you count traces in the viewer you know that each one stands for many.

Follow one trace from the first click to its final deletion.

What happens when things fail

If the agent is down, the SDK’s queue fills up. Well-built SDKs drop the oldest spans rather than block the application, because tracing must never break the thing it observes. If the gateway is overloaded, agents may buffer briefly and then drop data. If the backend is down, the gateway may keep data in a limited queue. In every case, the goal is the same: protect the customer’s request first, and accept that some trace data may be lost.

Choosing retention

Each signal answers a different question. The magic happens when you can jump between them in a single click.

Three signals, three roles

Logs that carry the trace ID

The simplest and most valuable link is to write the trace ID and span ID into every log line. Most logging libraries can do this automatically when tracing is set up. Then, from a slow trace, you can search your logs for that trace ID and see every log line from every service for that one request. Conversely, from an error log, you can copy the trace ID and open the whole trace.

Exemplars: from a number to a trace

Metrics are averages and totals, so they hide individual requests. An exemplar is a small link stored alongside a metric value that points to one example trace. A chart of slow requests can then show dots, and clicking a dot opens a real trace that contributed to that point. This is one of the fastest ways to move from noticing a spike to seeing its cause.

Metrics made from traces

Because spans record durations and outcomes, they can be turned into metrics, such as the rate, error count, and duration of each service and each call between services. Some backends and collectors do this for you. It gives a complete map of requests, errors, and latency between services, with no extra instrumentation.

flowchart LR
    M["Metric chart shows a spike"] -->|"exemplar link"| T["Trace shows the slow step"]
    T -->|"trace ID search"| L["Logs show the exact error"]
    L -->|"trace ID in log line"| T
    T -->|"span durations become"| M2["Service level metrics"]
        

The figure shows a typical investigation. A person sees a spike on a chart, clicks an example dot to open a trace, sees which span is slow, and follows the trace ID into the logs to read the exact error message. The same trace also feeds summary metrics for services.

Traces describe exactly how your system works and what your users do. Treat them as sensitive data.

What traces might reveal

Traces can contain web addresses with private values inside, database statements, error messages, customer identifiers, and internal service names. Attackers could use this to learn how your system is built. Careless attributes might also expose private data such as emails, names, or tokens. In many countries, laws about personal data apply to traces just as they apply to databases.

Controls to put in place

Danger from outside headers

The public can send requests with their own trace headers. If your system trusts them blindly, an outsider could push their trace IDs into your data, force your system to record every request, or use baggage to inject values. At the edge of your system, decide whether to accept incoming context or start a new trace, and cap the size of any baggage you accept.

Tracing is powerful, but it has real costs. Knowing both sides helps you invest where it pays off.

The trade-offs in plain words

When tracing is worth it

Tracing pays off most when requests cross many services, when teams are separate and cannot see each other’s code, and when problems are about slowness or partial failure rather than a single crash. If you have one or two services, good logs and metrics may be enough for now. As the number of services grows, the value of tracing grows faster than its cost.

Patterns are arrangements that many teams have found to work. Anti-patterns are habits that seem sensible but cause trouble.

Helpful patterns

Anti-patterns to avoid

A short list of habits that keep tracing useful, and the mistakes that most often spoil it.

Best practices

Common mistakes

A gentle start plan

In the first week, install the collector agents and the backend, and turn on automatic instrumentation for two or three services on your main customer journey. In the second week, fix any broken context, add trace IDs to logs, and build a first dashboard link. In the third week, add a few manual spans, choose a sampling plan, and add redaction rules. Then review with the team and grow from there.

A tracing system also needs watching. If it fails quietly, you will only find out when you need it most.

Numbers to watch

Collectors and backends report their own metrics. A few tell you most of what you need. Watch how many spans are received and how many are sent onward. Watch how many are dropped or refused, and why. Watch the size of the internal queues, the memory use of collectors, and how long storage takes to accept data. Watch also how much the query service is used and how slow it is.

Alerts worth having

Warn when the number of spans received drops suddenly, which often means a service lost its instrumentation or a collector is down. Warn when drops or refusals rise, when queues stay nearly full, and when collector memory approaches its limit. Warn also when the storage backend is failing to write. These alerts should go through your normal alert system, not through the tracing system itself.

Troubleshooting the usual suspects

Scaling the pipeline

As traffic grows, add more agent and gateway replicas and more storage capacity. Keep the routing by trace ID so that tail sampling still works. Lower the cost by sampling more, shortening retention, or removing spans and attributes that nobody uses. Review the top contributors to volume every few months, since one busy service can account for most of the data.

Three short stories show the ideas at work. The details are typical rather than taken from any one company.

Example 1: The mysterious slow checkout

Example 2: A hidden dependency

Example 3: Retries that hide a failure

What the stories share

In each story, the answer came from looking at real requests instead of averages and assumptions. Teams started with a real complaint, opened a small number of representative traces, found a single surprising step, and confirmed the fix by looking at traces again afterwards.

Short answers to questions that beginners often ask.

If you have only a few services, maybe not yet. As the number of services grows, tracing becomes the fastest way to see the path and timing of a single request across them. It complements, rather than replaces, logs and metrics.

A trace is the whole journey of one request. A span is one step within that journey. A trace is made of many spans arranged in a tree.

OpenTelemetry is an open standard and set of tools for creating and collecting traces, metrics, and logs. You do not have to use it, but it is the widely supported choice and keeps you free from being tied to one vendor.

A little. Creating spans and sending them takes some processor time and memory. With sensible sampling, batching, and limits, the cost is usually small, but you should measure it on your own workloads.

Almost always because context was not passed at some hop. Common causes are message queues, background threads, gateways that remove headers, services with no tracing, and mixed header formats.

Head sampling decides at the start of the request, cheaply, before it is known how it will turn out. Tail sampling decides after the whole trace is collected, so it can keep errors and slow traces, but it needs more memory and careful routing.

A DaemonSet, one per node, is simpler and uses fewer resources for most teams. A sidecar gives stronger separation per application and finer control, but costs more resources across many pods. Many teams start with a DaemonSet.

Partly. A mesh can create spans for network calls, giving a rough map. But it cannot see inside your programs, and your programs must still copy the trace headers from incoming to outgoing calls for the traces to connect.

Days to a couple of weeks is common, since most investigations happen soon after an event. Keep error and slow traces a little longer, and copy important ones into incident reports.

It can be, with care. Avoid personal data in attributes, redact risky values in the collector, encrypt connections, control access, and set short retention. Check the rules that apply in your country and industry.

An exemplar is a link attached to a metric value that points to an example trace. It lets you click from a spike on a chart straight to a real request that contributed to it.

Pick one important customer journey, turn on automatic instrumentation for the services on it, send data through a collector to a simple backend, and open a few traces with your team. Fix broken context, add trace IDs to logs, and grow from there.

A last look at the whole picture, in plain words.

Modern applications are built from many small services, and a single customer click may pass through a dozen of them. When something is slow or broken, metrics can tell you that there is a problem, and logs can tell you what a single program said, but only a trace shows the whole path of one request, step by step, with times.

A trace is a tree of spans. Each span is one piece of work, with a start, a duration, and helpful details. The trace ID and the parent link tie them together, and context propagation carries these identifiers from one program to the next. Lose the context, and the story breaks. Use a shared standard, and everything connects.

In Kubernetes, programs are instrumented with OpenTelemetry, automatically or by hand, and send spans to nearby agent collectors. A central gateway enriches, samples, and cleans the data, and forwards it to a backend where people can search and read it. Sampling keeps costs under control while preserving errors and slow requests. Linking traces to logs and metrics turns a spike on a chart into a cause in a few clicks. Finally, treat the data with care, and watch the pipeline itself, so it is ready when you need it.