The short answer first, then the details that make it useful.

A span represents a single unit of work done by a piece of software. Handling a web request is a unit of work. So is reading from a database, calling another service, resizing an image, or sending a message to a queue. For each such piece of work, software can create a span that records what the work was, when it began, how long it took, and whether it succeeded.

Spans are part of OpenTelemetry, often shortened to OTel. OpenTelemetry is a free, open source framework, hosted by the Cloud Native Computing Foundation, that gives developers one standard way to create and collect information about how their software behaves. That information is called telemetry, a word that simply means measurements sent from a distance. OpenTelemetry covers three kinds of telemetry: traces, metrics, and logs. Spans belong to the first kind.

A trace is the complete story of one request as it moves through a system. A trace is made of spans, just as a book is made of pages. One span tells you about one step. The whole trace tells you about the whole journey.

What a span tells you

• What happened. Every span has a name, such as “GET /orders” or “charge credit card.”
• When it happened. It records the exact moment the work began and the exact moment it finished.
• How long it took. The duration is simply the finish time minus the start time.
• Whether it worked. A status says if the operation succeeded or failed.
• Extra details. Notes called attributes can say which user, which product, or which database was involved.
• Where it fits. A span knows its parent, which is the span that caused it, so many spans can be drawn as a tree.

What a span is not

A span is not a log line. A log is a single sentence about a moment in time, while a span covers a stretch of time with a beginning and an end. A span is also not a metric. A metric is a number summarizing many events, such as “1,200 requests per minute,” while a span describes one specific event in detail. Finally, a span is not the whole request. It is only one step of it. The whole request is the trace.

Why spans matter to a business

Spans are not only a tool for engineers. Slow pages cost money, because shoppers who wait too long often leave without buying. When a team can see precisely which step is slow, it can repair the right thing instead of guessing, and repairs arrive in hours instead of weeks. Support staff also benefit: when a customer reports a failed order, someone can search for that order’s trace and see exactly where it stopped.

Spans also help teams plan. By looking at many traces over a month, an architect can see which services receive the most calls, which calls are the slowest, and which dependencies are quietly becoming critical. That evidence turns arguments about what to improve next into simple, calm conversations backed by real numbers.

Every good idea begins as a response to a painful problem. Spans are no exception.

In the past, most applications were one large program running on one machine. When something went wrong, a developer opened the log file and read it from top to bottom. That worked because everything happened in one place. Then companies began splitting their software into many small services spread across many computers. This style is called a distributed system, meaning the parts run in different places and talk to each other over a network.

Distributed systems brought a new headache. One customer request might touch twenty services, each writing its own separate log file on its own machine. Finding the slow or broken service meant searching twenty places and trying to guess which messages belonged to the same request. Engineers needed a way to follow a single request across all those boundaries.

Google solved the problem inside its own walls and described the solution in a 2010 research paper about a system called Dapper. Dapper introduced the idea of recording each step of a request as a small timed record linked to its parent step. Those records are the ancestors of today’s spans. After the paper appeared, other companies built open source tools inspired by it.

A handful of words explain nearly everything in the tracing world. Let us meet them one at a time.

Before looking inside a span, it helps to understand the family of ideas that surround it. Do not worry about memorizing them. Read the cards once, and the later chapters will keep reminding you.

The tree shape of a trace

Because every span (except the root) points to its parent, a trace naturally forms a tree. The root is at the top, and each step that it triggers hangs beneath it. Siblings, meaning spans with the same parent, may happen one after another or at the same time. Drawing the tree is the first thing most tracing tools do, and it is the reason a trace is so easy to read.

Open a span like a toolbox and look at every compartment.

Every span, no matter which programming language created it, carries the same set of fields. That sameness is the entire point of a standard. A tool that understands one span can understand all of them.

Name

The name should describe a kind of operation, not one individual case. A good name for a web request is “GET /orders/{id},” where the part in braces is a placeholder. A bad name would be “GET /orders/48291,” because that creates a different name for every order and floods your tools with endless variations. Names with a small, predictable set of values are said to have low cardinality, and that is exactly what you want.

Attributes

Attributes are pairs of a key and a value that describe the work. For a database span, attributes might say which database system was used and the name of the table. For a web span, they might include the method, the route, and the response code. OpenTelemetry publishes semantic conventions, which are agreed names for common attributes, so that everyone calls the same thing by the same name. Following them means your dashboards and other people’s dashboards understand each other.

Events

An event is a message with a timestamp that happens at one moment inside a span. For example, during a checkout span you might record an event when a discount code was applied and another when stock was reserved. When a program catches an error, it can also record the error as an event on the span, with the message and the place where it occurred. Events are lighter than creating new spans, which makes them ideal for noteworthy moments that take no meaningful time.

Links

Sometimes a span is related to another span, but it is not its child. A link points to another span’s identity without creating a parent relationship. This is helpful when one job collects many incoming requests into a batch. The batch span can link back to every request span that contributed, even though those requests belong to different traces.

Status

Status says how the operation ended. It has three possible values. Unset is the default and means “nothing special to report.” Error means the operation failed. Ok means the developer deliberately confirmed success, which is rarely needed. Choosing the right one takes a little care, as the note below explains.

Resource and scope travel with the span

Two more pieces of information accompany every span even though developers rarely set them by hand. The resource describes the program that produced the span, with details such as the service name, its version, and the cluster or machine it runs on. The instrumentation scope names the library that recorded the span, for example a web framework or a database driver. Together they answer two practical questions during an investigation: which service made this span, and which piece of code made it? Setting the service name correctly is one of the most valuable five minutes you can spend, because without it every span in your tool carries the same unhelpful label of an unknown service.

Not every span plays the same role. The kind of a span tells tools how to draw the conversation between services.

Each span carries a kind, a label that describes its role in a conversation. Tools use the kind to draw arrows between services and to calculate things like network time. There are five kinds.

Client and server spans come in pairs. When service A calls service B and waits, A creates a client span and B creates a server span. The time on A’s client span is always a little longer than B’s server span, and that gap reveals the time spent on the network and in queues between them. Producer and consumer spans work the same way for messages, except the producer does not wait for the consumer.

The kind matters because tools make decisions from it. A monitoring product uses server spans to count the requests a service handled, and it uses client spans to draw an arrow showing which service called which. If you label every span as internal, your service map will show a collection of unconnected dots instead of a web of conversations. Choosing the right kind is therefore a small act of kindness toward everyone who will read your traces later.

flowchart TD
    A["Root span: GET checkout, kind Server"]
    B["Child span: validate cart, kind Internal"]
    C["Child span: call payment service, kind Client"]
    D["Span in payment service: POST pay, kind Server"]
    E["Child span: query orders table, kind Client"]
    F["Child span: publish order created, kind Producer"]
    G["Span in email service: send receipt, kind Consumer"]
    A --> B
    A --> C
    C --> D
    D --> E
    A --> F
    F -.->|message queue| G

Read the diagram from the top. The root span is the web request arriving at the store. It has three children: an internal step that validates the shopping cart, a client span calling the payment service, and a producer span that publishes a message saying the order was created. The payment service answers with its own server span, which in turn makes a database call. Later, the email service picks up the message and sends a receipt. The dotted line shows that a queue sits between the producer and the consumer, so the two spans are connected by identity but not by a waiting caller.

A span can only join a trace if it knows which trace to join. This chapter shows how that knowledge is passed along.

When service A calls service B over the network, B is a completely separate program. It has no way to peek into A’s memory. So how does B know that its work belongs to the same trace? The answer is context propagation: A attaches the span context to the outgoing request, and B reads it from the incoming request.

Most systems carry the context in an HTTP header, a small label attached to a web request. The widely used standard is the W3C Trace Context format, and its header is called traceparent. It packs four pieces of information into one line: a version number, the trace ID, the ID of the calling span (which will become the new span’s parent), and a few flags, including one that says whether this trace is being recorded. OpenTelemetry uses this format by default. Older formats, such as the B3 headers from Zipkin, are also supported for compatibility.

sequenceDiagram
    participant U as User Browser
    participant A as Store Service
    participant B as Payment Service
    participant D as Database
    U->>A: 1. Request arrives with no trace
    Note over A: 2. Start root span and make trace ID
    A->>B: 3. Call with traceparent header
    Note over B: 4. Read header and start child span
    B->>D: 5. Query with new child span
    D-->>B: 6. Return rows
    B-->>A: 7. Return payment result
    Note over A: 8. End spans and send to collector
    A-->>U: 9. Send the page

Here is the story in order. In step one, the browser sends a request that carries no trace information, so the store service starts a brand new trace and creates the root span in step two. In step three it calls the payment service and adds the traceparent header. The payment service reads that header in step four and starts a child span with the same trace ID. When it queries the database in step five, it repeats the idea by creating another child span. When the answers flow back, each service ends its spans and sends them away for storage and display.

Using a shared header format matters because the services in a large company are rarely written in the same language or by the same team. One service may be written in Java, another in Go, and a third in Python. When all of them read and write the same traceparent header, a trace passes through every one of them without any translation. This is a quiet but important reason why standards win over private conventions.

Two things that are not the same

Do not confuse the trace context with baggage. The trace context carries only identity. Baggage is a separate bundle of key and value pairs, such as a customer tier or a test flag, that you deliberately want every downstream service to see. Baggage is powerful, but it travels with every call, so it should be small and must never carry secrets.

Crossing queues and background jobs

The same idea works when a message waits in a queue. The producer writes the span context into the message’s metadata, and the consumer reads it later. The consumer can either make its span a child of the producer span, or create a fresh trace and add a link back to the producer. Links are usually better for large batches, because a single consumer may handle hundreds of messages at once, and a tree with hundreds of parents would make no sense.

A span is born, does its job, and is sent away. Following that life explains most of the surprises people meet in practice.

Every span passes through the same stages, in the same order. Understanding the order makes it easy to see where things can go wrong, such as a span that never appears in your tracing tool.

flowchart LR
    A["1. Start span and record start time"] --> B["2. Make it the active span"]
    B --> C["3. Add attributes and events"]
    C --> D{"4. Did the work fail?"}
    D -->|Yes| E["Record the error and set status to Error"]
    D -->|No| F["Leave status Unset or set Ok"]
    E --> G["5. End span and record end time"]
    F --> G
    G --> H["6. Span processor queues it"]
    H --> I["7. Exporter sends a batch"]
    I --> J["8. Collector or backend stores it"]

Stage by stage

Starting. When work begins, the program asks a tracer to start a span. A tracer is simply the object that creates spans for one library or one piece of code. If an active span already exists, the new span automatically becomes its child. If none exists, the new span becomes a root.

Making it active. The new span is placed into the current context, a small storage area that follows the running code. Any span created next, deeper in the same flow, discovers the active span and attaches itself beneath it. That is how a tree forms without developers passing parents around by hand.

Enriching. While the work runs, the program adds attributes and events. It might note the number of items in the cart or record the moment a retry began. These small notes cost very little and are the most valuable part of the span when someone investigates a problem later.

Handling failure. If something breaks, the code records the error as an event and sets the status to Error. Failing to do so means the span looks healthy even though the operation was not.

Ending. The program calls end, which stamps the finish time. After that moment the span is frozen. Changes made to it afterward are ignored, so all notes must be added before the end.

Processing and exporting. A span processor receives every finished span. The usual choice is a batch processor, which collects spans in a queue and sends them in groups every few seconds. This is far cheaper than sending each span alone. An exporter then ships each batch to a collector or a backend.

A note about clocks

Every span reads time from the clock of the machine that created it. Inside one machine, durations are very trustworthy, because both the start and end come from the same clock. Across machines, clocks can disagree by a few milliseconds, so a child span on another computer may appear to start slightly before its parent or end slightly after it. This is called clock skew. It is normal and does not mean the trace is broken. Rely on the parent and child relationships for the structure of a trace, and on each span’s own duration for how long it took.

A span is only useful once someone can see it. Here is the road it travels from your code to your screen.

OpenTelemetry splits its work into clearly separated parts. This keeps your application code simple and lets you swap tools without rewriting anything.

flowchart LR
    subgraph APP["Your application"]
        CODE["Your code and libraries"]
        API["OpenTelemetry API"]
        SDK["OpenTelemetry SDK"]
        PROC["Span processor and sampler"]
        EXP["OTLP exporter"]
    end
    subgraph COL["OpenTelemetry Collector"]
        REC["Receiver"]
        CPR["Processors: batch, filter, redact"]
        CEX["Exporters"]
    end
    subgraph BACK["Backends"]
        B1["Jaeger"]
        B2["Zipkin"]
        B3["Grafana Tempo"]
    end
    CODE --> API
    API --> SDK
    SDK --> PROC
    PROC --> EXP
    EXP -->|OTLP| REC
    REC --> CPR
    CPR --> CEX
    CEX --> B1
    CEX --> B2
    CEX --> B3

Follow the arrows. Your code calls the API. The SDK, configured once at start-up, creates the spans and decides whether to keep each one. Finished spans enter the processor, and the exporter sends them as OTLP to the Collector. The Collector can add tags, remove sensitive details, group spans into efficient batches, and then send copies to as many backends as you like. Your application never needs to know which backend is in use.

It is also worth knowing what happens when things go wrong. The batch processor keeps finished spans in a queue of limited size. If the Collector or backend is unreachable for too long, the queue fills, and the oldest or newest spans are dropped so the application is never slowed down or crashed. This design protects your customers first and your telemetry second, which is exactly the right priority. It is also the reason why watching dropped-span counts is so important.

Recording every span of every request is wonderful and also expensive. Sampling is how teams find a sensible middle.

Imagine a busy service handling ten thousand requests per second. If each request creates fifteen spans, that is a hundred and fifty thousand spans every second. Storing and searching them all can cost more than running the service itself. Sampling means keeping only a portion of the traces and discarding the rest, while still leaving enough to answer your questions.

Head sampling

In head sampling, the decision is made at the very beginning of a trace, when the root span is created. A simple rule might say “keep one trace in ten.” The decision is written into the trace flags that travel in the traceparent header, so every downstream service follows the same choice. A sampler that respects the caller’s decision is often called parent-based. Head sampling is cheap and simple, but it has a weakness: it decides before knowing whether the request will turn out to be slow or broken, so it can throw away exactly the traces you wanted.

Tail sampling

In tail sampling, the decision is made after the whole trace has finished, usually inside the Collector. At that point rules can be much smarter, such as “keep every trace with an error, keep every trace slower than two seconds, and keep five percent of the rest.” The cost is that the Collector has to hold the spans of each trace in memory until the trace is complete, and all spans of one trace must reach the same Collector instance.

Choosing a starting point

Many teams begin by keeping a small percentage of traces, perhaps one in a hundred for very busy services and every trace for quiet ones. They then watch two things: whether the kept traces are enough to answer real questions, and whether the monthly bill is comfortable. If an investigation keeps failing because the right trace was thrown away, the rate goes up or a smarter rule is added. If the bill is too high, the rate goes down. Sampling is not a decision made once; it is a dial that you adjust as your system and your needs change.

Spans are rich in detail, and rich detail can contain things that should never leave the building.

Because spans describe real traffic, they can capture sensitive information by accident. A database span that records the full text of a query might include a customer’s email address. A web span that records every header might include a secret login token. Once such data lands in a tracing tool, many more people can see it than ever should.

Risks and remedies

Trusting the incoming header

Inside your own network, accepting the traceparent header from a trusted service is exactly what you want. At the edge, where requests arrive from the open internet, caution is wise. A stranger could send headers that force every request to be recorded, filling your storage and your bill. A common approach is to start a new trace at the public edge and store the outsider’s identity only as a link.

Retention and access

Two quiet questions deserve an answer before you launch. First, how long should spans be kept? Keeping traces for a couple of weeks is often enough for debugging, and shorter retention reduces both cost and exposure if something goes wrong. Second, who may read them? Treat your tracing tool like any other system that holds business data: give people only the access they need, and keep a record of who looked at what. These small policies cost almost nothing to set up and are painful to add after an incident.

Spans are one of three signals. They are strongest when the other two stand beside them.

OpenTelemetry offers three signals, and each answers a different question. Metrics tell you that something is wrong, quickly and cheaply. Traces, made of spans, tell you where it is wrong. Logs tell you why, in the words of the program itself. The three become far more powerful when they are connected.

A typical investigation

Watching the watchers

The pipeline that carries spans also needs monitoring. If it silently drops data, you will believe everything is fine when you are actually blind. Useful things to watch include the number of spans dropped because a queue was full, the number of failed exports, the size of the Collector’s memory, and the delay between a span ending and appearing in the backend. The Collector publishes metrics about itself for exactly this purpose.

Service maps and health at a glance

Because every client span names the service it called, tracing tools can draw a service map, a picture of boxes and arrows showing how your programs depend on one another. Colors or numbers on the arrows show how many calls flow along them and how many fail. For a newcomer joining a team, this map is often the first honest diagram of the system they have ever seen, since it is drawn from real traffic and not from someone’s memory.

Tracing with spans is a wonderful tool, and like any tool it has costs.

Common trade-offs in plain language

It helps to estimate the cost before you begin. Multiply the number of requests per second by the number of spans per request, then by the share you plan to keep after sampling. The result is the number of spans per second that your pipeline must carry and your backend must store. Doing this small piece of arithmetic early prevents unpleasant surprises, and it shows clearly why sampling and sensible span counts matter so much for busy services.

Patterns are tried-and-true recipes. Habits are the small routines that keep your traces useful instead of noisy.

Adding spans to a new service, step by step

Habits that pay off

• Name spans by what they do, not by who asked. Prefer “GET /orders/{id}” to a name with a real order number in it.
• Follow semantic conventions. Use the agreed attribute names for web calls, databases, and messaging, so every tool understands your data.
• Add useful attributes. Record things you would ask during an investigation, such as the type of customer plan, the number of items, or the region, but keep each value small.
• Record errors properly. When something fails, attach the error to the span and set its status to Error. Do not hide failures.
• Always end spans. Put the end call somewhere that runs even when an error occurs.
• Add service identity. Make sure every service sets its name and version in the resource so you can always tell who made a span.
• Test your traces. After adding instrumentation, send a test request and look at the tree. If it looks wrong to you, it will look wrong to the team during an outage.
• Review attributes like code. Ask in every review: could this value contain a secret or personal data?

Learning from other people’s mistakes is much cheaper than making your own.

An anti-pattern is a habit that looks sensible at first but causes trouble later. These four appear again and again in real tracing projects.

Myths and realities

Let us read a real-looking trace the way an engineer would.

Return to the online store from the beginning of this guide. A customer complained that checkout took almost half a second longer than normal. The tracing tool shows the following spans. Start and duration are measured in milliseconds from the moment the root span began.

Reading from the top, the whole request took 480 milliseconds. Validating the cart took only 20 and is not a problem. The payment call took 350 milliseconds, almost three quarters of the total. Inside it, the payment service spent 320 milliseconds working, so only about 30 milliseconds went to the network and waiting. Inside that, a single database query took 250 milliseconds. That one span is the bottleneck. The engineer opens it, reads the attributes, and sees that the query scanned an entire table because an index was missing. Adding the index fixes the problem. Without the span tree, the team might have spent days blaming the network, the payment company, or the web servers.

Where spans came from in real life

The whole story in a few simple ideas you can carry with you.