Before learning the idea, let us feel the problem it solves.

A quick reminder about traces and spans

A trace is the complete story of one request as it travels through your system. A span is one timed step in that story. When a customer places an order, there may be a span for the web gateway, a span for the order service, a span for the database query, and a span for the payment call. The spans are joined into a tree by a shared trace ID and by parent and child links.

A tree is not enough

Suppose a tool shows you only a tree of spans, each with a name and a duration. You can see that “order service” took 400 milliseconds and that inside it, “charge card” took 300. But many important questions are still unanswered.

• Which spans are the start of work in a new program? Where does one service end and the next begin?
• Which spans are calls to someone else? A call to the payment service is very different from a calculation inside your own code.
• How much of that 300 milliseconds was the network, and how much was the payment service thinking?
• Which services depend on which? Can you draw a map of the whole system from the data?
• What counts as a request that the service handled? This matters for counting requests per second and error rates.

The answer to all of these lies in a simple label on each span: its role in the conversation. That label is the span kind.

The short answer first, and then the pieces.

Span kind is a field on every span that describes the relationship between that span and its parent and children in the trace, especially whether the span is about a call that crosses from one program to another, and in which direction. It tells tools and people what role the span plays: asking, answering, sending, receiving, or just working internally.

Three simple facts

What span kind is not

• It is not the span name. The name says what the work is, such as “charge card.” The kind says what role the span plays.
• It is not a status. Status says whether the work succeeded. Kind says nothing about success.
• It is not an attribute you invent. It is a dedicated field with a fixed list of five values.
• It is not about the technology. A span for a database call and a span for a web call are both “client” spans, because both are the asking side of a conversation.

There are only five. Once you learn them, you know the whole list.

flowchart TD
    A["Span kind"] --> B["Synchronous conversations"]
    A --> C["Asynchronous conversations"]
    A --> D["No conversation"]
    B --> B1["Client: I am asking"]
    B --> B2["Server: I am answering"]
    C --> C1["Producer: I am sending a message"]
    C --> C2["Consumer: I am processing a message"]
    D --> D1["Internal: I am working inside my own program"]
    

The most common pair. One side asks, the other answers, and the caller waits.

In a synchronous conversation, the caller sends a request and waits for the reply before moving on. Web requests, database queries and remote procedure calls are all like this. Each side writes down its own side of the story in its own span.

sequenceDiagram
    participant O as Order Service
    participant P as Payment Service
    O->>O: Start CLIENT span for the call to payment
    O->>P: Request with trace identity in headers
    P->>P: Start SERVER span as child of the client span
    P->>P: Do the work
    P-->>O: Response
    P->>P: End SERVER span
    O->>O: End CLIENT span after the reply arrives
    

How the two spans are linked

When the order service makes its call, it starts a client span and passes that span’s identity in the request headers. The payment service reads those headers and starts a server span whose parent is the client span. So in the trace tree, the server span appears directly beneath the client span. The tree now clearly shows a call crossing from one program to another.

Why the two durations differ

The client span and the server span measure different things, and the difference is useful.

Where each kind is used

• Server spans: incoming web requests, incoming remote procedure calls, handling of an incoming network request of any kind.
• Client spans: outgoing web requests, remote procedure call clients, database queries, cache calls, calls to cloud services, and calls to any third-party service.

The quiet default kind, and the one most often misused.

An internal span describes work that happens entirely inside one program and does not cross a boundary to another program. Calculating a discount, validating a form, rendering a page, and sorting a list are all internal. This is also the default kind, which means that if you create a span and do not choose a kind, it will be internal.

Why the default is both helpful and dangerous

It is helpful because most spans you add by hand really are internal work. It is dangerous because if you manually create a span for something that really crosses a boundary, such as a call to another service or a database, and forget to set the kind, it quietly becomes internal. Nothing breaks. No error appears. But your service map loses an edge, your database disappears from the dependency list, and your latency analysis loses the client and server comparison.

When the sender does not wait for the receiver, the story changes.

In an asynchronous conversation, the sender drops off a message and moves on. The receiver picks it up later, perhaps seconds, minutes or hours afterwards. Message queues, event streams and background job systems all work this way.

sequenceDiagram
    participant O as Order Service
    participant Q as Message Queue
    participant E as Email Worker
    O->>O: Start PRODUCER span for send message
    O->>Q: Put message with trace identity in message headers
    O->>O: End PRODUCER span and continue with other work
    Note over Q,E: Time passes. Seconds, minutes or hours.
    E->>Q: Pick up message
    E->>E: Start CONSUMER span using the trace identity from the message
    E->>E: Process the message
    E->>E: End CONSUMER span
    

The key difference from client and server

With client and server spans, the client span contains the server span in time, because the client waits. With producer and consumer spans, the producer span typically ends before the consumer span even starts. They are connected by the trace identity carried inside the message headers, but they do not overlap in time. This is why the two pairs are separate kinds: they tell tools whether to expect nesting in time or a gap.

Producer

A producer span covers the act of creating and handing off the message. It is usually short, since it ends when the message has been accepted by the queue or broker.

Consumer

A consumer span covers the processing of the message. Its parent may be the producer span, giving one connected trace across time, or it may be a fresh span connected through a link back to the producer, which is useful when one consumer processes many messages at once.

Real messaging systems have several steps, and each can get its own kind.

Messaging is richer than a simple “send” and “receive.” A consumer program may wait for messages, fetch a batch, process each one, and then confirm to the broker that it has finished. The naming and kind guidance in the OpenTelemetry conventions separates these steps, and understanding them avoids confusion.

Batch processing and links

When a consumer receives many messages in one go, each message may belong to a different trace. It is not possible for one span to have many parents. The solution is a link: a connection from the processing span to each producer’s span, without being a parent and child relationship.

flowchart TD
    P1["Producer span of message 1 in trace A"] --> Q["Queue"]
    P2["Producer span of message 2 in trace B"] --> Q
    P3["Producer span of message 3 in trace C"] --> Q
    Q --> R["Receive span of kind client fetches a batch"]
    R --> C["Consumer span processes the batch"]
    C -.->|"link"| P1
    C -.->|"link"| P2
    C -.->|"link"| P3
    

Five questions are enough to pick correctly almost every time.

flowchart TD
    Q["What does this span cover?"] --> A{"Does the work cross to or from another program?"}
    A -->|No| I["Internal"]
    A -->|Yes| B{"Is the conversation synchronous, meaning the caller waits?"}
    B -->|Yes| C{"Am I the one asking or the one answering?"}
    C -->|Asking| CL["Client"]
    C -->|Answering| SV["Server"]
    B -->|No| D{"Am I sending the message or processing it?"}
    D -->|Sending| PR["Producer"]
    D -->|Processing| CO["Consumer"]
    

Common situations and the right kind

A tiny label unlocks several of the most useful views in an observability tool.

Service maps and dependency graphs

A service map is a picture with boxes for services and arrows for calls between them. Tools build it by looking for pairs: a client span in one service and a server span in another, joined by parent and child. Producer and consumer pairs give arrows for message flows. A client span with no matching server span, such as one calling a database or a third-party service, becomes an arrow to an outside dependency. Without correct kinds, these maps are empty or wrong.

flowchart LR
    G["Gateway"] -->|"client to server"| O["Order Service"]
    O -->|"client to server"| P["Payment Service"]
    O -->|"client only"| D["Database"]
    O -->|"producer to consumer"| E["Email Worker"]
    P -->|"client only"| X["External Card Provider"]
    

Request, error and duration metrics

The standard health measures for a service are the rate of requests, the number of errors and the duration of requests. To compute them from spans, tools usually count server spans, because each server span represents one request that the service handled. A tool that generates metrics from spans, such as the span metrics feature in the Collector, records the span kind as a label, so that you can look at server-side numbers and client-side numbers separately.

Latency breakdown

As we saw earlier, comparing the duration of a client span with its server span reveals how much time was spent in the network or queues. Tools can show this automatically, but only when the kinds are right.

Dependencies and external calls

Client spans to databases, caches and third-party services let you answer, “Which of my dependencies is slowest?” and “How much time do we spend waiting on that vendor?” This is vital for finding the real cause of slowness and for holding vendors to their promises.

Sampling and filtering

The kind is given to the sampler at span creation, so rules can treat server spans differently from internal ones. In the Collector, filters and rules can select data by kind, for example to drop noisy internal spans while keeping all server and client spans.

The same event can mean “failure” on one side and “normal” on the other. The kind explains why.

Imagine a web client asks a server for a page that does not exist. The server replies with the standard “not found” code, which is in the 400 range. Is that an error?

• From the point of view of the client, yes. It asked for something and did not get it. The client span should be marked as an error.
• From the point of view of the server, no. The server did exactly its job: it correctly told the caller that the page does not exist. Nothing is wrong with the server. The server span should be left without an error status.

The OpenTelemetry conventions for web requests say precisely this. For codes in the 400 range, the status is set to error on client spans and left unset on server spans. For codes in the 500 range, both sides treat it as an error, since the server itself failed.

The kind also guides which details a span should carry.

The semantic conventions, which are the shared naming rules of OpenTelemetry, describe different expected attributes for different kinds. The idea is natural: the asking side and the answering side know different things.

The route detail on server spans

A good example is the matched route. On a server span, the route pattern, such as “orders by ID”, is recorded, not the raw address with the actual ID. Using the pattern keeps names and groupings tidy, so that millions of different order IDs do not each become a separate group in your tools. Client spans cannot know this pattern, since it is decided by the server.

A look behind the scenes at a very small piece of data.

Set at creation, fixed forever

When code starts a span, it can pass the kind as an option. The SDK stores it in the span and never lets it change. There is no way to alter it later, because other parts of the system, such as the sampler, may already have used it.

Not carried between services

An important detail is that the kind is not part of the trace headers that pass from service to service. Those headers carry only the trace identity: the trace ID, the span ID, the flags and optional notes. So the kind of the server span is decided entirely by the receiving service, and the kind of the client span is decided entirely by the calling service. This is why both services must follow the same conventions: they cannot tell each other what role to take.

The protocol values

In the standard OpenTelemetry protocol, the kind is a small number: one for internal, two for server, three for client, four for producer and five for consumer. There is also a value zero meaning “unspecified,” which most tools treat the same as internal. You rarely see these numbers, since tools translate them into words, but it is helpful to know that “unspecified” and “internal” are usually treated alike.

Let us follow one customer order through a realistic system and watch each kind appear.

flowchart TD
    A["SERVER: Gateway handles the incoming web request"] --> B["CLIENT: Gateway calls the order service"]
    B --> C["SERVER: Order service handles the call"]
    C --> D["INTERNAL: Calculate total price"]
    C --> E["CLIENT: Order service runs a database insert"]
    C --> F["CLIENT: Order service calls the payment service"]
    F --> G["SERVER: Payment service handles the call"]
    G --> H["CLIENT: Payment service calls the card provider"]
    C --> I["PRODUCER: Order service sends an email message"]
    I -.->|"later"| J["CONSUMER: Email worker processes the message"]
    J --> K["CLIENT: Email worker calls the mail provider"]
    

Notice how reading only the kinds gives a skeleton of the whole system: where requests enter, where services call each other, which outside dependencies exist, and where work is handed off for later. That skeleton is what service maps and request metrics are built from.

Real systems have shapes that the simple examples do not show.

Gateways, proxies and meshes

A gateway or a proxy both receives and sends requests. It therefore has a server span for the request it receives, and a client span for the request it forwards. If a service mesh adds a small helper program, called a sidecar, beside each service, the sidecar may add its own server and client spans on each hop. A trace through a mesh can look long because each hop is described by several spans. This is normal.

Retries

If a client retries a failed call three times, there are two sensible shapes. The simplest is one client span for each attempt. A tidier shape wraps all attempts in one internal span, such as “call payment with retries,” and puts each attempt as a client span beneath it. This keeps the story readable and lets you see how many attempts were needed.

flowchart TD
    W["INTERNAL: Call payment with retries"] --> A1["CLIENT: Attempt 1 fails with a timeout"]
    W --> A2["CLIENT: Attempt 2 fails with a server error"]
    W --> A3["CLIENT: Attempt 3 succeeds"]
    A3 --> S["SERVER: Payment service handles attempt 3"]
    

Serverless functions

A serverless function is started by a trigger. If the trigger is a web request, the invocation is normally described like a server span. If the trigger is a message from a queue, it is like a consumer span. The conventions for this area can be detailed, so follow the guidance for your platform.

Background jobs and scheduled tasks

A job started by a timer, with no remote caller, has no incoming request to be a server for. Many teams use an internal span for the root of such a job and add client spans for the calls it makes. Some prefer to treat each job run as a server-like entry point. The most important rule is to be consistent across all your jobs, so that your tools treat them the same way.

Calls inside one program that look like services

If you split a program into modules that call each other in memory, these are internal calls, not client and server. Do not create client and server spans for in-process calls just because the design diagram calls them “services.” Kinds describe real boundaries, not drawing-board boundaries.

Streaming and long-lived connections

For long-lived connections such as streaming calls, the span for the whole call is still a client or server span, depending on side. Individual messages inside the stream are usually recorded as events on that span, rather than as separate spans, to avoid overwhelming the trace.

Calls to systems that cannot join the trace

When you call a database or a third-party service that does not propagate trace headers, your client span will have no server child. This is fine. It shows as an outside dependency. The absence of a server child is not a bug.

Most of the time, you do not have to think about it, until you write your own spans.

Auto-instrumentation chooses for you

Agents and instrumentation libraries for web frameworks, database drivers, message clients and similar tools set the kind automatically and follow the conventions. A web framework library creates server spans. An HTTP client library creates client spans. A database library creates client spans. This is one big reason auto-instrumentation gives clean maps with no effort.

Manual spans need your attention

When you create spans by hand, the default is internal. You must pass the kind yourself whenever your span represents a boundary-crossing operation. Doing so takes one extra option at span creation, but forgetting it silently produces incomplete maps.

Beware of duplicates

If an auto-instrumentation library already creates a client span for a web call, and you wrap the same call in your own client span, you will have two client spans stacked for one call. This confuses counts and makes traces look cluttered. If you want a wrapper that groups several calls, make it an internal span. Reserve client for the span that actually covers the single call.

The idea is older than OpenTelemetry. Knowing its origin explains some details.

Early tracing systems already needed to tell the two sides of a call apart. Zipkin, one of the first open-source tracing systems, recorded special timestamps for events such as “client sent,” “server received,” “server sent” and “client received.” It also used the same span ID for both sides of a call, so that a single span held both views. OpenTracing, an earlier standard for tracing code, introduced a tag called span.kind with values such as client and server, and later added producer and consumer for messaging.

When OpenTelemetry was formed from OpenTracing and OpenCensus, it kept the idea but made it a first-class field with five fixed values, and chose to give the client side and the server side separate spans with separate IDs, linked by parent and child. This made traces cleaner, because every span has exactly one owner and one timing view, and made it easier to combine data from many different programs.

Because tools such as Jaeger and Zipkin still exist, you may see kinds written in their own ways, such as a “span kind” tag or a field named “kind.” OpenTelemetry’s exporters translate between these styles, so the idea is the same everywhere.

An honest look at what span kind gives and what it asks of you.

Key trade-offs

The kind itself is harmless, but the details attached to each kind need care.

Because kind mistakes are silent, you must look for their symptoms.

• Empty or tiny service maps: usually means client and server spans are missing or all spans are internal.
• No dependencies shown for a database: database calls are probably not marked as client spans.
• Mismatched counts: if client spans calling a service far outnumber server spans in that service, trace headers may be lost on the way, or the server is not instrumented.
• Server spans with no client parent: normal at your public edge, but suspicious deep inside your system, where it may signal broken header passing.
• Share of internal spans: a very high share may hint that boundary-crossing spans are mislabelled.
• Duplicate client spans: two nested client spans for one call suggest a manual wrapper over an auto-instrumented call.
• Request metrics look wrong: if counts come from server spans, a missing server kind will leave metrics empty.
• Collector service graph health: if you generate graphs in the Collector, watch for unmatched or expired client and server pairs.

Common symptoms and likely causes

Habits worth copying and traps worth avoiding.

Good design patterns

Anti-patterns

A practical checklist for span kinds you can trust.

Best practices

• Ask the boundary question. Does this span cross to or from another program? If not, it is internal.
• Ask the waiting question. If the caller waits, use client and server. If not, use producer and consumer.
• Prefer automatic instrumentation for protocols. It sets kinds and attributes for you, following the conventions.
• Set the kind explicitly on manual boundary spans. Never rely on the default for calls out of your program.
• Make every entry point a server span. Every place where outside requests enter your service should start a server span.
• Pass context across every hop. Kinds only produce good maps when trace identity is carried through requests and messages.
• Use internal wrappers for grouping and retries. Keep client for the single call.
• Follow the conventions for status. Know the difference between client and server handling of 400-range responses.
• Check the map. After a release, look at the service map and make sure the expected arrows are present.
• Write a short internal guide. Document decisions for background jobs, batches and serverless, and apply them consistently.

Common mistakes beginners make

• Thinking span kind describes the technology, such as “database span,” when it describes the role in a conversation.
• Leaving manual spans at the default and wondering why maps are empty.
• Believing that the kind is sent in trace headers between services.
• Using client and server for in-process calls.
• Treating producer and consumer like client and server, and expecting them to nest in time.
• Wrapping auto-instrumented calls in manual client spans.
• Counting 404 responses as server errors.
• Forgetting that the kind cannot be changed after the span is created.

How span kind shows up in tools and platforms you may already know.

A typical journey for a growing company

Quick answers to questions beginners ask most.

If you remember only a few things, remember these.