A running program is a sealed box, and instrumentation is how we open a window in it.

When a program runs on a server far away, nobody can watch it work. It has no face and cannot tell you how it feels. Yet people need to know what it is doing. Why is the checkout slow today? Which step takes the longest? How many orders failed in the last ten minutes? Without any information coming out of the program, engineers can only guess.

The act of adding code or tools that make a program report on its own behavior is called instrumentation. The word comes from scientific instruments, such as thermometers and pressure gauges, that are attached to a machine so that people can read what is happening inside. The information that comes out is called telemetry, and it usually takes the form of traces, metrics and logs.

A short refresher on the big picture helps manual instrumentation find its place.

OpenTelemetry, often called OTel, is an open-source project that defines how software describes its own behavior. It is vendor neutral, which means no single monitoring company owns it. It handles three kinds of data, called signals, and provides libraries for many programming languages.

The API and the SDK

OpenTelemetry deliberately splits its libraries into two layers. The API is a thin set of calls that code uses to create telemetry, such as start a span or record a measurement. On its own, the API does almost nothing and costs almost nothing. The SDK is the real engine that processes the telemetry and exports it to a destination. The application owner installs and configures the SDK, and this split is what makes manual instrumentation safe for shared libraries.

It is the practice of writing OpenTelemetry API calls directly in your own code.

Let us define it carefully. Manual instrumentation in OpenTelemetry means that a developer deliberately adds code that uses the OpenTelemetry API to create telemetry at chosen places. The developer decides which operations deserve a span, which numbers deserve a metric, which events deserve a log, and which details deserve to be attached as attributes.

It is called manual because a human writes it, as opposed to automatic instrumentation, where a ready-made tool inserts the telemetry code for you. Manual instrumentation is the most precise and most flexible way to describe your program, because nobody knows your business logic better than you do.

Manual instrumentation is one of several ways to produce telemetry, and each has its place.

There are three broad styles of instrumentation in the OpenTelemetry world. Understanding all three makes it clear when manual work is worth the effort.

Automatic tools see the doors of your program, but not the rooms inside.

Automatic instrumentation is a wonderful starting point. It can tell you that a web request arrived, that it took two seconds, that it called a database and that the database took one second. But look at what it cannot see. It does not know that inside those two seconds your code calculated a discount, checked a loyalty tier, ran a fraud rule and chose a shipping provider. To the automatic tool, all of this is a black box between the incoming request and the outgoing call.

Automatic tools also do not know what is important to your business. They do not know that a certain customer type matters, that a certain feature flag is being tested, or that a certain step is the one the finance team cares about. Only you know that, and manual instrumentation is how you tell OpenTelemetry about it.

flowchart TD
  A["Incoming request span: created automatically"] --> B["Your business logic: invisible without manual work"]
  B --> C["Database call span: created automatically"]
  B --> M1["Manual span: calculate discount"]
  B --> M2["Manual span: run fraud rule"]
  B --> M3["Manual span: choose shipping provider"]
        

Manual effort pays off in a few clear situations.

A little one-time configuration is needed before any manual call does something useful.

The API alone only creates empty promises. For telemetry to actually leave the program, the application must set up the SDK once, usually when it starts. This setup is normally done in one central place, and individual pieces of code do not need to repeat it.

A span is one timed step, and manual tracing is the art of choosing which steps get one.

A trace is the story of one request, and a span is one chapter of that story. Each span has a name, a start time, an end time and a set of details. To create spans by hand, your code first asks the tracer provider for a tracer, which is named after the component or library that uses it. It then starts a span at the beginning of an operation and ends it when the operation finishes.

sequenceDiagram
  participant Code as Your Code
  participant T as Tracer
  participant S as Span
  Code->>T: Start span named calculate discount
  T-->>Code: Span, now the current span
  Code->>S: Add attributes and events
  Code->>Code: Do the work
  Code->>S: Set status if it failed
  Code->>S: End span
        

The parts of a span you control

Spans form a tree, and the library keeps track of where you are in it.

Spans are arranged in a family tree. The first span of a request is the root. Every span started while another span is active becomes its child. This nesting shows how one operation contains others, and it lets viewing tools draw the familiar waterfall picture of a request.

OpenTelemetry keeps a hidden record of the current span in what it calls the context. When you start a new span and make it current, any span you start afterward becomes its child automatically. When you end your span, the previous span becomes current again. As a result, you rarely need to pass parent information around by hand inside one program.

flowchart TD
  R["Root: handle checkout request"] --> A["Child: validate cart"]
  R --> B["Child: calculate discount"]
  B --> B1["Grandchild: load loyalty tier"]
  B --> B2["Grandchild: apply coupon rules"]
  R --> C["Child: charge payment"]
        

Attributes are the labels that turn a timeline into something you can question.

A span with only a name and a duration tells you how long something took. A span with attributes tells you which kind of case it was. Attributes are small named values, such as the customer tier, the payment method, the region or the number of items in a cart. With them, an engineer can ask questions like show me slow checkouts for premium customers in one region.

Follow the semantic conventions

OpenTelemetry publishes semantic conventions, which are agreed names and meanings for common attributes. For example, there are standard names for HTTP methods, database systems and messaging destinations. Using the shared names, rather than inventing your own, means that dashboards and tools built by others work with your data with no extra effort. For your own business concepts, create clear names and keep them consistent across teams, perhaps with a company prefix.

A trace that cannot show failures is only half useful.

When something goes wrong inside an operation, the span should say so. Manual instrumentation lets you do two things. You can set the span’s status to error, which flags the whole step as failed so that tools can highlight it and count it. And you can record the exception itself as an event on the span, which captures the error type, the message and often the place in the code where it happened.

Spans tell individual stories, and metrics tell the trend.

Manual instrumentation is not only about traces. You can also create instruments that record numbers over time. You ask the meter provider for a meter, then create the instruments you need, once, when the program starts, and reuse them everywhere. Each time something happens, your code records a value.

Metrics need careful attributes too

Each unique combination of attribute values creates a separate time series, which costs memory and money. Use attributes with a small, known set of values, such as payment method or region. Never use values that are different for every event, such as user identifiers or request identifiers. Those belong on spans or logs, not on metrics.

Logs join the story when they carry the identity of the request.

For logs, most teams do not call OpenTelemetry directly. They keep using the logging library they already know, and connect it to OpenTelemetry through a bridge. What manual instrumentation adds here is context: when a log is written inside an active span, the bridge stamps the log record with that span’s Trace ID and Span ID. Later, an engineer can jump from a slow span straight to the logs written during it.

You can also add structured attributes to your own log messages, such as the order number or the rule that fired, so that logs are searchable by detail rather than only by text. Keep the same care about secrets that applies to spans.

Spans in different services only join one trace if the context travels with the request.

Inside one program, the library tracks the current context for you. But when a request leaves your program and enters another one over the network, the memory is not shared. Something must carry the trace identity across the gap. This is called context propagation, and it works by writing the trace information into a header, a labeled piece of extra data that rides along with the call. The receiving program reads the header and continues the same trace.

sequenceDiagram
  participant A as Service A
  participant B as Service B
  Note over A: Manual span: prepare order
  A->>A: Inject trace context into outgoing headers
  A->>B: Call with tracing header
  B->>B: Extract context from incoming headers
  Note over B: Manual span: reserve stock, child of A's span
        

Automatic tools usually handle this for common web and messaging libraries. But when you use your own protocol, a custom queue or an unusual transport, you must perform the two steps manually: inject the context into the outgoing message, and extract it from the incoming message. Without both steps, the trace breaks into disconnected pieces.

The best setups use both, and they fit together naturally.

Because both styles use the same API and the same context, they cooperate without special effort. Automatic instrumentation creates the outer spans, such as the incoming web request and the outgoing database call. Your manual spans, started inside your code while the automatic span is current, become children of it automatically. The result is one complete tree, with the framework’s view at the edges and your business view in the middle.

flowchart TD
  A["Automatic: HTTP server span"] --> M1["Manual: validate cart"]
  A --> M2["Manual: calculate discount"]
  M2 --> D["Automatic: database client span"]
  A --> M3["Manual: choose shipping provider"]
  M3 --> H["Automatic: outgoing HTTP client span"]
        

More is not always better. The art lies in choosing the right amount.

New adopters often swing between two extremes. Some instrument nothing and remain blind. Others wrap every function in a span and drown in noise and cost. The wise path lies in the middle: instrument operations that are meaningful, that could be slow, that could fail, or that cross a boundary.

Good names are a gift to everyone who reads traces later.

Names are the first thing a person sees in a trace, so they deserve care. A good span name is short, describes the kind of work, and stays the same across requests. It names the operation, not the particular customer or order. Details that change from request to request belong in attributes.

Name your tracer and meter well

The tracer and meter are named after the component or library that owns them, and often carry a version. This name appears as the instrumentation scope, which lets engineers separate your code’s telemetry from that of frameworks and drivers. Create one tracer and one meter per component, and reuse them rather than recreating them for every call.

Good habits and bad habits repeat across teams.

Patterns that work

A helpful pattern is the wrapper helper: a small function that starts a span, runs your work, records errors and always ends the span, so developers cannot forget the cleanup. Another is instrumenting at boundaries, where your code meets other systems, since that is where time and failures hide. A third is start small and grow, adding spans and metrics in response to real questions asked during real incidents.

A short checklist that saves hours of pain later.

Every design choice has a price, and manual instrumentation is no exception.

Balancing effort and insight

Each manual call you add costs a little effort now and a little maintenance later, and it returns insight whenever someone investigates a problem. The best returns come from the few places where your code makes important decisions or talks to the outside world. Spread your effort there, lean on automatic tools elsewhere, and revisit your choices as real incidents teach you which questions matter.

Seeing how teams use manual instrumentation makes the ideas concrete.

It is writing OpenTelemetry API calls by hand in your own code, so that your program creates the spans, metrics and logs you choose, with the details you care about.

Automatic instrumentation is added by a tool or agent without code changes and covers well-known frameworks. Manual instrumentation is written by you and covers your own logic and business events.

No. They share the same API and context, so they combine naturally. Most teams start with automatic and add manual instrumentation to fill the gaps.

The API is the thin set of calls your code uses to create telemetry. The SDK is the engine that processes and exports it. Libraries should depend only on the API, and applications install the SDK.

Common causes are an unconfigured SDK, a sampler that dropped the trace, spans that were never ended, an exporter that cannot reach the Collector, or a program that exited before flushing.

A short, stable description of the kind of work, such as calculate discount. Put changing details such as identifiers into attributes instead.

Through context propagation. The sender injects the trace context into the outgoing message and the receiver extracts it, so the spans join the same trace.

No. Instrument calls to other systems, meaningful business steps and operations that may be slow or may fail. Too many spans add cost and noise.

Record the exception on the span and set its status to error, then make sure the span is ended on that path too.

Yes. Fetch the current span and add attributes to it, such as the customer tier, without creating a new span.

Slightly. Each span costs a little time and memory, which is negligible when placed sensibly and noticeable when placed inside very hot loops.

No. It uses the standard OpenTelemetry API, so you can change the exporter or backend without touching your instrumentation code.

Turn on automatic instrumentation, look at real traces, find the unexplained gaps in your own code, and add manual spans and a few business metrics there.

We began with a delivery van whose factory gauges work on their own, while the sensors for parcels and stops must be fitted by hand. Manual instrumentation in OpenTelemetry is that hand-fitted equipment. You use the thin API to create spans around meaningful operations, attach attributes that make traces searchable, record errors, create metrics for business numbers, and carry context across boundaries so that traces stay whole. The SDK, configured once by the application owner, turns those calls into real telemetry. Combined with automatic instrumentation, it gives you both the broad view of your framework and the precise view of your own decisions.