Before comparing two kinds of instruments, we need to be sure we agree on what a metric is and why software teams collect so many of them.

A metric is a number that describes something about your software, measured over time. How many requests did the website serve in the last minute? How much memory is the server using right now? How long did the average login take? Each of these is a metric. Unlike a log, which is a sentence describing one event, a metric is a compact number that is cheap to store and easy to draw on a chart.

Why do teams care so much? Because running software without metrics is like driving a car with a covered dashboard. You do not know your speed, your fuel level, or whether the engine is overheating until something breaks. Metrics are the dashboard. They tell you when traffic suddenly doubles, when a database starts to slow down, or when a queue of waiting jobs is growing faster than workers can handle it.

OpenTelemetry (often shortened to OTel) is an open-source project that defines a single, vendor-neutral way to create and send telemetry, which is the umbrella word for traces, metrics, and logs. It is backed by the Cloud Native Computing Foundation and supported by nearly every major observability product. The big promise is simple: you add OpenTelemetry to your code once, and you can send the data to almost any backend without rewriting your instrumentation.

Inside OpenTelemetry, metrics have their own rulebook, called the metrics API. At the center of that rulebook sits an idea called an instrument. An instrument is the tool you use to report a measurement. Different instruments exist because numbers behave differently. A count of orders only goes up. A count of people currently logged in goes up and down. The time a request takes is a different value every time. Giving each behavior its own instrument lets the system summarize the data correctly without guessing.

All of these instruments fall into two families, depending on who decides when the measurement is taken. If your code decides, the instrument is synchronous. If the OpenTelemetry system decides, the instrument is asynchronous. That single question, who decides when, is the key to the entire topic.

A handful of small terms appear again and again. Learn them once and the rest of the article will read easily.

One point about the API and SDK split is worth pausing on. Your application code, and especially shared libraries, talk only to the API. If nobody installs an SDK, those calls quietly do nothing and cost almost nothing. When an application owner installs the SDK, the same calls suddenly produce real data. This is why a library author can add metrics without forcing any particular monitoring setup on the people who use the library.

Two more small ideas will help. A resource is the description of the thing that produces the metrics, such as the service name, its version, and the machine it runs on. Every metric carries the resource, so you can tell one service from another without adding those labels yourself. A scope is the name of the meter that created an instrument, which tells you which library or component the number came from. Together, resource and scope answer the questions “which service?” and “which part of it?”, while attributes answer “which slice of its work?”.

Why spend time on vocabulary? Because most confusion in metrics comes from mixing up the layers. People say “metric” when they mean “instrument”, or “gauge” when they mean “any number that goes up and down”. Using the right word for each layer makes design conversations with your team short and clear, and it makes the official documentation far easier to read.

Another term you will meet is temporality. It describes how a total is reported. Cumulative means “everything since the program started”, like an odometer. Delta means “only what changed since the last report”, like a trip counter you reset after each stop. We will come back to this later, because it explains a surprising detail about how asynchronous instruments behave.

Everything else in this article grows from one simple split in who is in charge of timing.

A synchronous instrument is used by your code at the exact moment something happens, as part of the normal flow of the program. A request arrives, you call the instrument. A payment succeeds, you call the instrument. The measurement is created right there, in the same thread of work, with whatever information is available at that instant.

An asynchronous instrument is different. Your code does not call it during normal work. Instead, you hand OpenTelemetry a small function, called a callback, when you create the instrument. Later, when the SDK is about to collect metrics, it calls your function and asks, “What is the current value?” Your function reads the value and reports it. The word “asynchronous” here means the measurement happens on the SDK’s schedule, not on your code’s schedule.

This might seem like a small technical choice, but it changes almost everything: what kinds of numbers make sense, how much work your program does, how accurate the result is, and even what extra information can be attached. A good way to remember it is the pair of words events versus state. Synchronous instruments are made for events, things that happen. Asynchronous instruments are made for state, things that simply are.

Common Misunderstandings

Several wrong ideas about this topic are very common, so let us clear them up early. The first is that “asynchronous” means “faster” or “better”. It does not. It means the timing is controlled by the SDK. An asynchronous instrument is better only when the thing you measure is a state. The second is that synchronous instruments send data immediately. They do not. They update a summary in memory, and the sending happens later on the reader’s schedule. The word synchronous describes when you record, not when data leaves your program.

The third misunderstanding is that asynchronous instruments are only for system numbers like memory and CPU. They are very good for those, but they also suit business numbers, such as the number of items in a shopping cart service’s cache or the number of unprocessed rows in a table, whenever you can read the value on demand. The fourth is that you must pick one family for a whole program. In practice, almost every healthy service uses both, each in the place where it fits best.

OpenTelemetry offers four synchronous instruments. Each one fits a different kind of number.

Counter

A Counter counts things that only ever go up. You call it with a positive number each time something happens, usually the number one. It is never allowed to go down. Typical uses are the number of requests served, the number of orders placed, the number of errors seen, and the number of bytes sent.

The Counter does not remember each call as a separate record. It keeps a running sum in memory. When it is time to report, it hands over the sum. That is what makes it so cheap: a million calls become one number.

UpDownCounter

An UpDownCounter is like a Counter that is allowed to go in both directions. You add a positive number when something increases and a negative number when it decreases. It is the right choice for a value that rises and falls but where you can observe each change as it happens. Examples include the number of items in a queue (add one when a job arrives, subtract one when it leaves), the number of active connections, and the number of users currently inside a chat room.

The important feature is that the result is a sum of changes. Because it is additive, you can safely add up the values from many servers to get a company-wide total.

Histogram

A Histogram records individual values so that you can later see how they are spread out. It is the instrument of choice for things like how long a request took, how large a response was, or how many items a customer put in a basket. Instead of one number, you learn the story: most requests were fast, some were slow, and a few were very slow.

That spread matters because averages hide pain. If ninety-nine requests take one tenth of a second and one request takes ten seconds, the average looks fine, but one customer had a terrible time. A histogram keeps enough detail to let you ask about the 95th or 99th percentile, which means “how slow were the slowest few?” The SDK stores the values in buckets, which are ranges such as zero to five, five to ten, and so on. It counts how many measurements fall in each bucket and also tracks the sum and count, and often the minimum and maximum.

Gauge (synchronous)

A Gauge reports a value that is meaningful only as “the latest reading”. You call it when you learn a new value, and the SDK keeps the most recent one. A synchronous Gauge is a newer addition to the specification, so you should check that your language’s SDK supports it. It fits cases where a value changes because of an event you already handle, such as a user dragging a volume slider or a configuration change that sets a new rate limit.

The crucial rule for Gauges is that adding them together usually makes no sense. If three servers each report a temperature of 70 degrees, the combined temperature of the fleet is not 210 degrees. This “do not add” quality is what separates a Gauge from an UpDownCounter.

How a Synchronous Instrument Is Born and Used

Every synchronous instrument follows the same life story. First, at program startup, you ask a meter to create the instrument and you give it three pieces of identity: a name, a unit, and a description. The name is what people will search for on a dashboard, so it should be specific and stable. The unit tells readers whether the numbers are seconds, bytes, or plain counts. The description is one sentence of plain words explaining what the number means. After that, the instrument is stored somewhere your code can reach it, often in a shared variable or a small helper object.

Second, during normal work, your code calls the instrument with a value and, if it wants, a short list of attributes. A Counter is called with a positive amount to add. An UpDownCounter is called with a positive or negative amount. A Histogram or a Gauge is called with the actual reading. That is the whole job on your side. You never write the summing, the bucketing, or the sending. All of that is handled behind the scenes by the SDK, which is exactly why instruments are pleasant to use.

The third step is the one you do not see: the SDK adds your measurement into its in-memory summary and goes back to waiting. Because you create the instrument once and reuse it, there is no repeated setup cost. Creating a fresh instrument for every request is one of the most common beginner mistakes, and it wastes memory and time for no benefit.

Counter or Histogram: A Frequent Puzzle

Newcomers often wonder whether they need both a Counter and a Histogram for the same operation. The answer depends on what questions you want to answer. A Histogram already tells you how many measurements it received, because it keeps a count alongside its buckets. So a Histogram of request duration also tells you how many requests there were. If all you ever want is “how many”, a Counter is simpler and cheaper. If you want “how many, and how long did each take”, the Histogram covers both questions at once.

A useful way to decide is to imagine the chart you want to draw. A line that climbs over time and whose slope shows the rate of work is a Counter chart. A set of bands showing the median and the slowest one percent is a Histogram chart. If you can picture only the first chart, use a Counter. If you can picture the second, use a Histogram and enjoy the count for free.

What “Monotonic” Means

You will sometimes see the word monotonic in OpenTelemetry documents. A monotonic sum is one that never decreases. Counters and Observable Counters are monotonic. UpDownCounters and Observable UpDownCounters are not. This single word matters because monitoring tools use it to decide how to draw and calculate. For a monotonic sum, a drop in value is treated as a sign that the program restarted and the count began again from zero, and tools adjust their rate calculations accordingly. For a non-monotonic sum, a drop is simply a legitimate decrease. Choosing the wrong one makes rate calculations look strange, which is why picking the instrument that matches the true behavior of your number is so important.

There are three asynchronous instruments. Each is the “ask me when you need it” twin of a synchronous cousin.

Observable Counter

An Observable Counter reports a total that only grows, but you give it the whole total, not the increments. Your callback says, “The grand total so far is 8,914,201”, and the SDK works out the rest. This is perfect when something else is already keeping the running total for you. A good example is the total CPU time a process has used. The operating system keeps that number. You do not want to track every tiny slice of CPU yourself. You just want to read the operating system’s total whenever a report is due.

Observable UpDownCounter

An Observable UpDownCounter is the asynchronous version of the UpDownCounter. Your callback reports the current absolute value of something that can rise and fall and that can sensibly be added across instances. For example, “the queue currently holds 312 messages” or “this pool has 18 open connections”. The value is read when asked, not tracked change by change.

Observable Gauge

An Observable Gauge reports a snapshot of something that should not be added up: a temperature, a percentage of memory in use, the fill level of a cache, or the current speed of a fan. Your callback reads the value and reports it, and the SDK keeps that reading.

The Anatomy of a Callback

A callback is simply a small function that you write and hand over to the SDK. When the SDK calls it, the SDK also gives it a special helper, often called an observer or a result object. Your function does its reading and then passes each reading to that helper, along with any attributes. You do not return the numbers in the usual way. Instead you hand them over one by one, and the SDK gathers them. This design lets a single callback report many readings with different labels, such as one memory reading for each region of the heap or one queue size for each queue name.

Many SDKs also let you register more than one callback on the same instrument. This is useful in large programs where several independent components each own a slice of the truth. For example, each worker pool in an application might register its own callback on a shared “pool size” instrument, each reporting under its own attribute value. At collection time, the SDK runs them all and combines the results into one metric.

Why Not Always Use the Synchronous Version?

A fair question is why the asynchronous family exists at all, since a synchronous UpDownCounter or Gauge could in theory represent the same numbers. The answer is ownership. When the number lives inside code you do not control, such as a database driver, a thread pool library, or the operating system, you cannot easily place a recording call at every change. Even if you could, the library might change a hundred times a second, and recording each change would be wasteful when you only need the value once a minute. The asynchronous family lets you stay outside the machinery and simply peek in.

There is also a correctness argument. A running total built from many small updates is only as good as your discipline in never missing an update. A value read straight from the source is correct by construction. For anything that can be read directly, the asynchronous path trades a little detail for a lot of reliability.

Notice the pattern. For each style of number there is a synchronous and an asynchronous version. The Counter family has Counter and Observable Counter. The up-and-down family has UpDownCounter and Observable UpDownCounter. The snapshot family has Gauge and Observable Gauge. The Histogram stands alone: it has no asynchronous version, because a histogram needs to see individual values as they happen, and a callback that runs once a minute cannot reconstruct what happened between visits.

Seeing the full journey makes the difference between the two families obvious.

The picture below shows the two paths side by side. On the top path, your application code calls a synchronous instrument directly. On the bottom path, a callback function feeds an asynchronous instrument, but only when the reader asks for a collection. Both paths meet in the SDK, share the same aggregation machinery, and leave through the same reader and exporter.

flowchart LR
  A["Your Application Code"] -->|"call at the moment of the event"| B["Synchronous Instrument"]
  B --> C["Meter Provider and SDK"]
  D["Your Callback Function"] -->|"reports when asked"| E["Asynchronous Instrument"]
  E --> C
  C --> F["Aggregator in Memory"]
  F --> G["Metric Reader"]
  G --> H["Exporter"]
  H --> I["Collector or Monitoring Backend"]
  G -.->|"collection tick runs the callbacks"| D
    

Let us walk through the figure from left to right. Your application is the origin of the numbers. On the synchronous path, the application calls the instrument whenever something happens, and the instrument immediately hands the measurement to the SDK. On the asynchronous path, the application only registers a callback once, usually at startup. After that, nothing happens on that path until the reader asks.

The Meter Provider is the factory and manager that holds all meters and connects them to the SDK settings. The aggregator is the in-memory bookkeeper. For a Counter it keeps a sum. For a Histogram it keeps bucket counts. For a Gauge it keeps the last value. The metric reader decides when a snapshot of all those summaries is taken. Finally, the exporter ships the snapshot to a backend, often through the OpenTelemetry Collector, which is a separate program that receives, processes, and forwards telemetry.

The dotted line is the heart of the matter. When the reader performs a collection, it first runs every registered asynchronous callback so the data is fresh, then gathers everything. Collection can be triggered in two common ways. With a push style reader, a timer fires on a fixed interval, commonly every sixty seconds by default, and the SDK pushes the data out. With a pull style reader, such as the one used for Prometheus-format scraping, collection happens each time a monitoring server visits your metrics endpoint. In both cases, asynchronous callbacks run only at collection time.

Where the Collector Fits

Many production setups place an OpenTelemetry Collector between the application and the backend. The Collector is a small, separate program that receives metrics, can clean them up, batch them, drop unwanted labels, and forward them to one or more destinations. It sits outside your application, so it can be updated and tuned without redeploying your services. From the point of view of the instruments, nothing changes: synchronous calls and asynchronous callbacks behave the same whether the exporter talks to a Collector or straight to a vendor. But the Collector gives operations teams an extra place to enforce rules, such as removing a risky label before it reaches an expensive backend.

Think of the Collector as a postal sorting office. Letters, which are your metrics, arrive from many houses, each in its own envelope. The sorting office stamps them, groups them, and sends them on to the right city. The people writing the letters do not need to know the details of the roads behind the sorting office.

What actually happens in the moments after you record a measurement.

When your code calls a synchronous instrument, three things travel together: the value, the attributes, and the surrounding context. Context is the invisible bundle of information that follows a piece of work around, including the current trace. A trace is the story of one request as it moves through several services. Because a synchronous call is made in the middle of that work, the SDK can see which trace and which step of the trace the measurement belongs to.

This link between metrics and traces is valuable. A feature called exemplars lets the SDK attach a small sample, “here is one real trace that produced a value in this bucket”, to a metric data point. Imagine your dashboard shows a latency spike in the slowest histogram bucket. With an exemplar you can click straight through to an actual slow request and see what went wrong. Because asynchronous callbacks run on the SDK’s own timer, away from any particular request, they usually have no such trace to point at.

Next, the SDK finds the right bucket of memory for this combination of instrument and attributes. If you recorded a Counter with the attribute route equals checkout, the SDK finds or creates the slot for that exact label set and adds to its running sum. This lookup is quick, but it is not free, and it happens on your request path. On a service handling tens of thousands of requests per second, the cost of each call matters, which is why SDKs are heavily optimized and often use lock-free techniques to stay fast when many threads record at once.

The Danger of Too Many Labels

Each unique combination of attribute values creates a separate time series, which is one line on a chart. This count of unique combinations is called cardinality. If you label a request counter with the customer’s user ID and you have a million users, you have just created a million lines. Memory use, network traffic, and the bill from your monitoring vendor all explode. Most SDKs protect you with a default limit, commonly around two thousand series per metric, after which extra combinations are merged into a single overflow series. That safety net is helpful, but it means the extra detail is lost. Choose labels with a small, known set of values, such as HTTP method, route template, status class, or region.

Finally, the aggregator holds the summary until the reader collects. A synchronous Counter that is called a million times between two collections still produces one data point. This is the magic that makes metrics so much cheaper than logs for high-volume events.

Many Threads at Once

Real services do many things at the same time. Dozens or hundreds of threads may call the same Counter in the same instant. If the SDK handled this carelessly, two threads could try to update the same number together and one update would be lost. To prevent this, SDK authors use special techniques that let threads add to a shared total safely and quickly, without making everyone wait in a long line. As a user you do not have to think about this, but it is reassuring to know that the Counter in your program is safe to share across all your threads, and that you should share it rather than create copies.

The Lifetime of a Series

When you record a new combination of attributes for the first time, the SDK creates a new slot to hold its summary. That slot stays in memory so that the next measurement with the same attributes can find it quickly. Depending on the temporality setting, the SDK may keep the slot forever, or it may drop slots that received no new measurements since the last export. With cumulative temporality, most SDKs keep reporting the slot because the total still exists and monitoring tools expect to keep seeing it. With delta temporality, an idle slot has nothing new to say, so it can be released. This is one more reason that cardinality matters: slots take memory for as long as they live.

Why Synchronous Calls Must Stay Fast

The recording call runs inside your business logic. If a checkout request records five metrics, the checkout is now five small steps longer. SDK designers know this, so they keep the recording path as short as possible: find the slot, add the number, return. They avoid slow actions such as network calls or disk writes on this path. The expensive work of packaging and sending happens later, on the reader’s schedule, away from your request. This separation between a fast recording path and a slower exporting path is a core idea you will see again and again in telemetry design.

The life of a callback, from registration to reporting.

The asynchronous story begins at startup. You create the instrument and register your callback function. The SDK stores a reference to it and does nothing else. Then time passes. At each collection, the reader triggers the sequence shown below.

sequenceDiagram
  participant R as Metric Reader
  participant S as SDK
  participant C as Callback
  participant A as Application State
  participant E as Exporter
  R->>S: Collection tick arrives
  S->>C: Invoke the registered callback
  C->>A: Read the current value
  A-->>C: Return the value
  C-->>S: Report observation with attributes
  S->>R: Hand over the summarized data point
  R->>E: Export the batch
    

Step by step: the metric reader announces that it is time to collect. The SDK looks up every callback tied to every asynchronous instrument and runs them. Each callback reads something from the real world, such as a number inside a thread pool, a value from the operating system, or a property of a hardware sensor. It reports one or more observations, each made of a value and a set of attributes. The SDK takes those observations, summarizes them, and sends them out with the rest of the data.

Absolute Values, Not Increments

The most common surprise for newcomers is that an Observable Counter wants the total so far, not the change since last time. If your callback accidentally reports the increment instead, your chart will look flat or wildly wrong. The SDK is smart about the rest: if your backend prefers delta temporality, the SDK remembers the previous total and subtracts it to produce the change. If your backend prefers cumulative, the SDK passes the total straight through.

Rules for Writing a Good Callback

A callback runs inside the collection process, so it must behave politely. Keep these rules in mind.

• Be quick. A callback that takes seconds to run will delay collection for everything. The SDK may cancel slow callbacks after a timeout.
• Be safe to call from any thread. Callbacks can run on a background thread while your application keeps working. Reading shared state should be done with care.
• Do not do heavy work. Avoid running a big database query inside a callback. If a value is expensive to compute, compute it in the background and let the callback read the stored result.
• Report each label combination once per collection. Reporting the same attribute set twice in a single callback leads to unclear results.
• Handle errors gracefully. A callback that throws an exception should not take down your application. Catch problems and simply report nothing for that round.

One Callback, Many Instruments

Sometimes one expensive read gives you several numbers at once. Reading the operating system’s memory statistics might give you used, free, and cached memory in one call. Many OpenTelemetry SDKs allow a single callback to report to several asynchronous instruments together, so you read the source once. This keeps the numbers consistent with each other, because they all come from the same moment in time.

More Than One Reader

A single program can have more than one metric reader at the same time. For example, it might push data to a central Collector every minute while also offering a scrape endpoint for a local monitoring tool. Each reader decides its own collection moments. That means an asynchronous callback may run more often than you expect, once for each reader’s collection. This is another reason to keep callbacks cheap and free of side effects. A callback that is safe to run many times, at any moment, is a good callback.

Timeouts and Gaps

Collection is not allowed to hang forever. Most SDKs apply a timeout to the whole collection, and some apply one to each callback. If your callback is too slow, the SDK may stop waiting and move on, and that metric will simply be missing for the round. On a chart, this shows up as a gap or a flat stretch. If you see strange gaps in an asynchronous metric while synchronous metrics from the same service look healthy, the callback is the first place to look.

Staleness and Freshness

An asynchronous value is only as fresh as the moment it was read. If your callback returns a cached number that a background task refreshes every five minutes, then the chart will show steps that change every five minutes, even if you collect every ten seconds. This is not wrong, but it is worth knowing. Decide how fresh a value needs to be, and arrange the background refresh to match, so that the chart tells the truth about the thing you care about.

A Timing Subtlety

Because asynchronous instruments are sampled only at collection time, they cannot see what happened in between. If a queue grew to ten thousand items and shrank back to ten within a single minute, a callback running once a minute might never notice the spike. A synchronous UpDownCounter, which changes with every event, would have caught the change as it occurred, although its exported value is also a snapshot at collection time. This is a built-in trade-off, and we will return to it when we compare the two families.

A single table to hold in your head, followed by a few numbers to illustrate the cost difference.

Look at the “work per event” row. Imagine a service that handles twenty thousand requests per second. A synchronous call for each request means twenty thousand small updates every second. That is usually fine because the updates are tiny, but it is real work on the hot path. Now imagine an asynchronous gauge reading the number of open connections. It runs once per collection, perhaps once a minute. That is one read per minute, no matter how much traffic arrives. The cost is almost zero.

Thinking About Cost in Concrete Terms

Cost has three parts: the time your program spends, the memory it holds, and the data it sends and stores. For synchronous instruments, time grows with traffic, memory grows with the number of unique attribute combinations, and data grows with the same number of combinations rather than with traffic. That last point surprises many people. Ten million requests with three route values and three status classes produce nine series, not ten million data points. The traffic is folded into the sums and buckets.

For asynchronous instruments, time grows with the number of collections and the cost of each callback, memory is usually tiny, and data grows with the number of observations each callback reports. A callback that reports one value per customer account, on a service with a million accounts, is just as dangerous as a synchronous counter labeled by account. The family of the instrument does not protect you from high cardinality. Only careful choice of attributes does.

Finally, remember the bill from your monitoring vendor. Many vendors charge by the number of active series. A single careless label can multiply that number overnight. Teams that treat attribute design as a first-class engineering task, reviewed with the same care as a database schema, avoid most of the unpleasant surprises.

The reverse is also true. If you tried to measure “each request’s duration” with an asynchronous instrument, you would only ever see one value per minute and lose almost all the information. So the cost advantage of asynchronous instruments applies only when the number you want really is a state.

When you stand in front of a new number and wonder which instrument to use, follow this path.

flowchart TD
  S["I want to measure something"] --> Q1{"Can I hook into the moment it happens?"}
  Q1 -->|"Yes, it is an event"| Q2{"Do I care about the spread of values?"}
  Q2 -->|"Yes"| H["Histogram"]
  Q2 -->|"No, only a total"| Q3{"Can the total ever go down?"}
  Q3 -->|"No"| C["Counter"]
  Q3 -->|"Yes"| U["UpDownCounter"]
  Q1 -->|"No, I only read a current value"| Q4{"Does it only grow?"}
  Q4 -->|"Yes"| OC["Observable Counter"]
  Q4 -->|"No, and it adds across instances"| OU["Observable UpDownCounter"]
  Q4 -->|"No, and adding makes no sense"| OG["Observable Gauge"]
    

The figure compresses the logic into a few questions. Let us turn it into a practical routine you can follow every time.

Walking Through Three Tricky Cases

Case one: the number of logged-in users. Your login code adds one on sign-in and subtracts one on sign-out. This sounds like a perfect synchronous UpDownCounter. But sessions also expire silently after a timeout, and a user may close the browser without signing out. If the expiry path forgets to subtract, the count drifts upward. The safer design is to ask the session store how many active sessions exist and report that with an Observable UpDownCounter, since the store is the single source of truth.

Case two: the size of a file upload. Each upload has a size, and you care about the spread, from tiny avatars to huge videos. A Counter of total bytes would hide the spread. A Histogram of upload sizes shows how many are small and how many are enormous, and it also gives you the total through its sum. This is clearly an event with a value, so a synchronous Histogram wins.

Case three: the number of items in a database table. Counting rows on every insert and delete would mean touching every write path, and counting them with a query is slow. A reasonable design is to refresh the count in the background every few minutes and let an Observable Gauge or UpDownCounter report the stored result. The value is slightly delayed, but it is cheap, safe, and honest about being a snapshot.

Concrete situations that show each instrument in its natural home. These are illustrative patterns commonly seen in production systems.

Putting It All Together: An Order Service

Consider an order service in a large online marketplace. It receives an order, checks stock, charges a payment, and confirms the purchase. A thoughtful team would instrument it with both families, each placed where it fits.

At the front door, a synchronous Histogram records how long each order takes from arrival to confirmation, labeled by outcome. A synchronous Counter records orders by payment method and result. Inside the stock check, a synchronous Histogram records how long the inventory lookup takes, so a slow warehouse system is quickly noticed. These are all events, and the code owns each of them.

Behind the scenes, the service keeps a local cache of product details. An Observable Gauge reports how full the cache is, because the cache library already knows its own size. An Observable Counter reports the total number of cache evictions, again read straight from the cache library’s own statistics. An Observable UpDownCounter reports the number of orders waiting in the internal retry queue, read from the queue object when asked. None of these required the team to touch the cache or queue internals.

On the dashboard, the story is clear. The Histogram shows customers’ experience. The Counter shows business volume and failure rate. The cache and queue readings show the health of the machinery behind it. When latency rises, the team can glance at the queue and cache readings to see whether the cause is inside the service or in a dependency. This blend of event-style and state-style numbers is what a mature observability setup looks like.

Neither family is better. Each is better at a different job.

Synchronous Instruments

Asynchronous Instruments

When the Two Families Disagree

Occasionally you will measure the same underlying thing both ways and see slightly different numbers. Suppose a synchronous UpDownCounter tracks active sessions, and an Observable UpDownCounter reads the session store’s own count. The two can differ by a handful at any moment, because one is built from the stream of changes and the other is read from the source. Small differences are normal and are caused by timing. Large or growing differences are a warning sign that some code path is missing an update. Many teams deliberately run both for a while after launching a new feature, then keep the one that proves more trustworthy.

A Mental Picture to Carry Away

Picture two kinds of reporters at a busy airport. The first stands at the arrival gate and writes down every passenger who walks through, noting the airline, the time, and how long the luggage took to arrive. Nothing escapes this reporter, but the reporter must be at the gate all day and must write something for every single arrival. This reporter is the synchronous instrument. The second reporter wanders the terminal once an hour, looks at the departure board, counts the people in each queue, and reads the thermometer in the hall. This reporter writes very little and never disturbs the passengers, but misses anything that happened between walks. This reporter is the asynchronous instrument. A good airport newsroom employs both, because the passengers’ journeys and the terminal’s condition are both part of the story.

Every time you meet a new number to measure, ask which reporter you need. If you need the gate reporter, record at the moment of the event. If you need the terminal reporter, register a callback and let the SDK walk by when it is time. With that picture in mind, the long list of instruments stops feeling like a list to memorize and starts feeling like a natural set of tools, each placed where it works best.

Choosing Under Pressure

In a real project you will not always have time for a long analysis. A practical shortcut is to ask three quick questions. First, is this a thing that happened, or a thing that is? Second, do I need to know about individual occurrences, or just the current picture? Third, who already keeps this number? If the answers are “happened”, “individual occurrences”, and “my own code”, reach for a synchronous instrument. If the answers are “is”, “current picture”, and “someone else”, reach for an asynchronous one. Mixed answers mean you should think a little longer, and probably consider using both.

The Key Trade-offs

The first trade-off is accuracy versus cost. Synchronous instruments see everything but charge a tiny fee on every call. Asynchronous instruments charge almost nothing but only see snapshots. The second is convenience versus control. Asynchronous instruments are convenient when you can read existing state without touching code. Synchronous instruments give you finer control over what is recorded and with which labels. The third is drift versus sampling error. A synchronous UpDownCounter can drift if a decrement is missed during an error path. An asynchronous reading cannot drift, but it may misrepresent short bursts.

How raw measurements become the lines on your chart, and how you can reshape them.

Every instrument comes with a default way of summarizing its measurements. Counters and UpDownCounters, both synchronous and observable, are summarized as a sum. Gauges, both kinds, are summarized as the last value. Histograms use explicit buckets by default. This default is chosen so that the result matches what the instrument means.

You are not stuck with the defaults. OpenTelemetry offers views, which are configuration rules that change how an instrument is reported. A view can rename a metric, drop attributes you do not want, change the unit, or switch the aggregation. For example, you can ask a Histogram to use exponential buckets, a newer method that adapts its bucket sizes automatically and gives good accuracy across a wide range of values with less configuration. Views are also the safest way to control cardinality: if a library emits a metric with a risky label, you can remove that label in the SDK without changing the library.

Cumulative or Delta

Remember the odometer and the trip counter? Cumulative temporality reports the odometer. Delta temporality reports the trip counter. Some backends, notably Prometheus-style systems, prefer cumulative values and compute rates themselves. Others prefer deltas. The SDK can be configured for either. For synchronous Counters, the SDK either keeps a running sum forever or resets it after each export. For asynchronous Observable Counters, the SDK compares your reported total against the one before it to produce deltas. The instrument you chose does not need to change when the backend preference changes. This is one of the quiet strengths of the design.

Reading Percentiles from Buckets

A histogram does not store every value. It stores how many values fell into each range. When a chart shows the 99th percentile, the monitoring tool estimates it from those counts. If the buckets are wide, the estimate is rough. If one bucket spans everything between one second and ten seconds, the tool cannot tell whether the slow requests took two seconds or nine. This is why bucket design matters so much for latency numbers, and why exponential histograms are attractive: they keep the buckets narrow where it counts without you having to guess in advance.

It also explains why you should never average percentiles from different servers. The 99th percentile of one machine and the 99th percentile of another cannot be combined by averaging, because each hides its own distribution. Instead, combine the underlying bucket counts first and then compute the percentile from the merged result. Histograms are built to support exactly this kind of merging, which is a further reason to prefer them over gauges for anything that looks like a distribution.

Histogram Bucket Boundaries

A very common beginner mistake is forgetting that default histogram buckets are tuned for a particular range, often millisecond-sized values. If you record durations in seconds, almost every measurement may land in the first bucket, and your chart shows nothing useful. Always match your bucket advice to your unit, or use exponential buckets so the problem disappears.

A Worked Example: One Minute in the Life of a Service

Let us follow a small, imaginary service through a single minute so that all these ideas come together. The service handles web requests. It owns a synchronous Counter for finished requests, a synchronous Histogram for request duration, and an Observable Gauge for the number of open database connections. Push collection happens once a minute.

During the minute, eighteen thousand requests arrive. For each one, the code adds one to the Counter, with attributes for the route and the status class, and records the duration in the Histogram with the same attributes. The SDK quietly adds each number into its in-memory summary. Seven hundred of those requests are slow, and they land in the high buckets of the Histogram. The connection pool, meanwhile, rises to forty connections during a burst and falls back to twelve by the end of the minute. Nobody told OpenTelemetry about any of that, because the gauge is asynchronous.

When the minute ends, the reader announces a collection. The SDK runs the connection callback, which reads the pool and reports twelve. The SDK then gathers the Counter’s sums, the Histogram’s buckets, and the single gauge value, and hands them to the exporter. On the dashboard, the Counter shows eighteen thousand requests, the Histogram shows that the slowest one percent took far longer than the median, and the gauge shows twelve connections.

Notice what the dashboard did not show: the burst of forty connections. The gauge only saw the quiet moment at the end. If the burst matters, the team should add a synchronous Histogram of “time spent waiting for a connection”, which would reveal the pressure even if the pool size looked calm at collection time. This small example shows how the two families complement each other. One shows what happened. The other shows how things stand.

Running instruments in production means thinking about cost, health, and safety.

Performance

For synchronous instruments, the main cost is the work done on each call, plus the memory used by every unique attribute combination. Keep attribute sets small and stable. If you can, reuse the same set of attributes rather than building a new one on every call. For asynchronous instruments, the main cost is the callback itself and how often it runs. A callback that scans every file on a disk every sixty seconds is a bad idea. A callback that reads a cached number is a good idea.

Collection interval is a dial you control. A shorter interval gives finer-grained charts but more data to send and store. A longer interval is cheaper but hides short events. For asynchronous instruments, the interval also decides how often your callbacks run. Think of it as choosing how often the security guard makes rounds.

Monitoring Your Monitoring

Telemetry systems can fail too. Watch for signs such as a callback that regularly exceeds its time limit, metric series that suddenly multiply, exports that fail repeatedly, or a dashboard line that goes flat when it should be moving. A flat line from an asynchronous instrument often means the callback is throwing an error or returning nothing. A runaway number of series usually means a label with unbounded values slipped in. Many SDKs emit their own internal diagnostics and you should route those somewhere you will see them.

Security

Treat metrics as data that leaves your system. Keep personal information, secrets, and internal-only identifiers out of attributes. Use encrypted connections between your exporter, the Collector, and the backend. Limit who can reach any local metrics endpoint, since a pull-style endpoint can reveal details about your service, such as its version, route names, and load. Finally, remember that callbacks run code on a schedule. A callback should only read, never change anything, and should not trust input that comes from outside your service.

Testing and Debugging Your Instruments

Instruments deserve testing like any other part of a program. A simple approach is to run your code against an in-memory reader that keeps the collected data in a list instead of sending it anywhere. Your test performs some work, asks the reader to collect, and then checks that the expected numbers appear with the expected attributes. For asynchronous instruments, the same test also proves that the callback runs and reports sensible values. Because the collection is under the test’s control, there is no waiting for timers.

When something looks wrong in production, work from the outside in. First, check that the exporter is delivering data at all. Second, check that the metric name and unit match what the dashboard expects. Third, look at the attributes: a missing line often means the labels changed. Fourth, for asynchronous instruments, check the callback for errors and timeouts. Fifth, for synchronous instruments, check that the recording call is actually reached on every code path, including error paths. This methodical order saves hours of guesswork.

Alerting on Instruments the Right Way

Alerts built on metrics should respect the nature of each instrument. For Counters, alert on rates, such as errors per minute, rather than on raw totals, because a total only grows. For Histograms, alert on percentiles, such as the 99th percentile of latency, rather than the average. For gauges, alert on thresholds, such as memory above a safe level for several collections in a row. Requiring several consecutive readings helps because an asynchronous gauge is a sample, and a single odd sample should not wake someone at night.

Habits that work, and habits that quietly cause pain.

Patterns That Work

• Record at the boundary. Place synchronous instruments at clear entry and exit points, such as the start and end of request handling, so every path is covered once.
• Read, do not own. When another component already keeps a number, read it with an asynchronous callback instead of duplicating the tracking.
• Pair a Counter with a Histogram. A Histogram already yields a count, so you may not need a separate request counter, but pairing outcomes with durations tells a complete story.
• Use one callback for related readings. Group numbers that come from the same source so they stay consistent.
• Follow naming conventions. Use OpenTelemetry’s semantic conventions, the shared naming rules, so that names, units, and labels match what other tools expect.

Anti-Patterns to Avoid

Evolving Your Metrics Over Time

Metrics are not written once and forgotten. Services change, traffic grows, and questions evolve. A good habit is to review your instruments every few months. Remove metrics nobody looks at, because each one costs money and attention. Add metrics for new failure modes discovered during incidents. Revisit attribute choices that produced too many series. And when you must change a metric’s meaning, publish a new name instead of silently changing the old one, run both for a while, and migrate dashboards and alerts before retiring the old one. This careful, gradual approach keeps trust in your numbers, which is the most valuable asset any metrics system has.

A Short Review Checklist for Teams

When a teammate opens a pull request that adds a new instrument, a few quick questions catch most problems. Is the name clear and does it follow the shared naming rules? Is the unit set? Is the instrument created once and reused? Does the chosen instrument match the real behavior of the number, in particular whether it can decrease and whether it can be added across servers? Are the attributes bounded and free of private data? If the instrument is asynchronous, is the callback fast, safe, and read-only, and does it report absolute values? If it is a Histogram, do the bucket settings match the unit? Five minutes of review here prevents days of confusion later.

Other Common Mistakes

• Creating a new instrument on every request instead of creating it once and reusing it. Instruments should be created at startup and kept.
• Forgetting to set a unit, which leaves viewers guessing whether a value is milliseconds or seconds.
• Using a Counter for a value that can decrease, which is not allowed and may be ignored or rejected.
• Adding up gauge values across servers and treating the result as meaningful.
• Assuming the asynchronous callback runs on a precise clock. It runs when collection happens, which can drift from your expectation.

A short list you can paste into a team wiki and check before shipping new metrics.

• Decide first whether the number is an event or a state, and pick the family on that basis.
• Create each instrument once at startup and reuse it for the life of the program.
• Choose names that follow semantic conventions and include standard units.
• Keep attribute values from a small, known set. Review them in code review as carefully as you review a database schema.
• Make callbacks fast, read-only, and safe to run from any thread. Catch every error.
• Report absolute values from asynchronous instruments.
• Use Histograms for anything where percentiles matter, and match bucket settings to the unit.
• Set a sensible collection interval and review it when costs or accuracy needs change.
• Use views to trim, rename, or reshape metrics that arrive from third-party libraries.
• Alert on the health of the telemetry pipeline itself, not only on the services it observes.
• Document each important metric, including who owns it and what action to take when it looks wrong.

Quick answers to questions that come up again and again.

If you remember only a few things from this guide, remember these.

The difference between synchronous and asynchronous instruments in OpenTelemetry comes down to who decides when a measurement is taken. With synchronous instruments, your code reports at the moment an event happens. With asynchronous instruments, the SDK asks your callback for the current value when it collects. Everything else, from cost to accuracy to which numbers make sense, follows from that one idea.