Before we meet the Observable Gauge, we need to understand the world it lives in. This chapter explains the three words you will keep hearing: observability, telemetry and metrics.

When a program runs on your laptop, you can watch it directly. You can click buttons, read error messages and open the code. When a program runs in production, meaning on real servers serving real users, you usually cannot watch it directly. The servers might be in a data center thousands of kilometers away. There might be hundreds of copies of your program running at the same time. You cannot stand next to each one and look at it.

Observability is the ability to understand what is happening inside a system by looking at the information it sends out. If a system is observable, you can ask new questions such as “why did checkout get slow at 3 p.m.?” and find the answer from the data the system already produces, without having to change the program first.

The information a system sends out is called telemetry. The word comes from Greek roots meaning “measuring from far away.” Weather stations, space probes and medical heart monitors all send telemetry. Software does too.

The three main kinds of telemetry

Engineers usually talk about three kinds of telemetry, often called signals.

• Logs are timestamped text messages. Think of a ship captain’s diary: “10:02 a.m., storm approaching,” “10:40 a.m., changed course.”
• Traces follow one single request on its journey through many services. Think of the tracking page of a parcel that shows every warehouse it passed through.
• Metrics are numbers measured over time. Think of the dials on a car dashboard, written down once every few seconds.

This guide is about metrics. A metric is a measurement with a name, a number, a unit and some labels, recorded at a moment in time. When you collect the same metric again and again, you get a time series, which is simply a list of numbers in time order. Draw a time series on a chart and you get the familiar line graph on a monitoring dashboard.

Why metrics are so popular

Metrics are small. A single number takes a few bytes, so even a company that tracks millions of numbers can store them cheaply. Metrics are fast to query, which is why dashboards load in a second. And metrics are perfect for alerting: “wake up the on-call engineer if CPU utilization stays above 90 percent for five minutes.” Logs and traces are richer, but they are heavier. Metrics give you the big picture quickly.

Now, not all metrics behave the same way. Some only ever go up, such as “total number of orders placed.” Some go up and down, such as “number of people currently in the shop.” Some describe a spread of values, such as “how long each request took.” OpenTelemetry gives each behavior its own tool, called an instrument. The Observable Gauge is one of these tools, and by the end of this guide you will know exactly which job it was made for.

OpenTelemetry is the toolbox that the Observable Gauge belongs to. Let us look at how the toolbox is organized.

OpenTelemetry, often shortened to OTel, is an open-source project under the Cloud Native Computing Foundation. It defines a common way to create, collect and send logs, traces and metrics. Before OpenTelemetry, every monitoring product had its own libraries and its own formats. If you wanted to switch from one product to another, you had to rewrite the measuring code inside your application. OpenTelemetry fixes this by giving everyone one shared standard. You write your measuring code once, and you can send the data to many different products, including Prometheus, Grafana, Datadog, New Relic, Honeycomb and the monitoring services of the big cloud providers.

The main building blocks

For metrics, OpenTelemetry has a few important pieces. Learn their names now, because they appear again and again.

OTLP stands for the OpenTelemetry Protocol. It is the standard language that OpenTelemetry components use to talk to each other. Think of it as the shared shipping container format that every port in the world agrees on.

There is one more important part: the OpenTelemetry Collector. It is a separate program that sits between your applications and your monitoring backend. It receives data, can clean it up or filter it, and forwards it onward. Many teams run a Collector on every machine or in every cluster, so the application only has to send its data to a nearby neighbor rather than to the faraway backend.

graph LR
  A["Your Application Code"] --> B["Metrics API"]
  B --> C["Metrics SDK"]
  C --> D["Metric Reader"]
  D --> E["OTLP Exporter"]
  E --> F["OpenTelemetry Collector"]
  F --> G["Prometheus"]
  F --> H["Cloud Monitoring Backend"]
  G --> I["Grafana Dashboards"]
    

Read the figure from left to right. Your application code calls the Metrics API. The SDK behind the API keeps track of the numbers. A metric reader collects them on a schedule. An exporter packs them into OTLP and ships them to the Collector. The Collector forwards them to one or more backends, and finally a dashboard tool such as Grafana draws the charts. Every instrument, including the Observable Gauge, travels this same road.

The Observable Gauge has six siblings. Seeing the whole family together makes it much easier to understand what makes the Observable Gauge special.

An instrument is a named tool that you create once and then use to report measurements. OpenTelemetry defines seven instruments. They differ along two questions. First, how does the measurement arrive: does your code push it in at the moment something happens (synchronous), or does the SDK ask for it later on a schedule (asynchronous)? Second, what kind of number is it: a total that only grows, a value that goes up and down, or a spread of many values?

Do not worry if the table feels like a lot. We will come back to each row. For now, notice the bottom row. The Observable Gauge is the asynchronous instrument for values that are non-additive. That word is the key to the whole story, and Chapter 6 is devoted to it.

Synchronous versus asynchronous

A synchronous instrument is used inside your normal program flow. When a customer places an order, your code runs a line that says “add one to the orders counter.” The measurement is recorded at the exact moment the event happens, and your code is the one that decides when.

An asynchronous instrument works the other way around. You do not push values in. Instead, you hand the SDK a small function, called a callback, which knows how to read the current value. The SDK calls that function whenever it needs fresh data, usually once every collection cycle. This is why asynchronous instruments are also called observable instruments: the SDK observes the value by asking for it.

Now we put the pieces together and give a clear definition, then walk through what each part of the definition means.

An Observable Gauge is an asynchronous instrument in the OpenTelemetry Metrics API that reports the current value of something that can go up and down, and whose values should not be added together across different sources. You give it a callback. At every collection cycle, the SDK calls your callback, your callback reads the current value from wherever it lives, and the SDK records that value as a measurement.

Let us unpack the definition word by word.

“Gauge”

A gauge is a dial that shows the current level of something. Pressure gauges, fuel gauges and rain gauges all work this way. The word tells you that the number means “how much is there right now,” not “how much happened in total.”

“Observable”

Observable means “can be looked at on demand.” Your code does not shout out the value every time it changes. Instead, the value just sits there, and the SDK looks at it when it is time to take a reading. The callback is the pair of eyes the SDK uses to look.

“Non-additive”

This is the subtle part. A value is additive if adding the values from several sources gives a meaningful answer. If server A handled 500 requests and server B handled 300 requests, the total of 800 is meaningful. A value is non-additive if adding the values makes no sense. If room A is 20 degrees Celsius and room B is 22 degrees Celsius, saying “the total temperature is 42 degrees” is nonsense. You might calculate an average, a minimum or a maximum, but never a sum. The Observable Gauge is designed for these non-additive values.

The ingredients of an Observable Gauge

Whenever you create an Observable Gauge, you provide a few things.

• A name, such as room.temperature. This is how the metric is identified in your monitoring tool.
• An optional unit, such as degrees Celsius. Units follow a standard code system called UCUM, where Celsius is written Cel and a plain ratio between zero and one is written 1.
• An optional description, a short sentence in plain words explaining what the number means.
• A callback, the small function that reads the current value and reports it. The callback can report one value, or many values with different attributes, which are labels such as room=kitchen or room=garage.

The same idea in every language

OpenTelemetry has official SDKs for many languages. The spelling changes to match each language’s habits, but the idea is identical.

Because this guide is about understanding, not about typing code, we will focus on the idea. Once you understand the idea, translating it to your language takes a few minutes with the official documentation open next to you.

The callback is the heart of the Observable Gauge. Understanding it clears up most of the confusion beginners have.

A callback is a function that you write now but that somebody else will run later. You are saying to the SDK: “Here is a recipe for reading the temperature. Whenever you want a fresh reading, follow this recipe.” You do not decide when it runs. The SDK does.

This is a very different style from the Counter. With a Counter, your code is in charge: it calls the instrument whenever something happens. With an Observable Gauge, the SDK is in charge: it calls your code whenever it is time to collect.

When does the callback run?

The callback runs during a collection. A collection happens when the metric reader asks the SDK for the current state of all metrics. There are two common styles of reader.

• Periodic push. A periodic exporting reader collects on a timer and pushes the result to a destination. The default export interval in many SDKs is 60 seconds, and teams often shorten it to 10 or 15 seconds for faster dashboards.
• Pull on request. A reader that serves a Prometheus-style endpoint collects every time the Prometheus server comes to scrape. If Prometheus scrapes every 15 seconds, your callback runs about every 15 seconds.

In both styles, the callback runs once per collection, not once per second and not once per request. That matters a great deal for how much work your callback can safely do.

sequenceDiagram
  participant T as Timer or Scrape
  participant R as Metric Reader
  participant S as SDK
  participant C as Your Callback
  participant X as Thing Being Measured
  T->>R: Time to collect
  R->>S: Collect all metrics
  S->>C: Please observe now
  C->>X: Read current value
  X-->>C: 22.5
  C-->>S: Report 22.5 with attributes
  S-->>R: Metric data ready
  R-->>T: Send to exporter
    

Follow the arrows. A timer or a scrape request wakes up the reader. The reader asks the SDK to collect. The SDK calls your callback. The callback reads the thing being measured, such as a sensor, an operating system counter or an in-memory object, and reports the result back. The SDK packages the answer and the reader sends it onward. Between collections, nothing happens. The gauge is simply waiting for the next visit.

What this means in practice

Because the value is only read at collection time, an Observable Gauge gives you a sample of reality, not a full recording. If a temperature spikes for three seconds and falls back down between two readings, the gauge never sees the spike. This is not a bug. It is the nature of sampling, and it is an important thing to remember when you choose which instrument to use.

The most common beginner question is “which instrument should I pick?” The answer almost always comes down to one test: can I meaningfully add these numbers together?

Let us make the test concrete. Imagine you run ten copies of a web service. Each copy reports a number. Now ask yourself: if I add up the ten numbers, do I get something useful?

• Requests handled. Adding gives the total traffic of the whole service. Useful. This is additive.
• Open connections. Adding gives the total connections across the whole service. Useful. This is additive.
• CPU utilization as a percentage. Ten copies each at 40 percent do not add up to 400 percent of anything useful. You would want the average or the maximum. This is non-additive.
• Cache hit ratio. Ten copies each at 0.9 do not add up to 9. The sum is meaningless. This is non-additive.
• Battery level. Ten batteries at 80 percent do not add up to 800 percent. This is non-additive.

When the numbers are additive, the specification points you toward a Counter or an UpDownCounter, because backends can safely add those values across sources. When the numbers are non-additive, it points you toward a Gauge, and when the value is read on demand, specifically toward the Observable Gauge.

graph TD
  A["I want to measure something"] --> B{"Can the numbers from different sources be added into a meaningful total?"}
  B -->|"Yes"| C{"Does it only ever go up?"}
  B -->|"No"| D{"Do I already have the value when it changes?"}
  C -->|"Yes, count of events"| E["Counter or Observable Counter"]
  C -->|"No, goes up and down"| F["UpDownCounter or Observable UpDownCounter"]
  D -->|"Yes, I can record it at that moment"| G["Gauge, synchronous"]
  D -->|"No, I read it on demand"| H["Observable Gauge"]
  A --> I{"Do I need the spread of many values, like response times?"}
  I -->|"Yes"| J["Histogram"]
    

The figure above is a rough guide, not a law. Start at the top. If you care about the spread of many values, such as how long each request took, a Histogram is the right tool. Otherwise, ask whether the numbers can be added across sources. If they can, choose a Counter family instrument. If they cannot, choose a Gauge family instrument, and then ask whether you naturally have the value at the moment it changes or whether you would rather read it on a schedule.

Observable Gauge versus the synchronous Gauge

Both report a current, non-additive value. The difference is who decides when. With the synchronous Gauge, your code records the value at the moment it changes. A good example is a user dragging a volume slider: each time the slider moves, the code records the new volume level. With the Observable Gauge, the SDK reads the value on schedule. A good example is CPU utilization: the operating system always knows the value, and nobody in your code “makes it change.”

A simple rule of thumb: if the value changes because your code did something, consider the synchronous Gauge. If the value lives somewhere else and you simply want to look at it from time to time, use the Observable Gauge.

Observable Gauge versus Observable UpDownCounter

This pair confuses many people because both are observable and both go up and down. The difference is again additivity. The number of items waiting in a queue, summed across ten queue workers, tells you the total backlog. That is additive, so an Observable UpDownCounter fits. The utilization percentage of those same workers is non-additive, so an Observable Gauge fits.

Why does this matter if both just show a line on a chart? Because backends and the Collector sometimes combine data from many sources. A monitoring system that knows a metric is a sum can safely add values from many sources, while one that knows a metric is a gauge will keep each source separate or pick a last value. Choosing the right instrument tells every tool downstream how to treat your numbers.

Every time your callback reports a value, a small bundle of information is created. Knowing what is inside the bundle helps you design better metrics.

An observation is one reported value from your callback. When the SDK combines it with the instrument’s identity and the collection time, it becomes a data point in the final metric. Let us look at every piece.

Attributes: the superpower and the danger

Attributes let one metric describe many things. Instead of creating forty instruments for forty freezers, you create one instrument and give each observation an attribute that names the freezer. Dashboards can then show all freezers, filter to one, or group by building.

But every unique combination of attribute values creates a separate time series. If you add an attribute such as user_id with a million different values, you have created a million time series from a single instrument. This is called high cardinality, and it is one of the fastest ways to make a monitoring bill explode and a dashboard crawl. SDKs protect you with a default limit on how many distinct series one instrument may produce, which the specification suggests should be 2,000 by default, and extra series are folded into an overflow bucket. Treat that as a safety net, not a design target.

Now we open the hood. This chapter follows one reading, step by step, from the moment the instrument is created to the moment a number is ready to leave the application.

About aggregation and temporality

Two words from the specification are worth knowing. Aggregation means how the SDK combines many measurements into one data point. Temporality means whether a data point describes the total since the program started (cumulative) or only the change since the last report (delta). Temporality matters a lot for counters and histograms, because a total and a change are very different things. For a gauge, however, temporality does not apply. A temperature is a temperature; it is not “cumulative” or “delta.” This is one reason gauges are the simplest instrument to reason about.

Views: changing the default behavior

The SDK has a feature called views. A view is a rule that lets operators change how an instrument is processed without touching the instrumented code. With a view you can rename a metric, drop attributes you do not need, choose a different aggregation, or turn off a metric completely. This is very handy when a library you do not control produces an Observable Gauge with too many attributes. You can trim it from the outside.

What if you have more than one reader?

It is allowed to configure several readers, for example one that pushes to a cloud backend and another that serves a local Prometheus endpoint. In that case each reader performs its own collections on its own schedule, and the callback can be invoked once for each. This is why callbacks must be safe to run repeatedly and must not rely on being called exactly once per interval.

A reading does not stop at the application. It travels through several hands before it becomes a point on a chart. Let us follow the whole journey and then see how a series is born and how it fades away.

graph LR
  A["Sensor or Operating System"] --> B["Callback"]
  B --> C["SDK Last Value Aggregation"]
  C --> D["Periodic Reader"]
  D --> E["OTLP Exporter"]
  E --> F["Collector Receiver"]
  F --> G["Processors: batch, filter, add attributes"]
  G --> H["Collector Exporters"]
  H --> I["Time Series Database"]
  I --> J["Dashboards and Alerts"]
    

Start at the left. The real world holds the value: a sensor, the operating system, a counter in memory. The callback reads it. The SDK keeps the last value per attribute set. The periodic reader picks it up on schedule, and the OTLP exporter ships it to the Collector.

Inside the Collector, a receiver accepts the incoming data. Then processors work on it in order. A common processor is the batch processor, which groups many small points into bigger packages so the network is used efficiently. Another is an attribute processor, which can add or remove labels. Finally, exporters send the data to one or more destinations, such as a time series database. Dashboards and alert rules read from that database.

Life of a time series

Every unique combination of instrument name and attribute values forms one time series. For asynchronous instruments, the life of a series is controlled by what the callback reports.

• Birth. The first time the callback reports a new attribute combination, such as a newly installed freezer, a new series starts.
• Life. As long as the callback keeps reporting that combination at each collection, the series keeps growing, one point per cycle.
• Silence. If the callback stops reporting that combination, for example because the freezer was removed, the SDK does not invent a value. The series simply gets no new points.
• Staleness. Backends notice the silence. Prometheus, for example, treats a series as stale a few minutes after its last point, so it disappears from charts and instant queries.

This behavior is useful. When something is removed from the real world, its gauge quietly fades away instead of showing a false flat line. But it also means a series that disappears unexpectedly deserves attention, because “no data” might mean “the thing broke” rather than “the thing was removed.”

Where the resource fits in

Every point carries the resource of its producer. Suppose five copies of a service each report cpu.utilization. The resource, which includes the service name and a unique instance identifier, keeps the five series apart. Without it, five different values would fight over the same series and the chart would jump around randomly. Setting a proper instance identifier is a small step that prevents a large headache.

Theory is easier to hold when you see where it applies. Here are common places where an Observable Gauge is the natural choice. The scenarios are illustrative examples to build intuition, not descriptions of any particular company’s internal systems.

A design pattern is a proven way of solving a problem that keeps coming back. These are the patterns engineers use most often with Observable Gauges.

The cached reader pattern in more detail

Imagine the callback needs to ask a remote server for a number, and the server sometimes takes two seconds to answer. If the callback waits for that answer, the whole collection waits. If several callbacks do this, the collection can take longer than the interval between collections, and data will be late or lost.

The cached reader pattern separates the slow part from the fast part. A background worker fetches the number every ten seconds and stores it. The callback only looks at the stored number. The result is a collection that is always fast, at the price of data that can be up to one refresh interval old. For most monitoring needs that is a very good bargain.

No tool is perfect for every job. An honest look at the strengths and weaknesses of the Observable Gauge helps you decide when to reach for it and when to put it down.

The central trade-off: freshness versus cost

Shorter collection intervals give finer detail and quicker alerts, but they cost more. Each collection runs every callback, creates data points, sends them over the network and stores them. Halving the interval roughly doubles the data volume. Many teams start with 60 seconds, move important services to 15 seconds, and keep only a few critical gauges at 5 seconds.

Comparing the two gauge styles

When not to use an Observable Gauge

• When you need to count events, use a Counter.
• When you need to know how long things took or how big they were, and you want percentiles, use a Histogram.
• When the values can be meaningfully added across sources, use a Counter or an UpDownCounter, or their observable versions.
• When you must never miss a short spike, such as a brief overload, record each event with a Histogram or Counter instead of sampling a gauge.

These are the mistakes that appear again and again in real systems. Each one looks harmless at first, and each one can make dashboards lie or bills grow.

These habits keep Observable Gauges accurate, cheap and easy to understand. Think of them as a checklist you run through whenever you add a new gauge.

A short pre-launch checklist

• Is this value truly non-additive and read on demand? If not, reconsider the instrument.
• Does the callback finish quickly and handle its own errors?
• Does each attribute combination appear only once per collection?
• Are the unit, name and description clear?
• How many series will this create at full production scale?
• Is there a dashboard or alert that will actually use this number? A metric nobody looks at is only a cost.

Metrics look harmless because they are just numbers. But numbers and their labels can reveal more than you expect, and they can cost real money.

Security

Telemetry flows through several systems, so each hop deserves protection.

• Encrypt the transport. Use secure connections between applications, the Collector and the backend, so nobody can read or change data on the way.
• Authenticate senders. Backends and Collectors should accept data only from trusted sources, using tokens or certificates. Otherwise an attacker could flood your system with fake readings or hide a real problem.
• Least privilege inside callbacks. A callback that reads a database or a file should use credentials that can only read what it needs. A monitoring hook should never have the power to change or delete data.
• Protect the endpoint. If your application exposes a Prometheus-style scrape endpoint, do not leave it open to the whole internet. Restrict it to your monitoring network.
• Keep the Collector updated. It is a program that handles all your telemetry, so it should receive security updates like any other critical component.

Privacy

Attributes are text labels, and it is easy to put something sensitive into them without noticing: an email address, an account number, a patient name, a full web address that carries a secret token. Once it is in a metric label, it is copied to every system that stores or displays that metric, and it may live there for months.

The safest rule is simple: never put personal or secret data into metric attributes. Use stable, harmless identifiers such as a region name or a sensor group. The Collector can also scrub or drop attributes with processors, which gives you a second line of defense.

Cost

Monitoring bills usually grow with two things: the number of active time series and the number of data points stored per second. Observable Gauges affect both.

To estimate the data volume of a gauge, multiply the number of series by the number of collections per day. For example, 40 freezers collected every 15 seconds produce 40 times 5,760, which is 230,400 points per day for that one gauge. That is tiny. But 50,000 devices collected every 10 seconds produce more than 400 million points per day, and that deserves planning.

To keep costs under control, use sensible intervals, keep attributes bounded, drop gauges nobody uses, and use views or Collector processors to filter data you do not need. Some teams also keep fine-grained data for a few days and keep only averages for longer periods.

Knowing the parts is one thing. Putting them together in a sensible order is another. This chapter walks through the thinking, without any code, so that you can design a gauge confidently for your own system.

Let us use a friendly example: an online bakery that sells cakes. The kitchen has several ovens, and the owner wants to be warned if any oven drifts away from its target temperature. We will design the gauge from a blank page.

OpenTelemetry is only the first half of the story. The data must be understood by a backend. Here is how gauges are treated as they move through the most common destinations.

The data model underneath

OpenTelemetry defines a data model that describes what every metric looks like on the wire. Several metric shapes exist: sums, histograms and gauges, among others. A gauge data point in this model holds a few simple things: the attributes, the time of the reading and the value, which is either a whole number or a decimal. Notice what is missing. There is no “monotonic” flag and no temporality, because those ideas simply do not apply to a gauge. This simplicity is why gauges travel so smoothly through the pipeline.

Prometheus

Prometheus is one of the most popular open-source monitoring systems. It has its own metric types, and the one that matches an OpenTelemetry gauge is also called a gauge. When OpenTelemetry data reaches Prometheus, either through a scrape endpoint on your application or through the Collector, an Observable Gauge becomes a Prometheus gauge, and your attributes become labels.

There are a few things to watch for. Prometheus has strict rules for names and labels, so characters such as dots are converted to underscores. Some exporters add the unit to the name, so a metric named oven.temperature with the unit Celsius may appear with a suffix describing the unit. Resource information such as the service name is often moved into special labels or a dedicated info metric. Because the details depend on the exporter version and settings, the safest habit is to look at the real name in the Prometheus user interface before you write a dashboard query.

Prometheus also has its own idea of staleness. If a series stops appearing in a scrape, it is marked stale and vanishes from instant queries after a short time. This lines up nicely with how asynchronous instruments behave when a callback stops reporting an attribute combination.

The OpenTelemetry Collector

The Collector treats gauges as one more metric shape. It can receive them over OTLP, transform them, and export them to many backends. A few processors are especially relevant to gauges.

• A batch processor groups points for efficient sending.
• An attributes or resource processor adds, changes or removes labels, which is a handy way to enforce your cardinality and privacy rules centrally.
• A filter processor drops metrics you do not want to pay for.
• A memory limiter protects the Collector from running out of memory under heavy load.

The Collector also has receivers that produce gauges by themselves. The host metrics receiver is a good example: it reads machine-level numbers such as processor utilization, memory and disk figures and emits them without any code in your application. Using it for machine-level data and your own Observable Gauges for application-level data is a clean division of labor.

Cloud and commercial backends

Most commercial monitoring products and cloud monitoring services accept OpenTelemetry data, either directly over OTLP or through a vendor-supplied distribution of the Collector. They map an Observable Gauge to their own concept of a gauge or “last value” metric. The mapping is almost always straightforward, but two points deserve a check: how they treat units, and how they limit the number of distinct series per metric. Read the backend’s documentation for its limits before you launch a gauge with many attributes.

A gauge in a small program is simple. A gauge across thousands of copies of a service running in many regions raises new questions. This chapter covers the ones that matter most.

Many copies, one metric name

In a big system, the same program runs on hundreds or thousands of instances. Each instance has its own Observable Gauge with the same name, and each reports its own value. Thanks to the resource, every instance produces its own series, tagged with the service name and a unique instance identifier. The backend therefore holds one series per instance.

Now someone asks, “how busy is the service overall?” The answer is not a sum. It is a statistic: the average utilization across instances, the maximum to find the busiest instance, and perhaps a high percentile such as the ninety-fifth to see how unevenly the load is spread. This is the main reason the specification makes you choose a gauge for non-additive values. It reminds everyone down the line to combine them with statistics rather than with addition.

Choosing a collection interval at scale

With a thousand instances reporting every ten seconds, a single gauge with a handful of attributes produces millions of points per day. If you have hundreds of gauges, the numbers add up fast. Large organizations often choose a default interval of 30 or 60 seconds, then lower it only for the gauges that drive fast alerts. They also use the Collector to keep a short-term, high-resolution copy of important data while storing a smaller, averaged copy for the long term.

Keeping the pipeline healthy under load

When many applications send data to a Collector at the same time, the Collector needs enough memory and processing power. Common protections include running several Collector instances behind a load balancer, giving each a memory limiter, using queues with retry so that short outages do not lose data, and sizing batches sensibly. If the backend slows down, queues fill up. A good setup drops the oldest data in a controlled way rather than crashing the whole Collector.

Because gauges represent the present, old gauge data is usually worth less than old counter data. A temperature reading from an hour ago, delivered late, is rarely important when you already have newer readings. That makes gauges a sensible candidate for “drop when overloaded” rules, while totals and counts deserve stronger delivery guarantees.

Clocks and time

Each reading carries a timestamp. If a machine’s clock is wrong by several minutes, its gauge points will appear at the wrong place on the chart, or be rejected by a backend that does not accept data from the future or far past. Keeping all machines synchronized with a time service is a quiet but essential part of reliable monitoring.

Restarts, deployments and autoscaling

• Restarts cause a short gap in a gauge and then it resumes. Because gauges do not accumulate, there is no reset problem.
• Rolling deployments create new instances with new instance identifiers, so old series end and new ones begin. Dashboards that group by instance will show this churn, so group by service for steady charts.
• Autoscaling adds and removes instances during the day. Make sure alerts are written against statistics across the service, not against the existence of one particular instance.
• Short-lived jobs that run for less than one collection interval may never be observed at all. For batch jobs and scripts, report their results with a different approach, such as recording a final value just before they exit.

A metric that is wrong is worse than no metric, because people trust it and make decisions with it. A little testing before launch prevents most embarrassing mistakes.

Test the callback by itself

The callback is ordinary program logic, so it can be tested like any other function. Replace the real source, such as a probe or an operating system call, with a fake that returns known values, and check that the callback reports the expected numbers with the expected attributes. Also test the unhappy paths: what does the callback do if the source returns nothing, returns an impossible value, or throws an error? A good callback reports nothing for a broken source rather than a misleading zero.

Test the whole path with an in-memory reader

Most SDKs offer a metric reader that keeps collected data in memory instead of sending it away. In a test, you can configure the provider with such a reader, create your gauge, trigger a collection, and then inspect what was produced: the metric name, unit, value and attributes. This is a fast way to catch typos in names, wrong units and missing attributes without needing a real backend.

Test with realistic scale

Before a launch, estimate how many series the gauge will create in production: multiply the number of values of each attribute together, then multiply by the number of instances. Run a staging environment with similar numbers and watch the memory use of the application, the time the callback takes, and the load on the Collector and backend. Surprises are much cheaper to discover in staging.

Test the alert, not just the number

The final test is whether the gauge actually helps someone act. Cause a controlled problem in a test environment, such as turning down an oven target in a simulator or filling a test disk, and confirm that the chart moves and the alert fires and reaches the right person. Many teams discover at this stage that an alert was sent to a channel nobody reads, or that its threshold was set in the wrong unit.

A validation checklist

Once your gauges are flowing, two new jobs begin: using the data well, and making sure the data itself stays healthy.

Reading gauge data on dashboards

Because a gauge shows a current level, it is usually drawn as a simple line chart. A few habits make these charts more useful.

• Compare across sources with statistics, not sums. To see how a group of servers is doing, show the average, the maximum and sometimes the minimum, instead of the total.
• Use the maximum for danger. If one freezer out of forty is too warm, the average still looks fine. The maximum tells you the truth.
• Smooth over time with care. Query tools let you take the average or maximum of a gauge over a window of time, for example the highest value in the last five minutes. This helps you catch spikes that a single reading would miss, as long as readings are frequent enough.
• Show the threshold. Draw a horizontal line for the limit you care about, so the chart answers “are we safe?” at a glance.

Alerting on gauges

Gauges are natural candidates for threshold alerts. A good alert has three parts: a clear condition, a duration, and a severity. “CPU utilization above 90 percent for ten minutes” is much better than “CPU utilization above 90 percent right now,” because a short burst is normal but a long plateau is a problem.

Another valuable alert is for missing data. If a gauge that normally reports every 15 seconds suddenly goes silent, that is an incident in itself. It can mean the application crashed, the callback is failing, or the pipeline is broken. Without a “no data” alert, a silent gauge looks just like a healthy system with nothing to report.

Monitoring the monitoring pipeline

The OpenTelemetry Collector exposes its own internal metrics, such as how many data points it received, how many it sent, how many it refused and how full its queues are. Watching these tells you whether your telemetry is arriving safely. SDKs can also log errors from callbacks and exporters. Make sure those logs go somewhere a human can find them.

A troubleshooting walk-through

Suppose a gauge you expect to see is missing or looks wrong. Work from the closest point to the farthest.

Short, direct answers to the questions beginners ask most often.

You started with a car dashboard and ended with a full picture of how an Observable Gauge works from callback to chart. Here is everything that matters, in one place.

The Observable Gauge is the OpenTelemetry instrument for “what is the value right now?” You register a small callback once. The SDK calls it on every collection, the callback reads the current value from the real world, and the value travels through the reader, the exporter and usually the Collector to your dashboards and alerts. Because it only looks when asked, it adds no cost to your application’s busy paths, but it also means it sees a sample of reality rather than a full recording. Choosing it correctly, keeping callbacks fast, keeping attributes bounded and always setting clear units are the habits that make it a dependable tool.