Before we talk about labels, we need to understand what is being labeled.

Every running system produces behavior that can be counted and measured. How many requests arrived? How long did they take? How much memory is in use? How many items are waiting in a queue? Engineers collect these measurements continuously and watch them over time, because a change in a number is often the first sign that something is wrong. These continuously collected numbers are called metrics.

OpenTelemetry is an open-source, vendor-neutral standard for producing and moving telemetry data. It handles three main kinds of data, called signals: traces, metrics, and logs. Traces follow individual requests, logs record individual events, and metrics summarize behavior as numbers over time. Metrics are the cheapest and fastest of the three to store and to read, which is why they power most dashboards and alerts.

Why Raw Totals Are Not Enough

A single total for the whole system is rarely enough. Knowing that your service handled ten thousand requests tells you almost nothing if you cannot tell which kinds of requests, which customers, or which software versions were involved. When an error rate climbs, the first question is always “where?” Is it all requests, or only one route? Is it one region or all of them? Metric attributes exist to let a single metric answer those questions without creating hundreds of separate metrics.

A metric is more than a number; it has a name, a unit, a type, and a history.

In OpenTelemetry, a metric is a named stream of measurements about one aspect of your system. Your code does not usually create metrics directly. Instead, it creates an instrument, which is a small tool for recording measurements of a particular kind. The instrument’s type determines how the numbers are combined, and the library then turns the recorded measurements into the metric that is eventually exported.

The Parts of a Metric

The Common Instrument Types

Some instruments are asynchronous, also called observable. Instead of your code calling them every time something happens, the library calls a function you provide at regular intervals to read the current value, which suits things like memory use or queue length that exist all the time rather than happening at moments.

A metric attribute is a label that says which group a measurement belongs to.

A metric attribute is a key-value pair attached to a measurement to describe the circumstances in which it was taken. When your code records that a web request took a certain amount of time, it can also say which method the request used, which route it matched, and what kind of response it produced. Those descriptions are the attributes. The metrics system then keeps separate running totals for each different combination of attribute values.

This is the crucial difference between metric attributes and the attributes on traces. On a trace, each span is stored individually, so attributes simply decorate it. On a metric, individual measurements are not kept; they are combined into summaries. The attributes decide how that combining happens: measurements with the same attribute values are merged together, and measurements with different values are kept apart.

Why Attributes Are So Useful

With attributes, one metric can answer many questions. A single request-duration metric, labeled by route and status, can show the overall speed of the service, the speed of one route, the speed of only failed requests, or a comparison between routes. Without attributes you would need a separate metric for every combination, which would be unmanageable.

Who Sets the Attributes?

In most real applications, metric attributes come from two sources. Automatic instrumentation libraries wrap popular web frameworks, database drivers, and messaging clients, and they record standard metrics with standard attributes without you writing any measurement code. Manual instrumentation is the code you write yourself to measure things specific to your business, such as the number of orders placed or the time taken by a pricing calculation. For these you decide which attributes to include.

A healthy approach is to rely on automation for the broad technical metrics and use manual metrics sparingly, with a few carefully chosen attributes. Because anyone on the team can add an attribute with a single line of code, it is worth agreeing on a short review habit: whenever a new metric attribute is proposed, someone asks how many values it can take.

Every unique combination of attribute values becomes its own stream of numbers.

A time series is a sequence of numbers recorded at regular moments, such as a request count measured every minute. When a metric has no attributes, there is exactly one time series. When it has attributes, there is one time series for every different combination of attribute values that actually occurs. This simple rule is the most important idea in this entire guide, because it explains both the power and the danger of attributes.

flowchart LR
  M["One metric: request count"] --> A["method=read, status=success"]
  M --> B["method=read, status=error"]
  M --> C["method=write, status=success"]
  M --> D["method=write, status=error"]
        

A Worked Example

Suppose a metric counts web requests and carries two attributes: the method, which can be read or write, and the result, which can be success or error. Four combinations are possible, so four time series are created. If you add a third attribute for the region, with three possible regions, the number of series becomes twenty-four. Each new attribute multiplies the total, it does not simply add to it.

Only Real Combinations Count

The multiplication gives a worst case. In practice only combinations that actually occur create a series. If a rarely used route never returns a particular error, that combination never appears and costs nothing. Even so, systems that grow in traffic and variety tend to see more and more combinations over time, so the worst case matters when planning.

The rules are simple and the same ones used by attributes across all signals.

Keys

An attribute key is a non-empty piece of text naming the label. For a given measurement, each key appears only once. Keys are case sensitive, so a key spelled with a capital letter is a different key from the same word in lowercase, and the two would create separate series. By strong convention, keys are lowercase, with dots separating levels of a namespace and underscores separating words inside a level. Following this convention makes your attribute names look familiar to everyone who reads them.

Values

Attribute values can be text, true-or-false values, whole numbers, decimal numbers, or arrays of one of those types. In practice, metric attributes are almost always short pieces of text or small whole numbers, because they need to be grouped and compared. A value should come from a small, predictable set, which is the central rule we will return to in the cardinality chapters. A value that can take millions of different forms is not a good metric attribute.

Where the Unit and Name Belong

Beginners sometimes place the metric’s own unit or name inside an attribute. For example, a metric measuring memory should carry its unit in the unit field, not as an attribute that says “bytes”. Likewise, the thing being measured belongs in the metric name. Attributes are for describing the circumstances of the measurement, not for repeating what the metric already says.

Attributes appear at several levels, and each level has a different job.

In OpenTelemetry, attributes appear at more than one level. Understanding the levels prevents the common mistake of putting information at the wrong place. The three levels that matter for metrics are the resource, the instrumentation scope, and the individual data point.

flowchart TD
  R["Resource: service name, version, environment"] --> S["Scope: instrumentation library"]
  S --> M["Metric: request duration"]
  M --> D1["Data point with attributes: read, success"]
  M --> D2["Data point with attributes: write, error"]
        

Metric Attributes versus Span Attributes

Span attributes decorate individual operations in a trace, and each span is stored separately. Metric attributes, in contrast, define groups. This makes the cost of a high-variety value very different. A span attribute with a million possible values is fine: it just makes a million spans searchable. A metric attribute with a million possible values creates a million time series, which can overwhelm a monitoring system. We will explore this difference in detail soon.

Agreed names for common situations make metrics comparable across tools and teams.

If every team invented its own names for the same ideas, no dashboard could work across services. One team might call the web method “verb”, another “method”, and another “http_method”. To prevent this, the OpenTelemetry community maintains semantic conventions, a published list of standard names for metrics and their attributes in common situations such as web servers, databases, messaging systems, and operating systems.

Using standard names has a large practical benefit. Observability tools that understand the conventions can automatically recognize your metrics and offer ready-made dashboards and alerts. Automatic instrumentation libraries for popular frameworks already emit metrics that follow these conventions, so you often get standard attributes without writing any code.

Notice the Careful Choices

The standard attributes for metrics were chosen to keep variety low. For example, the standard web metrics use the route pattern, such as the template of a page address with placeholders for variable parts, rather than the full address with real identifiers inside. They use the numeric status code, which has a small fixed set of values, rather than the full response message. These choices are deliberate protection against runaway series counts, and they are a good model for your own custom attributes.

Standards Evolve

The conventions are living documents. Names have been refined over time for clarity and consistency, and some areas are more mature than others. Older instrumentation may emit older names, while newer versions emit the newer names. Many libraries provide a transition option that emits both for a while, so dashboards and alerts can be migrated safely. Always check the stability label of the convention you rely on.

A good metric attribute has a small, known, and stable set of possible values.

The most important rule for choosing metric attributes is the bounded-set rule: the attribute’s possible values should come from a small, finite set that you could write down in advance. If you can list all the values on a single page, the attribute is probably safe. If the list could grow forever, or depends on user input, the attribute is dangerous.

Questions to Ask Before Adding an Attribute

Turning an Unbounded Value into a Bounded One

Sometimes a value you care about is naturally unbounded, but a simplified version of it is perfectly safe. The trick is to group the raw value into a small number of categories before recording it. A full page address becomes a route pattern. An exact response time becomes a histogram bucket, which the metric does for you. An exact number of items in a basket becomes a size band such as small, medium, or large. A customer name becomes a plan tier. A raw error message becomes a short error category with a handful of possible values.

This grouping step is a small piece of design work, but it pays off enormously: you keep most of the insight at a tiny fraction of the cost. When you cannot find a sensible grouping, that is a strong signal that the detail belongs on a trace or in a log instead.

Good Attributes

Dangerous Attributes

Cardinality is the count of unique series, and it is the main thing that makes metric systems slow or expensive.

Cardinality means the number of distinct values or distinct combinations. For a single attribute, it is the number of different values. For a whole metric, it is the number of distinct time series produced by all attribute combinations together. Low cardinality means a manageable number of series, while high cardinality means a very large number.

Why High Cardinality Is Harmful

Every time series needs memory in your application, network capacity to send, space in the backend to store, and processing power to query. When series counts grow into the hundreds of thousands or millions, applications use more memory, exports become heavy, queries slow down, and monthly bills climb sharply, since many monitoring services charge by the number of active series. In severe cases the monitoring system itself can fail, at the worst possible time, when you most need it.

The Danger of Slow Growth

Cardinality problems rarely announce themselves. A new attribute may look harmless in testing, where only a handful of values appear, and then explode in production where real users create millions of combinations. A problem may also build slowly as new versions, new pods, or new customers each add a few more series. Teams often discover the issue only when a bill arrives or a dashboard stops loading. For this reason, monitoring the number of series per metric is a healthy habit.

Following one measurement from your code to a dashboard shows where attributes are used and where they can be controlled.

A measurement begins as a single call in your application and eventually becomes a point on a chart. Along the way it passes through several stages, and attributes play a role at each one. Knowing the stages helps you decide where to fix problems.

sequenceDiagram
  participant App as Application code
  participant SDK as OpenTelemetry library
  participant Col as Collector
  participant Back as Metrics backend
  participant User as Engineer
  App->>SDK: Record a value with attributes
  SDK->>SDK: Group by attributes and combine
  SDK->>Col: Export summarized data on a schedule
  Col->>Col: Receive, process, export
  Col->>Back: Deliver the data
  User->>Back: Chart, filter, and group by attributes
        

Stage 1: Recording

Your code, or an automatic instrumentation library, calls an instrument with a value and a set of attributes. For a synchronous instrument this happens every time the event occurs. For an asynchronous instrument, a callback function reports values and attributes at regular intervals.

Stage 2: Aggregation in the Library

The OpenTelemetry library does not store each individual measurement. Instead it keeps one running summary per unique set of attributes. A counter keeps a running sum, a histogram keeps counts per range, and a gauge keeps the latest value. This is where the multiplication of series actually happens: each new attribute combination creates a new summary in memory.

Stage 3: Export on a Schedule

On a regular interval, often every minute or so, the library collects all its summaries and exports them. The metric carries the attribute sets, so receivers know which group each number belongs to. Applications can report either cumulative totals since the start or only the change since the last report, a choice called temporality. With change-only reporting, groups that were idle in an interval do not need to be sent, which can lower cost for attribute sets that appear only occasionally.

Stage 4: Processing in the Collector

The Collector can inspect, rename, add, or remove attributes on the way through. This is a powerful place to fix mistakes centrally, which the next chapters explain.

Stage 5: Storage and Querying

Finally, the backend stores each series and lets people ask questions. Filtering by an attribute narrows the chart to a group, and grouping by an attribute draws one line per value. Ignoring an attribute combines the groups back together.

The OpenTelemetry library gives you tools to shape and protect metrics before they leave your application.

You do not have to accept every attribute that instrumentation code provides. The OpenTelemetry library offers several controls that let you shape metrics without changing the code that records them. This is especially valuable when instrumentation comes from a third-party library that you cannot easily change.

Views in Everyday Terms

Think of a view as a set of instructions placed between your code and the exporter. If a library records a metric with six attributes but you only care about two, a view can keep those two. The measurements that differed only in the other four attributes are merged back together, shrinking the number of series dramatically. The code that records the measurements does not change at all.

Cardinality Limits and the Overflow Series

Recent versions of OpenTelemetry libraries include a built-in safety net. Each metric has a maximum number of distinct attribute combinations it will track. If new combinations keep arriving beyond that cap, the library stops creating new series and instead adds the extra measurements to a single special overflow series, marked with a dedicated attribute that signals overflow. You do not lose the totals, but you lose the detail, and the presence of the overflow series is a loud warning that an attribute is too varied. Default limits and exact behavior depend on the language and version, so check the documentation of the library you use.

The Collector is the central workshop where attributes can be added, renamed, removed, or merged.

The OpenTelemetry Collector is a standalone program that sits between your applications and your metrics backend. Its processors can adjust metrics in flight, and attributes are among the most common things to adjust. Because the Collector is central, you can fix or improve attributes for many services at once, without redeploying any of them.

The Processors Behind These Tasks

Several processors work with metric attributes. Some offer straightforward actions such as inserting, updating, and deleting attributes. A more flexible transform processor uses a small expression language to make conditional changes to metric data points. A processor designed for metric transformations can also aggregate away labels, which is the step that merges series after an attribute is removed. A filter processor can drop metrics that match given conditions. The right choice depends on whether you only need to change labels or also need to combine the numbers afterward.

Where to Control Attributes: Library or Collector?

Both places work, and many teams use both. Controlling attributes in the library saves memory in your application and network traffic, because unwanted series are never created. Controlling them in the Collector is easier to manage centrally and does not require redeploying applications. A sensible approach is to use libraries and views for the attributes you know are harmful, and the Collector as a flexible safety net for everything else.

What happens to attributes when they reach the storage and querying layer?

Once metrics arrive at a backend, attributes become the handles you use to ask questions. Most backends call them labels or tags, and treat each unique combination as a separate series. How attributes appear and how they are charged for varies between products, so it is worth understanding the common patterns.

Naming Changes Along the Way

Some popular backends have stricter naming rules than OpenTelemetry. For example, systems in the Prometheus family do not allow dots in names, so dots are typically converted to underscores. Metric names may also receive suffixes that reflect their unit or type. This conversion is usually automatic, but it means the name you see in a dashboard may look slightly different from the name in your code. Knowing this prevents confusion when searching for a metric.

Resource Attributes Are Handled Separately

Because resource attributes are the same for all metrics from a service, some backends store them separately instead of copying them onto every series. In Prometheus-style systems, they are commonly kept in a separate informational metric, and a few key resource attributes, such as the service name, may be promoted into ordinary labels so they can be used for filtering. Check how your backend and Collector are configured, so that you know which resource attributes are available as labels when you build charts.

Querying with Attributes

In practice, you do three things with attributes when you query. You filter, which narrows a chart to a single group, such as one region. You group, which draws a separate line for each value of an attribute, such as one line per route. And you aggregate, which ignores an attribute and sums or averages across it, such as the total across all regions. Because the backend can always combine finer groups into coarser ones, recording a useful attribute gives you more flexibility than leaving it out.

When a metric attribute cannot carry a unique identifier, there is a smarter way to link a number to a specific example.

Metrics are summaries, so they cannot tell you which specific request caused a spike. That is where traces shine, but how do you get from a suspicious chart to the right trace? Putting a trace identifier in a metric attribute would be a cardinality disaster, because every identifier is unique. OpenTelemetry offers a better solution called an exemplar.

An exemplar is a small sample attached to a metric data point. It carries one actual measured value along with the trace and span identifiers that were active when it was recorded, and sometimes a few extra attributes. Instead of creating a new series for each identifier, the metric simply keeps a few example measurements per data point. A chart can then show a dot on a spike, and clicking it takes you straight to a real trace that contributed to that spike.

flowchart LR
  A["Slow request recorded"] --> B["Metric data point: request duration"]
  A --> C["Exemplar: sample value plus trace link"]
  B --> D["Dashboard chart"]
  C --> D
  D --> E["Click the dot to open the trace"]
        

Why This Solves a Real Problem

Exemplars let you keep metrics low in cardinality while still offering a path to detail. You get the best of both signals: the cheap, fast, aggregated overview of metrics and the rich, individual detail of traces. They depend on your application using both signals together, and on a backend that can display them, but where they are available they are an elegant answer to the temptation of high-cardinality metric attributes.

Seeing attributes in familiar situations makes the ideas concrete.

Attributes make metrics far more useful, but every label has a price.

Thinking About Ownership

Metrics that many people depend on need an owner. When a dashboard, an alert, and a monthly report all rely on the same attribute, a casual change can quietly break all three. Nominate a person or team to own each important metric, record what each attribute means and which values it can take, and announce changes in advance. Treat this documentation as part of the metric itself, because a label that nobody can explain soon becomes a label that nobody trusts.

Finding the Balance

The goal is not the most attributes, but the right few. Too few, and your metrics cannot answer basic questions such as which route is failing. Too many, and your costs climb and your system slows. Start with the attributes defined by semantic conventions, add a small number of business categories that answer your most common questions, and grow carefully. Remember that traces and logs are available for the detail that metrics cannot afford to carry.

A Simple Starting Recipe

If you are unsure where to begin, try this. First, make sure each service reports its name, version, and environment as resource attributes. Second, enable automatic instrumentation so that web, database, and messaging metrics arrive with their standard attributes. Third, add at most a few business attributes with small fixed value sets, such as plan tier and region. Fourth, set cardinality limits or views as safety nets. Fifth, review the series count per metric each month. This modest recipe delivers most of the value at a predictable cost.

A handful of habits prevent most problems.

Habits Worth Building

Mistakes to Avoid

Quick answers to the questions beginners ask most often.

The idea is the same: a name paired with a value that splits a measurement into groups. Other systems use the words label, tag, or dimension. OpenTelemetry uses the word attribute.

Span attributes describe one stored operation. Metric attributes decide which group a combined measurement belongs to, so each new value can create a new time series. That is why metric attributes must have small, bounded sets of values.

A sequence of values recorded over time for one specific combination of metric and attributes. A metric with attributes produces one time series for every combination that actually occurs.

The number of distinct values or combinations. For metrics, it usually means the number of distinct time series. High cardinality makes metrics expensive and slow.

There is no fixed number. Think in terms of the product of values: if the multiplied total stays in the hundreds or low thousands, you are usually safe. A few attributes with few values each is better than one attribute with unlimited values.

It is strongly discouraged, because every user creates a new series. Put user identifiers on traces and logs, and use exemplars to link a metric to specific traces.

A rule in the OpenTelemetry library that changes how a metric is exported without changing the recording code. Views can rename metrics, change how they are summarized, and keep only chosen attributes.

A safety net in recent library versions. When a metric exceeds its limit of distinct attribute combinations, extra measurements are folded into one special series marked as overflow. Seeing it means an attribute has too many values.

Resource attributes identify the producer, so different producers naturally produce different series, but they are shared by all metrics from that producer and are often stored separately. Attributes that change often, such as a pod name in a fast-changing cluster, can still add to the total number of series over time.

Some backends restrict characters in names, so dots are converted to underscores, and unit or type suffixes may be added. The data is the same; only the spelling is adjusted.

A metric attribute is a key-value label attached to a measurement that decides which group the measurement belongs to. Because metrics combine measurements instead of storing each one, every unique combination of attribute values becomes its own time series, which makes attributes both the source of a metric’s power and the main driver of its cost. Choose attributes from small, bounded sets, reuse the standard names, record the same keys consistently, and keep identifiers for traces and logs, linking them to metrics through exemplars. Use views, limits, and Collector processors as safety nets, watch the series count, and your metrics will stay useful, fast, and affordable.