Before we talk about time, we need a shared vocabulary for how OpenTelemetry turns application behaviour into numbers.

OpenTelemetry, often shortened to OTel, is an open standard for producing and moving telemetry data out of software. Telemetry comes in several signals: traces describe the journey of a single request, logs describe individual events, and metrics describe numbers that summarise behaviour over time. Metrics are the cheapest of the three to store, which is why teams use them for dashboards and alerts. Aggregation temporality lives entirely inside the metrics signal.

Your application does not talk to a metrics backend directly. Instead, your code uses an API to record what happened, and a separate SDK decides how to summarise those recordings and ship them out. This separation is deliberate. Library authors can add instrumentation using only the API, and the people who run the application decide later, through SDK configuration, where the data goes and in what form.

One more idea matters here. Each unique combination of a metric name and attribute values is called a time series. If your request counter has an attribute for the route and the route can be one of ten values, you have ten series. The SDK keeps one running summary per series, and as we will see later, whether it keeps that summary forever or forgets it regularly is directly controlled by temporality.

It is the rule that says which stretch of time a reported metric value covers.

Whenever the SDK summarises measurements, two separate questions must be answered. The first question is what mathematical operation to use: add the values, remember the last value, or count how many fell into each bucket. That is the aggregation. The second question is over which period the operation applies: everything since the application started, or only since the previous report. That is the temporality. The word comes from “temporal”, meaning related to time.

The OpenTelemetry data model defines exactly two answers to the second question. In cumulative temporality, every report covers the period from a fixed starting moment up to the present, so each new report contains everything the earlier ones did, plus the latest activity. In delta temporality, every report covers only the period since the previous report, so each report stands alone and the starting moment moves forward each time.

It also helps to know that temporality is a property of the data on the wire, not of the instrument in your code. Your developers call the same add method on a counter regardless of which temporality the operators chose. That is a major design win: instrumentation code never needs to change when you move from one backend to another that prefers a different style.

Two ways of writing the same truth on the report sheet.

Why two flavors exist at all

Different tools in the observability world grew up with different habits. Prometheus, one of the most influential metrics systems, stores ever-growing counters and expects the query layer to work out rates. Other systems, especially ones born in the era of StatsD, store the number of events per interval and simply add them up when you query. OpenTelemetry wanted to speak to both worlds without forcing everyone into one style, so the data model includes a field that declares the temporality of each metric stream, and the SDK can be configured to produce either.

The same activity, written two ways

Suppose a service handles 120 requests in the first minute and 90 in the second. A cumulative stream reports 120 and then 210. A delta stream reports 120 and then 90. A person reading the delta stream must add to find the total. A person reading the cumulative stream must subtract to find the activity of one minute. Neither is more correct. They are two lenses on the same facts, and each lens makes certain questions easier and certain failures more painful.

Every data point carries two clocks, and together they tell you everything.

Inside a metric data point there are two timestamps. The start time marks the beginning of the period the point describes. The time, sometimes called the end time, marks the moment the value was captured. The relationship between these two timestamps across successive points is what makes temporality visible. You do not need to guess the flavor by looking at numbers, because the pattern of the timestamps reveals it.

flowchart TB
  subgraph CUM["Cumulative: the start time stays fixed"]
    direction LR
    C1["Point 1: T0 to T1"] --> C2["Point 2: T0 to T2"] --> C3["Point 3: T0 to T3"]
  end
  subgraph DEL["Delta: the start time moves forward"]
    direction LR
    D1["Point 1: T0 to T1"] --> D2["Point 2: T1 to T2"] --> D3["Point 3: T2 to T3"]
  end
        

In the cumulative picture, every window begins at the same moment, usually the time the SDK first saw the series or the time the process started. Each window is longer than the last, because it stretches further into the present. In the delta picture, windows sit side by side like tiles on a floor. The end of one tile is exactly the beginning of the next, with no overlap and no gap.

Why the start time matters so much

The start time is what lets a consumer detect a counter reset. If a cumulative series reports a total of 5,000 and then a total of 40, a careful consumer looks at the start time. If the start time jumped forward, the process restarted and the counter began again from zero, so the 40 is new activity and nothing was lost. Without a trustworthy start time, the consumer would see a mysterious drop and could not tell a restart from a bug. Prometheus-style systems infer resets purely from a value going down, while the OpenTelemetry model gives the extra hint of an explicit start time.

Reading a data point step by step

Not every kind of metric has a “since when” question to answer.

The OpenTelemetry metrics data model defines several metric types, and temporality applies only to some of them. The ones that accumulate something over a period carry a temporality, and the ones that simply capture a moment do not. Understanding this split prevents a lot of confusion, because people often search for a temporality switch on gauges and cannot find one.

The special case of up-down counters

An up-down counter measures something that can rise and fall, such as the number of items currently in a queue or the number of active connections. It is still a sum, so it still has a temporality. But a delta stream of an up-down counter is awkward to use, because each point only says how much the value moved, and you need the full history of moves to know the current level. For this reason, several of the standard preference modes keep up-down counters cumulative even when everything else is delta, a detail we will return to in Chapter 8.

Let us follow one counter through five minutes and one restart.

Imagine a checkout service that counts every order request it handles. The SDK exports once per minute. In minute one the service handles 120 requests, in minute two 90, in minute three 150, in minute four none at all because it is the middle of the night for most customers, and in minute five 60. Here is how the same story looks in the two flavors.

The quiet minute

Minute four is instructive. In cumulative temporality the series keeps reporting 360, a flat line that clearly says “nothing new happened”. In delta temporality, SDKs commonly skip series that had no activity in the window, so the point may simply not be sent. A dashboard that treats a missing delta point as an unknown rather than a zero can show a gap, while a dashboard that treats it as zero shows the truth. This is why delta consumers must be written with care about missing points.

The restart

Now suppose the process restarts between minute five and minute six and handles 40 requests in minute six. The delta stream just reports 40, and nothing special happens, because delta streams have no running total to lose. The cumulative stream reports 40 with a new, later start time. A consumer sees the total drop from 420 to 40, notices the start time moved, and correctly concludes the counter was reset. If the consumer is naive and simply subtracts consecutive values, it would compute a negative number of requests, which is why proper rate functions include reset handling.

The same idea for histograms

For a latency histogram, the story is identical but with bucket counts instead of a single number. A cumulative histogram reports how many requests have ever fallen into each bucket since the start. A delta histogram reports how many fell into each bucket during the last minute only. If you want to know whether latency got worse in the last minute, the delta histogram answers it directly, whereas the cumulative one needs you to subtract the previous snapshot bucket by bucket before you can see the recent shape.

A metric passes through several hands before it is stored, and each hand has a say.

flowchart LR
  A["Application code records measurements"] --> B["SDK instruments"]
  B --> C["Aggregation per series"]
  C --> D["Metric reader applies temporality preference"]
  D --> E["OTLP exporter"]
  E --> F["Collector with optional conversion"]
  F --> G["Backend storage and queries"]
        

The first and most important decision point is the metric reader inside the SDK. Each reader is configured with a temporality preference, which is a rule that maps every instrument type to either cumulative or delta. Because readers are separate objects, one SDK can have several readers with different preferences, for example one that feeds a local Prometheus endpoint with cumulative data and another that exports to a remote vendor with delta data. The aggregation is computed independently for each reader.

The second decision point is the exporter configuration. In practice, most teams set the preference through an environment variable on the OTLP metrics exporter, so operators can change it without rebuilding the application. The third point is the Collector, which can receive data in one temporality and convert it to another using processors. The fourth point is the backend, which decides what it can accept and what it does with the data.

The standard configuration gives you three ready-made recipes.

OpenTelemetry has two families of instruments. Synchronous instruments are called by your code at the moment something happens, such as a counter increment when a request arrives. Asynchronous instruments, also called observable instruments, are read by a callback that the SDK triggers at collection time, such as a gauge reading current memory or a counter reading a total maintained elsewhere. This split matters for temporality, because it changes what the SDK knows and what it has to remember.

The specification and the SDKs recognise three named preferences for the OTLP exporter, usually selected through an environment variable on the metrics exporter. Cumulative is the default, and it reports every instrument cumulatively. Delta reports delta for counters, observable counters, and histograms, while keeping up-down counters and observable up-down counters cumulative. The low-memory preference reports delta only for synchronous counters and histograms, keeping everything else cumulative.

Why the low-memory mode exists

Observable counters present an interesting puzzle. Your callback reports the current absolute total, for example “this process has read 9,800,000 bytes so far”. To turn that into a delta, the SDK has to remember the previous reading for every series and subtract. That remembered state is exactly the memory cost that delta temporality is supposed to eliminate. So the low-memory preference leaves observable instruments cumulative and applies delta only where the SDK can genuinely drop state after each export.

Why up-down counters stay cumulative

As described earlier, a delta up-down counter only tells you the movement, so consumers need every point to reconstruct the level. Losing one point permanently corrupts the level. Keeping these instruments cumulative avoids that fragility, and that is why even the delta preference leaves them alone.

The safe, forgiving default, and the price it charges in memory.

With cumulative temporality, the SDK keeps a running summary for every series from the moment that series is first seen. At each export, it reads the current summary and sends it without clearing anything. The next export sends a summary that includes everything from before. This approach is conceptually simple, and it matches the way many classic metrics systems already work, which is why it is the default.

The strength: resilience to loss

Networks drop packets, Collectors restart, and exporters sometimes time out. In a cumulative stream, a lost report is a minor annoyance. The next report that arrives contains the full total, so the long-term picture is intact. The only thing lost is the resolution of the chart during the gap. Operators who value dependable totals, for billing-like counters or service level calculations, often rely on this property.

The strength: simple, stateless consumers

A backend that receives cumulative points does not need to remember anything about the sender to store a value. It can write each point independently. If many replicas of a service send data, each replica produces its own series, and the backend can combine them at query time. There is no need to ensure that all data from one sender lands on one machine.

The cost: memory that never shrinks

Because the total since the beginning must stay correct, the SDK can never forget a series. If a metric has an attribute that takes many different values over time, such as a user identifier or a temporary pod name, each new value creates a new series that stays in memory until the process ends. SDKs usually impose a cardinality limit and fold excess series into an overflow bucket, but that limit is a safety net, not a design goal.

Lean and current, but unforgiving when something goes missing.

With delta temporality, the SDK accumulates measurements during a reporting window and, when it exports, sends the window’s summary and then resets its state. The next window starts from zero. The SDK therefore only needs to hold information about series that were active in the current window, and series that went quiet cost nothing at all.

The strength: bounded memory

The amount of memory in use tracks the number of series active in a single window, not the number of series ever seen. For short-lived or highly dynamic workloads, this can be the difference between a healthy process and one that slowly bloats. The SDK can also stop sending idle series, which reduces network traffic and storage in the backend.

The strength: easy window questions

Because each point is already a per-window amount, questions like “how many errors occurred in the last five minutes” turn into simple additions of the last few points. Histograms are even more attractive, because each point shows the true distribution of that window without any subtraction.

The cost: fragility and statefulness downstream

A lost delta point is permanently lost. No later report will repair the total, and the long-term sum quietly under-counts. Delta data also pushes complexity downstream, because anything that wants a running total, such as converting back to cumulative for a Prometheus-style store, must remember the previous total and see every point in order. When several replicas report into a pipeline of several Collector instances, the points of a single series must all reach the same instance or the arithmetic breaks.

Temporality is one of the quiet levers behind your observability bill.

Cardinality is the number of distinct series a metric produces. It is the product, roughly, of the number of distinct values each attribute can take. A request counter with a route attribute of 20 values and a status attribute of 5 values can produce up to 100 series. Add a customer identifier with 50,000 values, and you may have five million. Every series costs memory in the SDK, bytes on the wire, and storage and indexing effort in the backend.

Temporality interacts with cardinality in three ways. First, inside the SDK, cumulative keeps all series forever while delta keeps only recent ones. Second, on the wire, cumulative re-sends idle series at every interval, while delta does not. Third, in the backend, both temporalities create the same number of distinct series for the same active workload, so delta is not a cure for a cardinality explosion, only a way to limit its lingering effects.

Churn: the hidden multiplier

Churn means series appear and disappear frequently, as happens when containers are replaced often and each carries a unique name in its attributes. Under cumulative temporality in a long-running process, churn from within the process fills the SDK memory. Under delta, churned series simply fade out after their last active window. In both cases, the backend still sees a continuous stream of brand new series, which is why resource attributes that change on every deployment should be chosen carefully.

The best setting is often decided by the system on the receiving end.

A metric stream is only useful if the destination understands it. Backends differ widely in what they accept, and the temporality chosen at the source must match the destination’s expectations or be converted along the way. The following examples describe common patterns in the industry. Because products change quickly, treat these as orientation and confirm the current behaviour in each vendor’s own documentation.

When the sender and the receiver disagree, a translator can sit in the middle.

The OpenTelemetry Collector is a standalone service that receives, processes, and exports telemetry. Its contrib distribution includes processors that convert between the two flavors. One converts cumulative data into delta, and another converts delta data into cumulative. These let a platform team standardise on one temporality inside the company while serving destinations that need the other.

sequenceDiagram
  participant S as SDK
  participant C as Conversion processor
  participant B as Backend
  S->>C: Cumulative total 120
  Note over C: First point, remember 120
  S->>C: Cumulative total 210
  C->>B: Delta 90
  S->>C: Cumulative total 360
  C->>B: Delta 150
  S->>C: Cumulative total 40 after restart
  Note over C: Value dropped, treat as a reset
  C->>B: Delta 40
        

Cumulative to delta: remember the last value

To convert cumulative into delta, the processor stores the last total it saw for each series and subtracts it from each new total. This is straightforward, but it has two awkward moments. The very first point of a series has nothing to subtract from, so processors usually treat it specially, for instance by dropping it or by using the start time to decide. And when a total drops, the processor must decide whether that was a reset and how to account for it.

Delta to cumulative: add up everything, in order

To convert delta into cumulative, the processor keeps a running total per series and adds each delta to it. This demands that the processor sees every point of a series, in order, and that it never loses its state. If the Collector restarts, the running total restarts, and downstream consumers will see a reset that is an artefact of the pipeline rather than of the application.

The same question needs a different approach depending on the flavor.

Most mistakes with temporality do not happen in the SDK. They happen in dashboards and alerts, where someone writes a query that suits one flavor and runs it against the other. The key habit is to know, for every metric you chart, which temporality is stored, and to use the matching approach.

The classic trap: charting the raw value

If you chart a raw cumulative counter, you see a line that climbs steadily and occasionally collapses to zero. It looks dramatic but tells you nothing about current load. The useful chart is the rate. Conversely, if you apply a rate function to a delta stream that your backend never converted, you may get meaningless results, because the function expects a growing series.

The subtle trap: summing across replicas

With cumulative data from many replicas, the right sequence is first to compute a rate per replica, which handles each replica’s own resets, and then to add the rates together. If you add the raw totals across replicas first and then compute a rate, a single replica restarting makes the combined total dip, and the rate function misreads it as a reset of the entire fleet. This is a well-known pitfall in the cumulative world, and delta temporality avoids it naturally.

A short set of questions that settles most real-world cases.

flowchart TD
  A["Where are the metrics going?"] --> B{"Does the backend require cumulative?"}
  B -->|Yes| C["Use cumulative"]
  B -->|No| D{"Short lived or serverless workload?"}
  D -->|Yes| E["Prefer delta"]
  D -->|No| F{"Very high cardinality or heavy churn?"}
  F -->|Yes| G["Consider delta or low memory, and filter attributes"]
  F -->|No| H["Cumulative is a safe default"]
        

A telemetry pipeline is itself a production system and deserves care.

Watching the pipeline

Because temporality mistakes tend to fail quietly, you should monitor the telemetry pipeline itself. Track how many data points the SDK exports, how many the Collector receives, how many it refuses or drops, and how many the backend accepts. A sudden drop in accepted points after a configuration change is a classic sign that a backend is rejecting a flavor it does not support. A sudden jump in the number of series may mean an attribute with unbounded values was introduced.

For delta pipelines, pay extra attention to dropped data, since every dropped point is a permanent hole in your totals. For cumulative pipelines, watch memory in your applications and Collectors, since they accumulate state. For converted pipelines, monitor the memory and restart frequency of the converting tier.

Spot-checking the numbers

A healthy habit is to compare a metric against an independent source from time to time. For example, compare the total requests reported by your metric with the count in your load balancer’s own statistics for the same period. A persistent gap suggests lost delta points, a misconfigured conversion, or a query that treats one flavor as the other.

Security considerations

Temporality does not change the sensitivity of the data, but the surrounding setup deserves attention. Metrics travel over OTLP, and the connection between SDKs, Collectors, and backends should use encryption and authentication so that nobody can read or inject numbers. Attributes may carry sensitive information by accident, such as email addresses or account numbers, and since each unique value becomes a series, they are also a cardinality hazard. Keep such values out of metric attributes altogether.

What works in practice, and the traps that catch even experienced teams.

Patterns that work

Anti-patterns to avoid

Short answers to the questions people ask most often.

No. The export interval is how often data is sent. Temporality is what each sent value means, either a total since a fixed start or only the change since the last report. You can combine any interval with either flavor.

Cumulative is the default for the OTLP exporter in the standard specification. Delta and low-memory modes are opt-in through exporter configuration.

No. A gauge records the latest observed value at an instant, so there is no accumulation period to describe. Temporality applies to sums, histograms, and exponential histograms.

No. Instrumentation code stays the same. Temporality is chosen in SDK or exporter configuration, which is why operators can change it without involving developers or rebuilding the application.

A delta up-down counter only reports movements, so reconstructing the current level requires every point. One lost point corrupts the level permanently. Keeping these instruments cumulative avoids that fragility.

Not really. It limits how long the SDK remembers idle series, but active series are still created and still stored by the backend. Controlling attributes is the real remedy.

The total starts again from zero and the start time moves forward. Consumers that understand resets will treat the drop as a restart and continue counting correctly from the new value.

Yes. You can configure two metric readers in the SDK with different preferences, or send one flavor to a Collector and convert it for each destination with the conversion processors.

A delta needs a previous value to subtract from. The first cumulative point has no earlier baseline, so conversion processors usually treat it specially, often by using it only to initialise state.

Neither wins universally. Cumulative is more forgiving and fits Prometheus-style systems, while delta is leaner and fits short-lived workloads and delta-preferring vendors. The right answer depends on your backend, workload, and tolerance for lost data.

Aggregation temporality answers a deceptively simple question: when the SDK reports a metric value, how far back in time does that value reach? In the cumulative answer, it reaches back to a fixed starting point, giving a resilient but memory-hungry stream. In the delta answer, it reaches back only to the previous report, giving a lean but fragile stream. The data model captures the choice through a temporality field and a pair of timestamps, the SDK lets you select it through a reader preference, and the Collector can translate between the two when needed.

The wise way to use this knowledge is to start from the destination, choose the flavor it expects, keep the pipeline consistent, convert only at the edge, control cardinality separately, and make sure every dashboard and alert query matches the flavor that is actually stored. Like the odometer and the trip meter, both flavors are honest. Trouble only starts when we forget which one we are reading.