Cardinality is simply the count of different things, and in telemetry the thing being counted is the number of unique streams of data.

The Core Idea in Simple Words

Cardinality means “how many different values exist.” If a label called http.request.method can only be GET, POST, PUT, DELETE or PATCH, its cardinality is about five. If a label called user.id can be any one of ten million customers, its cardinality is ten million. A label with a small, fixed list of values is called low cardinality. A label that can take almost endless values is called high cardinality.

In metrics, cardinality matters because each unique combination of a metric name and its attributes becomes a separate time series. A time series is a list of numbers recorded over time, such as “requests per second for the checkout service, in the production region, returning status 200.” Change any one attribute value and you get a brand new series that has to be stored separately.

Why This Is a Multiplication Problem

The dangerous part is that cardinality multiplies. If a metric has three attributes with 10, 20 and 50 possible values, the worst case is 10 times 20 times 50, which is 10,000 series. Add a fourth attribute with 1,000 values and the worst case jumps to ten million. This is why a harmless looking change, such as adding one more label to a popular metric, can quietly turn a cheap dashboard into an expensive incident.

Active Series and Churn

Two related words matter here. Active series are the streams that are receiving data right now. Churn is how quickly old streams disappear and new ones appear. A system that restarts containers every few minutes, each with a fresh random name, creates new series constantly even if the total number alive at any moment looks small. Backends usually pay for both, because every new series must be indexed, even if it only lives for a short time.

How OpenTelemetry Decides What Counts as One Series

Inside the metrics model, a stream is identified by the instrument that produced it, the scope of the library that created that instrument, the resource describing the producer, and the exact set of attribute keys and values on the data point. Change any of these and you have a different stream. This matters because two services using the same metric name still produce separate streams, and the same service running in two regions produces separate streams again. Counting cardinality therefore means counting every distinct combination across all of these parts, not just the attributes you wrote in your code.

Why Cardinality Is a Quiet Failure

Most failures announce themselves with errors. Cardinality problems usually do not. The application keeps working, the pipeline keeps accepting data, and the charts keep drawing. The damage shows up slowly as rising memory, slower queries and larger bills, and it often becomes visible only when a limit is finally hit and data starts to be rejected. This quiet nature is why prevention and early detection matter so much more than clever recovery after the fact.

An explosion is never only a backend problem, because the cost shows up at every hop between your code and your storage.

Following the Journey of a Data Point

In OpenTelemetry a measurement starts inside your application, where an instrument records a value together with a set of attributes. The SDK, which is the OpenTelemetry library running inside your process, groups those measurements into streams and aggregates them. At a fixed interval it exports them, usually to an OpenTelemetry Collector. The Collector can change, filter and route the data, and finally sends it to one or more backends where it is stored and queried.

flowchart LR
  A["Application code and instruments"] --> B["SDK aggregation and views"]
  B --> C["Exporter over OTLP"]
  C --> D["Collector receivers"]
  D --> E["Collector processors"]
  E --> F["Collector exporters"]
  F --> G["Backend storage and index"]
  G --> H["Queries, dashboards, alerts"]
        

What Each Stage Pays For

The Golden Rule of Placement

Because cost travels downstream, the best mitigation happens as early as possible. A label that never gets created costs nothing anywhere. A label dropped in the Collector still costs SDK memory and network bytes. A label dropped in the backend has already been paid for in every earlier stage. The rest of this post follows this rule, moving from the earliest control to the latest.

You cannot manage what you have not measured, so advanced teams treat cardinality like a budget with owners, limits and alerts.

Estimating the Worst Case Before You Ship

Before adding a metric, list every attribute and estimate how many distinct values each can take over a normal week. Multiply the numbers to get the theoretical maximum. This gives a ceiling, and the real number is usually lower because many combinations never happen together. For example, a payment provider label and a country label rarely combine in every possible way.

The difference between theoretical and real series count is called sparsity. Sparse metrics are cheaper than the math suggests, but sparsity is also deceptive. A single bad deployment, a bot attack or a misconfigured client can suddenly make rare combinations common, and the real number moves quickly toward the ceiling.

Choosing a Budget Unit

Mature teams usually express budgets in one of three ways. Pick the one that matches how your backend charges and how your teams are organised.

Growth Rate Matters More Than Size

A metric with fifty thousand stable series may be fine. A metric that grows from five thousand to fifty thousand in one hour is a warning sign, because it suggests an attribute is tied to something unbounded such as an identifier or a timestamp. For this reason alerts on the rate of growth are often more useful than alerts on absolute size, since they fire while the problem is still small.

Walking Through a Budget Example

Suppose a checkout service reports a latency histogram with three attributes: route template with 40 values, status class with 5 values and region with 6 values. The worst case is 40 times 5 times 6, which is 1,200 attribute sets. If the histogram has 15 buckets plus a count and a sum, the backend may store about 17 series for each set, which is roughly 20,400 series. That might be acceptable. Now suppose someone adds a customer tier attribute with 4 values and a feature flag attribute with 60 values. The worst case becomes 1,200 times 4 times 60, which is 288,000 attribute sets and nearly five million series. The change took one line of code and increased the cost more than two hundred times.

This example shows why a simple review habit pays off so well. Doing the multiplication before merging takes two minutes and reveals the danger at once, while discovering it in production takes days of investigation and a difficult conversation about the bill.

The strongest mitigation is a good decision made while writing instrumentation, long before any pipeline is involved.

Use Bounded Values Only on Metrics

A healthy metric attribute has a small, predictable list of possible values. Good examples are HTTP method, status class (such as 2xx or 5xx), region, deployment environment, service version and a named operation. Unhealthy examples are user identifiers, session identifiers, order numbers, email addresses, full URLs with query strings, raw error messages, request IDs and timestamps.

Follow Semantic Conventions

OpenTelemetry publishes semantic conventions, which are agreed names and meanings for common attributes. Using them keeps your data consistent, but it also helps cardinality in a quiet way. The conventions explain which attributes are meant for metrics and which are better for traces. Some are even marked as safe only when the value is a template rather than a raw value, such as the HTTP route. Starting from the conventions means you inherit hard-earned lessons from the community instead of rediscovering them in production.

Pick the Right Signal for the Detail

When you really need a high-cardinality detail, such as which customer was affected, do not put it on a metric. Put it on a span attribute or a log record, where each event is stored on its own and no series has to be created. Metrics are for answering “how many and how fast,” while traces and logs are for answering “which exact one.” Splitting the questions this way keeps each signal cheap and useful.

When you cannot change the instrumentation, perhaps because a library you do not own created it, the SDK gives you a way to reshape metrics before they leave the process.

What a View Is

A view is a rule that you attach to the metrics SDK. It matches one or more instruments by name, type or the library that created them, and then changes how they are reported. Through a view you can rename a metric, change its description or unit, choose a different aggregation, and, most importantly for this topic, decide which attributes are kept. Attributes that are not on the keep list are removed and the measurements that differ only by those attributes are merged together.

Allow-lists Are Safer Than Deny-lists

There are two ways to prune attributes. A deny-list removes a few known bad attributes, while an allow-list keeps only a few known good ones. For cardinality control the allow-list is usually safer. A deny-list quietly breaks the day a developer introduces a new risky attribute that nobody listed, whereas an allow-list silently ignores anything new until someone reviews it and adds it on purpose.

The SDK Cardinality Limit and Overflow

The OpenTelemetry metrics specification also describes a cardinality limit for each metric stream, with a default of two thousand distinct attribute sets in SDKs that implement it. When the limit is reached, new combinations are not given their own series. Instead they are folded into a special overflow series that carries the attribute otel.metric.overflow set to true. Your process stays healthy, you keep an accurate total, and you gain a visible signal that something is wrong.

You can change the limit through a view for a specific stream. Be careful with this, because raising it makes the symptom disappear without fixing the cause. A smarter use is to lower the limit for metrics you know should be small, so that any surprise growth is caught immediately.

How a metric is aggregated decides how many streams it produces and how long the SDK must remember them.

Histograms Multiply Series

A histogram records how values spread across ranges, called buckets, such as “how many requests finished in under 100 milliseconds.” In many backends one histogram attribute set becomes one series per bucket, plus a count and a sum. If a histogram has twenty buckets and a thousand attribute sets, that is more than twenty thousand series from a single instrument. Histograms are the most common hidden source of cardinality surprises.

Exponential Histograms

OpenTelemetry offers exponential histograms, which use buckets that grow by a constant ratio and adjust their resolution automatically. They have a maximum bucket count, and when it is reached the SDK lowers the resolution rather than adding more buckets. This gives good accuracy across a very wide range of values with a bounded size. You also avoid choosing bucket boundaries by hand, which is a task that teams often get wrong. Check that your backend supports native exponential histograms, because converting them into classic buckets at the edge can bring the cost back.

Delta versus Cumulative Temporality

Temporality describes whether each export reports the change since the last export, which is delta, or the running total since the process began, which is cumulative. This choice has a big effect on memory. With cumulative reporting, the SDK must remember every series that has ever been seen, because each export repeats its running total. With delta reporting, a series that received no new data can be forgotten after it has been exported, so short-lived attribute combinations do not stay in memory forever.

Drop What Nobody Uses

Some libraries emit many metrics that no dashboard or alert ever reads. A view can switch such instruments to the drop aggregation, which discards them at the source. Reviewing which metrics are queried, and removing the rest, is often the single fastest win in a cost reduction project, and it carries no risk to the metrics you actually rely on.

Choosing the Right Instrument Type

OpenTelemetry offers several instrument types, and the choice quietly shapes how many series you create. A counter only goes up and is ideal for events such as requests served. An up-down counter can rise and fall, which suits things like items in a queue. A gauge records the latest value of something, such as current temperature. A histogram records a spread of values. Picking a type that does not match the question often pushes teams toward extra labels to make up for what the instrument cannot express.

Observable Instruments and Per-Entity Reporting

Some instruments are observable, which means the SDK calls your code at each export interval and asks for the current values. They are very convenient for things like memory use or connection counts. The risk is that a callback can loop over a large list, such as every customer, every file or every queue, and report one value for each entry. That quietly creates one series per entity, and because the callback runs again at every interval, the cost repeats continuously.

A safer habit is to report totals or a small number of meaningful groups from observable callbacks. If you truly need per-entity detail, report only the top few entries, or the entries that are outside a healthy range, and keep the complete list in logs or a database where it belongs.

Units, Names and Consistency

Naming looks like a cosmetic matter, but it affects cardinality too. If two teams record the same measurement under slightly different metric names, units or attribute keys, the backend stores them as separate families and the total series count rises for no benefit. Agreeing on one name, one unit and one set of attribute keys for each common measurement keeps the data tidy, avoids duplicate work and makes later cleanup much easier.

The OpenTelemetry Collector sits in the middle of the pipeline, which makes it the best single place to enforce rules across every service at once.

Why a Central Layer Helps

Changing instrumentation in two hundred services takes months, and some services are not even owned by your team. The Collector lets a platform team apply one policy to all of them. It receives data from many sources, runs it through a chain of processors, and passes it on. Each processor does one small job, and the order of the chain matters a great deal.

The Main Tools in the Chain

Normalising Values Instead of Only Deleting Them

Deleting an attribute loses information completely. A gentler approach is to normalise it, which means turning many raw values into a few meaningful ones. A path like “/users/8821/orders/55” can become “/users/{id}/orders/{id}”. A user agent string with a hundred variations can become “browser,” “mobile app” or “bot.” A status message can be reduced to its category. Normalising keeps the useful breakdown while removing the explosion.

flowchart TD
  R["Receiver accepts OTLP data"] --> M["Memory limiter protects the Collector"]
  M --> N["Transform normalises values"]
  N --> F["Filter drops unused metrics"]
  F --> A["Attributes removes risky labels"]
  A --> G["Aggregate fewer series"]
  G --> B["Batch groups for export"]
  B --> X["Exporter sends to backend"]
        

Stateful Processors Need Their Own Limits

Some processors remember streams, such as those that convert delta data into cumulative data. They keep one entry per unique series, so a high-cardinality input becomes a Collector memory problem. These components usually offer settings to cap how many streams they track and to forget idle ones after a period. Always set both, and always monitor them, otherwise the safety net becomes a new single point of failure.

Scaling Stateful Processors Across Many Collectors

Real systems run many Collector instances behind a load balancer. This creates a subtle problem for any processor that remembers streams. If data points from the same series land on different instances, each instance sees only a slice of the story. Conversion between temporality types can then produce wrong totals, and each instance keeps its own copy of the same series, which wastes memory and multiplies the cost you were trying to reduce.

The usual answer is a two-layer design. A first layer of Collectors receives data from applications and applies the stateless steps, such as filtering, renaming and attribute cleanup. A second layer holds the stateful steps. Between them, a load-balancing exporter routes data by a chosen key, such as the service name or the full identity of the series, so every point for a given series always reaches the same second-layer instance. With this routing, each stream lives in exactly one place, memory is shared fairly, and results stay correct.

Pipelines per Tier and per Signal

The Collector lets you define several pipelines, each with its own receivers, processors and exporters. Use this to apply different rules to different needs. A critical pipeline might keep full detail for a small list of alerting metrics and send them to a protected backend. A best-effort pipeline might aggressively drop labels and send the remainder to cheaper storage. Traces and logs can have their own pipelines too, with sampling in one and attribute cleanup in the other. Separate pipelines make policies easy to read, easy to change and easy to explain to the teams affected by them.

Many explosions do not come from the metric attributes at all, but from the resource attributes that describe where the data came from.

What a Resource Is

A resource in OpenTelemetry is a description of the thing producing telemetry, such as the service name, version, host, container, pod and cloud region. It is attached once to a whole batch of data, which is efficient inside OpenTelemetry. The trouble starts when a backend copies every resource attribute onto every series, because then a single changing identifier multiplies everything it touches.

The Usual Suspects

Promote Only What You Truly Query

When the backend is a Prometheus-style system, the usual guidance is to keep most resource attributes off the individual series. Instead, store them once in an informational series that can be joined in when needed, and promote only a short, reviewed list of resource attributes into labels. Typical safe choices are service name, namespace, environment and region. This keeps queries fast while still letting you look up the full detail on demand.

You can keep the ability to jump from a number to a specific example without turning that example into a label.

The Problem Engineers Are Trying to Solve

Imagine a latency chart shows a sudden spike. The natural question is “which request was slow?” Many teams respond by adding a request identifier or trace identifier as a metric label. That answers the question but creates a new series for every single request, which is the worst possible case of cardinality.

What an Exemplar Is

An exemplar is a small sample attached to a metric data point. It carries the measured value, the time, and the identifiers of the trace and span that produced it. It lives beside the series instead of inside its identity, so adding exemplars does not create new series. A dashboard can then show a dot on the chart, and clicking it opens the exact trace behind that measurement.

Sampling Exemplars Wisely

The SDK does not attach an exemplar to every measurement. It uses a filter and a reservoir, which is a small box that keeps only a few samples per interval. A common filter keeps only measurements that happened inside a sampled trace, so the linked trace is guaranteed to exist in storage. Without that rule, you might click an exemplar and find that its trace was never kept.

Common Exemplar Pitfalls

Exemplars solve one problem well, but they bring a few traps. The first is storage support. Not every metrics backend keeps exemplars, and some keep them only for certain metric types, so you should test the full path from chart to trace before relying on it. The second is sampling alignment. If the trace is dropped by a sampler, the exemplar points to nothing, which frustrates engineers during an incident. Making exemplar selection depend on whether the trace was sampled prevents this.

The third trap is size. Each exemplar carries a small amount of extra data, such as its own attributes. If you copy many attributes onto exemplars, payloads grow and you pay in network and storage even though series count stays flat. Keep exemplars lean, and let the trace hold the detail.

Where Exemplars Fit in Daily Work

The best way to use exemplars is as the first step of an investigation. A latency chart spikes, you click a dot on the spike, and you land on a trace that shows which downstream call was slow. From there you can read the span attributes, including customer identifiers, without those identifiers ever having been labels. This workflow gives engineers the speed they like about rich labels and gives the platform team the control it needs over series count.

Cardinality is not only a metrics concern, and a very common explosion happens when metrics are generated automatically from traces.

Span-Derived Metrics

A connector can read incoming spans and produce metrics such as request counts, error counts and latency histograms. This is handy because teams get consistent numbers without writing new instrumentation. The danger is that the connector builds dimensions from span attributes. If a span name contains a raw identifier, or a chosen dimension has unlimited values, the connector will faithfully create a new series for each one.

The span metrics connector therefore comes with safeguards, such as a limit on how many distinct combinations it will track, with extra combinations folded into an overflow bucket, plus a time after which idle series are forgotten. Treat these safeguards as required settings rather than optional extras. Choose the dimensions deliberately and keep them to a short list of bounded attributes.

Traces: Cost Comes from Volume, Not Series

Traces do not create time series, so attribute values like a customer identifier are acceptable on spans. The cost control for traces is sampling. Head sampling decides at the start of a trace whether to keep it, which is cheap but blind to what happens later. Tail sampling waits until the trace is finished and keeps the interesting ones, such as errors and slow requests, which is smarter but needs memory in the Collector to hold traces while waiting.

Logs and Their Hidden Cardinality

Logs are stored as individual records, but many backends also index log attributes to make searches fast. Every distinct indexed value can cost storage and query time. Keep the indexed fields to a few stable ones, such as service, severity and environment, and keep unique content in the message body or in non-indexed fields. When logs are turned into metrics, apply the same bounded-value rules that you use for any other metric.

Baggage and Accidental Attribute Spreading

OpenTelemetry has a feature called baggage, which carries small key-value pairs along with a request as it travels between services. It is useful for passing context such as a tenant name or an experiment group. The danger appears when a component is configured to copy every baggage entry into span or metric attributes automatically. A single value added at the front door then appears in every downstream service, and if that value is high cardinality, every service now has a problem.

Treat baggage like a shared bulletin board. Only approved keys should be copied into attributes, and only bounded ones should ever reach metrics. A review of which keys are allowed, and a Collector rule that removes unexpected ones, keeps the board tidy.

Events, Links and Status on Spans

Spans can also carry events, which are timestamped notes, and links to other traces. These do not create series, but they do add volume. A loop that adds an event for every item processed can make a single span enormous. Backends often cap the number of events and attributes per span, and anything above the cap is silently dropped. Keep events for meaningful moments, such as a retry or a cache miss, rather than for every iteration of a loop.

Error Handling Without Explosions

Error information is a classic source of unbounded values. Teams are tempted to record the full exception message as an attribute, but messages often contain identifiers, file paths or timestamps. Record a stable error type on the span and on any related metric, and keep the full message in the span event or the log record. This preserves the ability to count errors by kind while avoiding a unique series for every failure.

Even with every upstream control in place, the backend should still defend itself, because the final layer is the one that protects your budget.

Ingestion Limits

Most metrics backends can cap what they accept. Self-managed Prometheus-compatible systems allow limits on the number of labels, the length of label values and the number of samples or series per scrape target. Multi-tenant systems in this family, such as Grafana Mimir and Cortex, let operators set a maximum number of active series for each tenant, so one noisy team cannot take down everyone else. When a limit is hit the system rejects the extra data and reports an error, which is annoying but far better than an outage.

Relabelling and Rewriting at the Edge

Backends and the agents that feed them often support relabelling rules. These rules can drop series, remove labels or rewrite values as data arrives. They are a useful last-resort filter and can fix problems immediately when instrumentation cannot be changed quickly, though they work on data that has already crossed the network.

Aggregating After the Fact

Another powerful idea is to store a pre-aggregated version of a metric. You keep a detailed, short-retention copy for debugging and a reduced copy, with the risky labels removed, for long-term dashboards. Some products offer this as a managed feature that analyses which labels are actually queried and suggests dropping the rest. Streaming aggregation and recording rules do the same job on self-managed systems.

Understanding How You Are Billed

Different platforms charge in different ways, and your mitigation priorities should follow the bill. Some charge by the number of active series over a period, which makes cardinality the dominant cost. Others charge by the amount of data ingested, which makes sample frequency and payload size matter more. A few charge by the number of queries or by storage, which rewards short retention. Read the pricing model carefully before choosing which controls to invest in first, because a technique that saves a lot on one platform may save very little on another.

Retention as a Cardinality Tool

Cost is not only about how many series exist, but also how long they are kept. Keeping detailed series for a few days and reduced series for months gives you the best of both. Short retention suits debugging data, because you usually investigate a problem within hours or days. Long retention suits business and capacity trends, which do not need fine detail. Setting retention per tier turns a permanent cost into a temporary one for the data that deserves it least.

Querying High-Cardinality Data Safely

A single heavy query can hurt a backend as much as a heavy ingestion spike. Encourage dashboards to filter by service and environment first, and to avoid grouping by labels with huge value counts. Use recording rules or pre-aggregated views for the dashboards that many people open all day, so the expensive calculation is done once instead of on every refresh.

The best mitigation plan includes a way to notice growth early, ideally before the invoice arrives.

What to Measure

Beyond those three, watch the Collector’s own health numbers. OpenTelemetry Collector exposes internal metrics about points accepted, points refused, points sent and send failures, as well as memory use and queue size. A rising number of refused points means the memory limiter is shedding load. A rising queue usually means the backend is slow, often because it is struggling with the series you are sending.

Top-N Analysis

When something grows, you need to find the culprit fast. A top-N report lists the metrics, services and labels that contribute the most series. The label view is especially useful because it shows exactly which attribute is exploding. Many backends provide such a cardinality explorer, and it is worth running on a schedule rather than only during emergencies.

Catch Problems Before Production

The cheapest detection happens in the development pipeline. Teams can declare their telemetry in a shared schema, check pull requests against it, and run tests that count series produced during a realistic load run. The OpenTelemetry community maintains tooling, such as Weaver, for defining and validating telemetry schemas and conventions, which fits neatly into this kind of automated gate. A failing check on a pull request costs minutes, while a failing dashboard in production costs hours.

Load Testing Your Telemetry

Most teams load test their applications, but few load test their telemetry. A simple practice is to run a realistic traffic pattern in a staging environment, including odd inputs such as very long paths, many different clients and bursts of errors, and then count the series that appear. If the count is far above your estimate, you have found a problem while it is still free to fix. Repeat the test after major releases, because new features often introduce new attributes.

Reading the Warning Signs

Tools alone do not stop explosions, because the real cause is almost always a person adding a label with good intentions and no feedback.

Ownership and Accountability

Every metric should have an owner, and every owner should be able to see what their metrics cost. When cost is invisible, nobody feels responsible for it. Show each team a simple monthly view with their series count, their growth and their share of the platform budget. Teams that can see the number tend to fix the problem themselves without being asked.

Golden Paths and Shared Defaults

A golden path is a well-lit, easy route that most teams will follow by default. In observability this means a shared instrumentation library or starter template that already has sensible views, bounded attributes, exemplars enabled and safe resource settings. When the easy path is also the safe path, most teams stay inside the guardrails without thinking about it.

Tiers of Telemetry

Not every metric deserves the same treatment. Many organisations create tiers, such as critical, standard and best-effort. Critical metrics feed alerts and service level objectives, so they get generous limits and protected capacity. Best-effort metrics, such as experimental or debugging data, get strict limits and shorter retention. Separate Collector pools for each tier prevent a noisy experiment from affecting an important alert.

Reviewing Instrumentation Like You Review Code

Instrumentation deserves the same care as any other code that touches production. A good review looks at three questions. First, which attributes are being added, and can each one be listed in advance or at least bounded by a clear rule? Second, how often will the code path run, and does each run create a new combination? Third, is the same detail already available on a trace or a log, which would make a metric label unnecessary? A short checklist built from these questions can be added to the pull request template, and it takes only a minute to apply.

Training and Shared Language

Cardinality is easy to explain but surprisingly hard to feel until you have seen a bill or a crash. Short workshops that walk through a real past incident work better than slides of rules. Give engineers a simple vocabulary, such as bounded, unbounded, churn and overflow, and they will start using the words in design discussions. Once a team shares the language, the habit becomes part of the culture rather than a chore enforced by the platform group.

Showback and Chargeback

Two models help make cost visible. Showback simply reports what each team uses, without sending an actual invoice. Chargeback moves the cost to each team’s own budget. Showback is the gentler place to start, and it is often enough to change behaviour. Chargeback can be powerful but needs fair measurement and agreement about shared infrastructure. Either way, the main idea is that the people who decide to add a label should be able to see what it costs.

Knowing the common mistakes by name makes them easier to spot in a design review.

When series counts are climbing right now, a calm and ordered response matters more than a clever fix.

Stabilise First, Diagnose Second

During an incident the first goal is to stop the damage, not to understand everything. Protect the backend and the Collector, then trace the cause. A temporary drop rule that you later refine is far better than a perfect analysis delivered after the system has already failed.

flowchart TD
  S["Alert: series growth is high"] --> C{"Is it real growth?"}
  C -- "No" --> D["Fix the dashboard or alert"]
  C -- "Yes" --> T["Find top metric and top label"]
  T --> K["Contain centrally in the Collector"]
  K --> V["Verify alerts still work"]
  V --> X["Owner fixes the source"]
  X --> G["Add a permanent guard"]
        

After the Incident: A Learning Review

Once the system is stable, hold a short blameless review. The aim is to understand which control should have caught the problem and why it did not, rather than to find someone to blame. Ask whether the attribute should have been rejected during code review, whether an SDK limit would have contained it, whether the Collector rules covered it, and whether an alert fired early enough. Each answer points to a concrete improvement.

Prepare Runbooks Before You Need Them

The middle of an incident is a bad time to learn how to edit Collector configuration safely. Write short runbooks in advance that describe where to find the top-N view, how to add a temporary rule, how to roll it out, how to check that alerts still work and how to remove it again. Practising the steps in a quiet period, for example during a game day, turns a stressful scramble into a routine task.

Now that each control has been explained on its own, this chapter shows how they fit together in one coherent architecture for a growing company.

The Starting Situation

Picture a company with about two hundred services running on a container platform. Several teams use different languages. Some services are written in-house, while others are third-party tools that already emit their own metrics. The company sends telemetry to one central metrics backend, one trace store and one log store. Costs have doubled in a year, dashboards are slow, and the platform team has been asked to bring the situation under control without slowing down product teams.

The Layered Architecture

The design uses four layers that match the pipeline you have studied. At the application layer, a shared instrumentation package provides approved attribute names, bounded route templates, exemplars switched on and a set of SDK views that allow only reviewed attributes. At the edge layer, an agent-style Collector runs close to each workload and applies stateless cleanup, such as removing known risky attributes, normalising paths and dropping unused metrics. At the gateway layer, a second pool of Collectors handles the stateful steps and routes data consistently by series. At the storage layer, the backend enforces per-team series limits and keeps a short-retention detailed copy next to a long-retention reduced copy.

flowchart LR
  A["Apps with shared instrumentation and SDK views"] --> B["Edge Collector: stateless cleanup"]
  B --> C["Load-balancing layer: route by series"]
  C --> D["Gateway Collector: stateful steps and limits"]
  D --> E["Critical tier backend"]
  D --> F["Best-effort tier backend"]
  E --> G["Detailed short retention"]
  E --> H["Reduced long retention"]
  F --> H
        

How Each Layer Earns Its Place

Where Each Kind of Detail Goes

A useful way to keep the design honest is to decide, for every piece of information, which signal should carry it. Counts, rates and latency distributions go on metrics with bounded attributes. The story of a single request, including customer identifiers and long strings, goes on spans. Human-readable events and error messages go in logs with a small set of indexed fields. The bridge between them is the trace identifier, carried by exemplars on metrics and by correlation fields in logs. With this split, nothing needs to be duplicated on a high-cardinality label.

Cardinality controls can break dashboards and alerts if they are introduced carelessly, so the order and pace of the rollout matter as much as the controls themselves.

Phase One: Measure Before You Change Anything

Begin by building a clear picture of the current state. Collect the total number of active series, the top metrics, the top services and the top labels. Record which dashboards and alerts are actually used, because a surprising share of metrics are never queried. This baseline serves two purposes. It tells you where the biggest savings are, and it gives you a before-and-after comparison that proves the project worked.

Phase Two: Take the Safe Wins First

Start with changes that have almost no risk. Drop metrics that nobody reads, remove obviously unbounded labels such as raw URLs and request identifiers, and strip resource attributes that never appear in any query. These changes usually remove a large portion of the series with very little discussion. They also build trust, because teams see savings without losing anything they care about.

Phase Three: Add Guardrails

Once the easy savings are captured, put protective controls in place so growth does not return. Introduce SDK limits, Collector memory limiters and stream caps, and backend per-team series limits. Start these in a reporting or alert-only mode if possible, so teams learn how close they are to the limits before anything is rejected. After a short period, switch the limits to enforcing mode.

Phase Four: Change the Defaults

Publish the shared instrumentation package and the telemetry contract. Make new services use them by default, and gradually move older services across. This is the long-term fix, because it stops new problems at their source. Pair it with a review step in the delivery pipeline that checks attributes against the approved list.

Communicating With the Teams You Affect

Technical changes succeed or fail on communication. Tell teams what you are changing, why you are changing it, and how they can ask for an exception. Give owners a simple way to request that a label be kept, together with a short explanation of the business need. Many requests turn out to be unnecessary once the cost is explained, and the genuine ones can be handled with a targeted allowance rather than a global exception.

Every mitigation technique trades something away, so the skill is in choosing the right trade for each situation instead of applying every technique everywhere.

Detail versus Cost

The central tension is that more detail helps debugging, while less detail lowers cost. There is no single right answer, because it depends on how the data is used. A payment service with strict reliability goals may justify rich detail on a few key metrics, while an internal batch tool may need only a handful of totals. The decision should be made per metric tier, not for the whole company at once.

Central Control versus Team Freedom

Central Collector policies are fast and consistent, but they can surprise teams if rules change without notice. Team-owned SDK views give owners freedom but require every team to do the work. A healthy balance is to set central rules as a safety floor and let teams add their own improvements on top, with clear documentation of what the platform enforces.

Early versus Late Intervention

Acting early at the source saves the most, but it needs code changes, review and deployment. Acting late in the backend is quick, but it saves the least and risks silent breakage. Plan to use late controls for emergencies and early controls for lasting fixes, and make sure each emergency leads to a permanent change upstream.

A Simple Decision Table

Short answers to the questions that come up most often when teams start taking cardinality seriously.

There is no universal number, because it depends on your backend and budget. Many teams treat a few thousand series per metric as comfortable and anything in the tens of thousands as needing a justification. What matters most is that the number is bounded and that someone owns it.

Only if the user population is small and fixed, such as a handful of internal tenants. For a public product with many customers, keep identifiers on traces and logs and use exemplars to connect them to metrics.

Use both for different reasons. The SDK protects process memory and prevents unwanted data from leaving the application. The Collector gives you a central policy that covers every service quickly, including ones you do not control. Instrumentation design remains the best fix of all.

It tells you that a stream reached its cardinality limit, and that new attribute combinations were merged into a single catch-all series. Totals stay correct, but detail is lost, so the signal should lead you to find and fix the attribute that is growing without bound.

They help because a histogram stays one compact data structure with a bounded size, instead of many classic bucket series. The benefit depends on your backend storing them natively. If the backend converts them to classic buckets, much of the saving disappears.

Each new pod name can create a whole new set of series, and the old ones remain in the index for a while. Frequent restarts or autoscaling therefore create churn, even if the number of pods alive at any moment is steady.

Not always. It helps the SDK forget idle series, which is good for high-churn workloads, but it only works well if your backend accepts delta data or if a Collector component converts it for you. Check the backend and keep conversion components well limited.

Not for data that was already discarded. That is why tiered strategies are useful: keep short-lived, detailed data for debugging, and a reduced version for long-term use. Traces and logs can hold the detail that metrics should not.

Make cost visible per team, give them easy shared defaults, and add a lightweight review step. Most engineers will happily follow good practice once the cost and the safe path are both obvious.

Yes. The concepts of bounded attributes, resource hygiene, exemplars, central policy and backend limits apply to nearly every observability platform. Only the names of the settings and the way cost is charged will differ.

Start with a top-N report of series by metric name, then break the worst metric down by each label and count the distinct values. The label with the largest and fastest growing value count is almost always the culprit. If your backend has no such report, you can sample a few minutes of data in the Collector and count distinct values there. Doing this regularly, not only during incidents, helps you spot slow growth early and makes the next investigation much shorter because you already know what normal looks like for each service.

It can, if an alert or dashboard filters or groups by that label. Before removing anything, search your alert rules, dashboards and recording rules for the label name and check query logs for recent use. If it is used, decide whether the alert really needs that detail or whether a coarser grouping would do. When a label must go and an alert depends on it, update the alert first and remove the label afterwards, so there is never a moment when the alert silently stops working.

Cardinality describes how many distinct series exist, while churn describes how fast series appear and disappear. A system can have modest cardinality at any single moment but very high churn, for example when containers restart constantly with fresh names. Both cost money, because backends must index every new series even if it lives only briefly. Fixing churn usually means removing short-lived identifiers from labels and, where possible, using delta temporality so that idle streams are forgotten quickly instead of being carried forward by the SDK.

Not usually. A busy customer-facing service with many routes and regions legitimately needs more series than a small internal tool. A fair approach is to set a baseline for each tier and let teams request more with a short explanation. Revisit limits regularly, because the right number changes as services grow. Uniform limits tend to either strangle important services or leave small ones with far more room than they need, and neither outcome supports healthy cost control or reliable alerting.

They solve different problems. Cardinality concerns the number of metric series, while tail sampling controls how many traces are stored. However, they interact through span-derived metrics. If you generate metrics from spans after sampling has dropped many of them, the numbers will be too low, so metrics are usually generated before sampling decisions. Keeping that ordering in mind prevents misleading dashboards and also lets you apply dimension limits on the metrics path while still keeping only the interesting traces for later investigation.

It can be, particularly when the log backend indexes every field. Each distinct field name and value adds index entries, so unpredictable fields can inflate storage and slow searches. Keep a small set of indexed fields that you search often, such as service, severity and environment, and place everything else in the message body or in fields that are stored but not indexed. When you turn logs into metrics, apply the same bounded-value rules that you use for any other metric attributes.

The measurements are not lost entirely. Their values are combined into one series, so totals such as the overall request count remain correct. What disappears is the breakdown by the individual attribute combinations that did not fit under the limit. This makes the overflow series a useful honesty signal, because it tells you the numbers are right but the detail is incomplete. You can then decide whether to bound the attribute, raise the limit for a justified reason, or move the detail to traces.

A good rhythm is a quick weekly look at the top contributors and a deeper quarterly review of budgets, limits and emergency rules. Weekly checks catch slow growth, while quarterly reviews adjust to business changes such as new regions or products. Every temporary rule should have an owner and an expiry date, and the quarterly review is a natural moment to remove those that are no longer needed, so the configuration stays understandable and does not collect forgotten surprises.

Yes, and the simplest ideas give the most value. Keep metric attributes bounded, put identifiers on traces, use a memory limiter in the Collector, drop metrics nobody reads and set a series limit in the backend. A small team can do all of this in an afternoon. The heavier parts of this guide, such as tiered Collector pools and chargeback, matter mostly when many teams share one platform. Start with the basics and add layers only when the size of the system calls for them.

Ask the question “how many different values can this attribute take?” every time someone adds one to a metric. It takes seconds, it costs nothing, and it prevents the majority of explosions before they begin. Combine it with a growth-rate alert and a visible budget for each team, and you have a small, durable system that protects your pipeline even as people, services and traffic keep changing around it over the coming years.

Cardinality explosions happen because each unique combination of attributes becomes its own stream that must be stored, shipped and indexed. They are expensive because the cost multiplies as the data travels from the application to the backend. The most effective teams respond by placing controls at every layer, starting with careful design at the source, adding protective views in the SDK, applying central policy in the Collector, and keeping firm limits in the backend. They also invest in detection and in team habits, so that problems are noticed early and fixed at their cause rather than patched after each incident.

If you remember only one idea, remember the golden rule of placement. The earlier you remove a risky label, the more of the pipeline you protect. A label that is never created is free everywhere, while a label dropped late has already been paid for at every earlier step.