Before we meet the tool, let us feel the gap it fills.

A trace follows one request through a system. It is made of spans, and each span describes one unit of work, with a name, a start time, an end time, a status, and a set of labels called attributes. If you open a single trace, you can see exactly what happened to one request: which services it visited, how long each step took, and where it failed. This is wonderful for investigating one specific problem.

But think about the questions people ask every morning. How many requests did the checkout service handle in the last hour? What fraction failed? Is the typical request slower than yesterday? Did the slowest one percent get worse? These are questions about many requests at once, and they are best answered with metrics, which are numbers collected over time and drawn as lines on a chart. A chart with a line for request rate, a line for error rate, and a line for latency is the classic dashboard every operations team wants. It is often called the RED view, short for Rate, Errors, and Duration.

One way to get metrics is to ask every application team to add metric code to their services. This works, but it takes a long time, each team may do it differently, and some services, such as old ones or third-party ones, cannot be changed at all. Another way is to notice that the traces already contain everything needed. Every span has a start and an end, so its duration is known. Every span has a status, so you can tell success from failure. Every span has a name and attributes, so you can group by endpoint. If something simply counted and measured the spans as they flow by, the metrics would appear for free. That something is the spanmetrics connector.

A short glossary keeps the rest of the article smooth.

A few more words will come up. Temporality describes how a running total is reported. Cumulative means “the total since the start”, like a car’s odometer. Delta means “only what changed since the last report”, like a trip counter you reset at each stop. Sampling means keeping only some traces instead of all of them, usually to save cost. A backend is the monitoring product or storage system that finally keeps data and draws charts. And an exemplar is a small sample attached to a metric point that links it to one real trace, so you can jump from a number on a chart to an example request behind it.

One idea in particular deserves attention before we continue. Imagine three kinds of labels: resource labels describing the service that produced the data, scope labels describing which code unit produced it, and span labels describing one single request. The spanmetrics connector reads all of these and decides which ones become the labels, or dimensions, of the metrics it creates. Choosing those wisely is the most important decision in using it.

A clear definition, a short history, and a list of what it does and does not do.

The spanmetrics connector is a component of the OpenTelemetry Collector that reads spans from a traces pipeline and produces metrics in a metrics pipeline. For every span it sees, it increases a counter of how many calls there were, and it records the span’s duration into a histogram. It groups these by a set of dimensions, such as service name, span name, span kind, and status code. The output is a family of ordinary OpenTelemetry metrics that any metrics backend can store and chart.

To appreciate the word connector, picture two separate pipelines in the Collector: one for traces and one for metrics. Normally, they do not talk to each other, because a trace pipeline ends in a trace exporter and a metrics pipeline begins with a metrics receiver. A connector is a special part that plays both roles at once. In the traces pipeline, it appears in the list of exporters, so spans flow into it. In the metrics pipeline, it appears in the list of receivers, so metrics flow out of it. It is the bridge between the two worlds.

A Short History

Before connectors existed, the same idea was provided by a spanmetrics processor, which sat in the traces pipeline and tried to emit metrics from there. Processors are designed to pass data of one kind along the same pipeline, so producing a different kind of data from inside a processor was awkward. The Collector then introduced connectors as a proper, general way to convert one signal into another. The spanmetrics idea was rebuilt as a connector, and the older processor was deprecated. If you read older tutorials that use the processor, know that new setups should use the connector, and that some names and settings differ between the two.

What It Does Not Do

• It does not change, drop, or delay your spans. They continue to their normal destinations.
• It does not look inside the content of spans beyond the fields and attributes you choose to use as dimensions.
• It does not replace metrics that your applications record directly, which are often more precise and richer.
• It does not decide which traces to keep. That is the job of sampling, and, as we will see, its position relative to sampling is critical.

Seeing the whole path makes the connector’s role clear.

The Collector is built from receivers, which accept data; processors, which change it; and exporters, which send it on. A pipeline chains these together for one signal type. Connectors add a fourth kind of part that links one pipeline to another. For the spanmetrics connector, the arrangement looks like this.

flowchart LR
  A["Applications and SDKs"] --> B["Receiver for Traces"]
  B --> C["Traces Pipeline: memory limiter and batch"]
  C --> D["Exporter to Trace Backend"]
  C --> E["Spanmetrics Connector as exporter"]
  E --> F["Spanmetrics Connector as receiver"]
  F --> G["Metrics Pipeline: batch"]
  G --> H["Exporter to Metrics Backend"]
  H --> I["Dashboards and Alerts"]
    

Let us walk through the figure. Applications send spans to a receiver. The spans pass through the traces pipeline, which might contain safety and efficiency steps such as a memory limiter and a batch processor. At the end of the pipeline, the list of exporters has two entries: the normal exporter that sends spans to the trace backend, and the spanmetrics connector. Both receive every span. The connector counts and measures them. On the other side, the same connector is listed as a receiver in the metrics pipeline, where it hands over the metrics it has built. From there, the metrics go through any metric processors and out through a metrics exporter to a backend that draws dashboards and fires alerts.

One Configuration, Two Roles

In the Collector’s configuration, the connector is defined once in its own section, much like receivers and exporters. Then it is named in two places: once among the exporters of a traces pipeline, and once among the receivers of a metrics pipeline. This double listing is what makes it a bridge. If you list it in only one place, the Collector will refuse to start, because a connector must be connected at both ends. This is a helpful safety check.

Agents and Gateways

The Collector can run as an agent next to each application, or as a gateway that many agents send data to. You can run the connector in either shape, but the gateway is more common, because it sees the traces of many services in one place and can keep one consistent set of metric rules. However, running it in a gateway with several copies of the Collector raises a scaling question that we will treat in its own chapter.

Two main families of metrics, and an optional third.

Calls: How Many Requests

The first metric is a calls counter. Every time a span arrives, the counter for that span’s group goes up by one. A counter only ever grows, like an odometer. From it, a metrics backend can compute a rate: how many calls per second over a recent window. This gives you the R in RED. Because the status of the span is one of the dimensions, the same counter also gives you the E: the calls whose status is error, divided by all calls, form the error rate.

Duration: How Long Requests Take

The second metric is a duration histogram. Each span’s duration, which is its end time minus its start time, is dropped into a bucket, a range such as “between 100 and 200 milliseconds”. The histogram counts how many spans landed in each bucket, and also tracks the total number of spans and the sum of all durations. From these, your backend can estimate the median, the 95th percentile, the 99th percentile, and the average. This gives you the D in RED.

Events: An Optional Extra

Spans can carry events, which are small timestamped notes inside a span, such as “retry started” or an exception record. The connector can optionally count these as well, with their own dimensions, producing an events counter. This is useful when events matter as much as the spans themselves, for example to chart how often a retry happens. It is off by default, and each extra metric costs memory and storage, so enable it only when it is needed.

Names and Namespaces

The metrics carry names that start with a configurable namespace, which is a prefix that keeps them grouped and separate from other metrics. Older versions used short names, and newer versions use longer names with a fixed prefix for traces-derived span metrics. When the metrics reach a backend with stricter naming rules, such as a Prometheus-style system, dots are turned into underscores, and suffixes may be added, such as a suffix for totals or for units. As a result, the same metric can look slightly different in different tools. Always look at the actual names in your backend before writing queries.

Dimensions decide how many lines your charts have, and how much your system costs.

A single line showing “all requests” is rarely useful. You want to see requests split by service, by endpoint, and by outcome. The labels that do this splitting are called dimensions. The connector comes with a sensible default set. It uses the service name, taken from the resource that describes the producer, the span name, the span kind, and the status code. With these four, you can already draw a chart per service, per operation, split by whether the call was a server handling a request, a client making one, or an internal step, and by whether it succeeded or failed.

Adding Your Own Dimensions

You can add more dimensions by naming attributes that appear on your spans. For example, you might add the HTTP method, the HTTP status code, the route, the region, or a deployment label. For each added dimension, you can also give a default value, used when a span does not carry that attribute. Without a default, spans lacking the attribute are grouped without that label; with a default, they are grouped under the default value, which makes charts tidier and prevents confusing gaps.

Removing Default Dimensions

You can also exclude a default dimension. For instance, if your span names are noisy and you only care about services, you could exclude the span name dimension to get fewer, broader lines. This is a powerful way to control cost, but it reduces detail, so make the choice consciously.

The Cardinality Danger

Each unique combination of dimension values makes one separate series, which is one line on a chart. If you have 20 services, 50 span names each, 3 span kinds, and 3 status codes, that is up to 9,000 combinations, which is already sizeable. Now imagine adding a dimension with a thousand possible values, such as a customer ID. The number of series could multiply by a thousand, reaching millions. Memory in the Collector grows, the network carries more data, and the bill from the metrics backend grows with it. This is called a cardinality explosion, and it is the single most common way for a spanmetrics setup to hurt a company.

A Small Worked Example of Counting Series

Numbers make the danger of cardinality easier to feel, so let us count for a small, imaginary shop-like system. Suppose it has 10 services. On average, each service has 20 distinct operations, meaning 20 span names. Most spans are server spans, but some are client spans, so say 2 span kinds are in use. Three status values are possible: unset, ok, and error, though in practice only two or three appear for each operation.

Multiply these: 10 services times 20 operations times 2 kinds times 3 statuses gives 1,200 combinations. That is a very manageable number. Now suppose the team adds the HTTP method as a dimension, with about 4 common values. The total becomes 4,800. Suppose they also add a region label with 5 values. The total becomes 24,000. Each of these is still reasonable, but notice how quickly the number grows with each added dimension, since each one multiplies the total rather than adding to it.

Now remember the duration histogram. If each series carries a histogram of 15 buckets, many backends will store 15 extra lines for each of the 24,000 combinations, which is 360,000 lines for the histogram alone, plus the calls counter. This is why a team that seems to have added “just two small labels” can see its storage and bill grow many times over. The lesson is to do this multiplication before changing the configuration, not after the invoice arrives.

Span Names Need Special Attention

The span name is a dimension by default, so its values must be few and stable. A good span name describes the operation in general terms, such as “GET /orders/{id}”, with a placeholder for the changing part. A bad span name includes the actual value, such as “GET /orders/84231”, which makes a new name for every order. Most instrumentation libraries produce good names automatically, but custom code and old libraries sometimes produce bad ones. If your span names contain IDs, fix them at the source, or use a processor earlier in the pipeline to normalize them, before the spans reach the connector.

Safety Limits

The connector has safety limits to protect itself. A cardinality limit caps how many series it will track, and beyond that limit, extra combinations are merged into an overflow bucket instead of creating new lines. An expiration setting can forget series that have not been seen for a while, freeing memory. These limits turn a disaster into a degraded but survivable situation, but they hide detail, so they are a safety net and not a design plan. The real protection is choosing dimensions with care.

The duration metric is only as good as the buckets that hold it.

A histogram does not store every duration. It keeps a count for each range, called a bucket. The boundaries of the buckets are chosen in advance. When you later ask for the 99th percentile, your backend estimates it from the bucket counts. If the buckets are wide where your latencies live, the estimate is rough. If they are narrow, it is accurate. So the choice of buckets directly affects how trustworthy your latency charts are.

Explicit Buckets

By default, the connector uses a list of explicit bucket boundaries, a fixed series of values that range from a few milliseconds up to many seconds. The defaults suit many web services, but not all. A service whose requests usually take between 20 and 40 milliseconds will see most of its data fall into only a couple of default buckets, giving crude percentiles. A batch system whose operations take minutes will see everything fall into the top bucket. You can replace the list with boundaries that match your service. A good rule is to place more boundaries around the values you care about, such as around your target latency.

Exponential Histograms

A newer option is the exponential histogram. Instead of a fixed list, it uses buckets that grow by a constant ratio and adjusts itself to the data it sees. It gives good accuracy over a very wide range of values without any manual tuning, and it is often smaller than a hand-tuned explicit histogram. The catch is that your metrics backend must understand this format. Support has been growing, but it is not universal, so check before relying on it.

Choosing the Unit

The duration unit defaults to milliseconds, and it can be changed to seconds. This choice affects the metric’s name and its values in your backend, and some systems prefer seconds as the standard unit. Choose once, early, and keep it, because changing it later makes old and new data look inconsistent.

Following a single span through the connector shows exactly how the numbers are made.

When a batch of spans arrives at the connector, it walks through them one by one. For each span, it performs a small set of steps. First, it builds a key that identifies the group the span belongs to. The key combines the resource information, such as the service name, with the values of the chosen dimensions. Second, it looks up the running totals for that key. If this is the first time it has seen the key, it creates new totals. Third, it adds one to the calls counter for that key. Fourth, it computes the span’s duration and records it in the histogram for that key.

flowchart TD
  S["Span arrives at the connector"] --> K["Build a key from resource and dimension values"]
  K --> L{"Has this key been seen before?"}
  L -->|"No"| N["Create new counter and histogram for the key"]
  L -->|"Yes"| E["Use the existing counter and histogram"]
  N --> C["Add one to the calls counter"]
  E --> C
  C --> D["Record the span duration in the histogram"]
  D --> W["Wait for the next flush"]
  W --> F["Emit all current metrics to the metrics pipeline"]
    

Look at the last two boxes. The connector does not send a metric for every single span. That would be as heavy as sending the spans themselves. Instead, it keeps running totals in memory and, at a regular interval called the flush interval, emits the current totals as metrics to the metrics pipeline. Many thousands of spans become a handful of data points per series per interval. This is the heart of why metrics are so much cheaper than traces: they summarize instead of recording each event.

The Duration Comes From the Span

The connector does not measure time itself. It uses the start and end times that the application recorded inside the span. So the duration it reports is the duration the instrumentation saw. If a span’s clock was wrong, or the span was started late, the metric inherits that error. This is one reason that metrics recorded directly in applications can be more accurate for some uses.

Status and Kind

Each span has a status, which is typically unset, ok, or error, and a kind, which says whether the span is a server handling a request, a client making a request, a producer or consumer in messaging, or an internal step. The connector writes these values as dimensions, and they are very useful. A common dashboard shows only server-kind spans, so that each request is counted once at its entry point, rather than counting the same request again at every client call along the way.

Why Duration Is Not the Same as User Waiting Time

It is tempting to read the duration histogram as “how long users waited”. That is close, but not exact. A span measures the time spent inside one piece of work, as seen by the code that created it. The user’s real waiting time also includes the journey over the internet, the time in browsers or mobile apps, and any queues before the first span begins. A server span that reports 120 milliseconds may belong to a request that felt like half a second to the person holding the phone.

This does not make the metric less useful. It means you should name it honestly on dashboards, for example as “server processing time”, and compare it with measurements taken at the edge, such as load balancer or browser timings, when the true user experience matters. Together they show where the delay lives: inside your services, or outside them.

Resource Grouping

Metrics are grouped by the resource of the spans that produced them, so that each service gets its own set of metrics. The connector keeps separate totals for each distinct resource, using a chosen set of resource attributes as the key. If you keep too many resource attributes in the key, such as an instance ID that changes on every restart, the number of series can grow. A small, stable set of resource attributes keeps the grouping sensible.

The Connector Looks at One Span at a Time

An important detail is that the connector does not wait for a whole trace to finish. It never holds spans in a waiting room to see how the story ends. Each span is counted the moment it arrives, and then it is let go. This makes the connector fast and light on memory, because the only things it stores are the running totals, not the spans themselves.

It also means the connector cannot answer questions that need the whole trace. It cannot tell you how long the entire request took from the first service to the last, and it cannot tell you how many spans a trace contained. Those questions belong to other tools, such as processors that make sampling decisions after seeing a full trace, or the trace backend itself. The connector is a counter and a stopwatch for single spans, not a historian of whole journeys.

Late and Out-of-Order Spans

Spans do not always arrive in a neat order. A service might be slow to export, a network link might be congested, or a batch might be retried. The connector counts a span when it sees it, not when the span actually happened. So if a large group of spans from ten minutes ago suddenly arrives all at once, the calls counter will jump in the current interval, and the chart will show a bump that did not really happen now.

This is usually a small effect, but it is worth knowing when you investigate odd spikes. If a bump appears right after a network problem or after a service restart, late delivery is a likely explanation. Another cause is clock skew, which means two machines disagree about the time. Because durations are calculated from the start and end times inside the span, a span whose end time is earlier than its start time, due to a wrong clock, can produce strange or even negative durations. Keeping machine clocks synchronized with a time service avoids many of these surprises.

Where the Connector Finds Dimension Values

When you name an attribute as a dimension, the connector must find its value. It first looks among the attributes of the span itself. If it does not find the name there, it looks among the attributes of the resource, which describe the service that produced the span, such as its name, its environment, or its region. If the attribute is in neither place, the connector uses the default value you have given, if there is one, or leaves the label out.

This lookup order is helpful. It means you can ask for a deployment environment label without caring whether your instrumentation put it on each span or once on the resource. However, it also means that if a span attribute and a resource attribute share the same name, the span’s own value wins. Keep names distinct and well documented, so that nobody is surprised about which value they are seeing on a chart.

How the totals are reported over time affects how your charts behave.

Cumulative or Delta

The connector can report its counters in two ways. With cumulative temporality, each report carries the total since the connector started, so the numbers only grow. With delta temporality, each report carries only what happened since the last report. Many metrics backends, especially Prometheus-style ones, expect cumulative values and calculate rates themselves. Other backends prefer deltas. Choose the one that matches your backend. For most Prometheus-style setups, cumulative is the right choice, and it is the usual default.

The Flush Interval

The flush interval decides how often the connector emits its current totals. A shorter interval gives fresher charts but more data to send. A longer interval is cheaper but makes charts lag behind real events. A common default is around a minute. Remember that this interval adds to the delay between something happening and the chart showing it. If you alert on span-derived metrics, include this delay in your calculations, along with the time your backend takes to scrape or ingest.

Restarts and Counter Resets

Because the connector keeps its totals in memory, a restart of the Collector clears them. With cumulative temporality, the counter then starts again from zero. Most backends understand this: when a counter suddenly drops, they treat it as a reset and continue calculating rates correctly. So a restart does not usually break rate charts. It can, however, cause a brief dip or a gap, and it means that long-term totals taken from the raw counter are not reliable across restarts. Always use rates, not raw totals, when charting.

Expiration

If a series has not been seen for a long time, for example an endpoint that was retired, the connector can be told to forget it after a set period. This frees memory and keeps charts tidy. If no expiration is set, series live as long as the Collector does, which in a long-running gateway with changing span names can slowly grow memory use. Setting a sensible expiration is good housekeeping.

Where the connector sits relative to sampling decides whether your numbers are true or wrong.

Keeping every trace can be very expensive, so many systems sample: they keep only a portion, such as one in ten, or only the slow and failed ones. Sampling is a good idea for storage. But it has a hidden effect on metrics derived from traces. The connector counts the spans it sees. If it sees only the sampled spans, then its counts are only a fraction of the real traffic, and its latency histogram is built from a biased selection. A request rate chart would then show one tenth of the true rate, and, if the sampler favors errors or slow requests, the error rate would look far worse than reality.

flowchart LR
  A["All spans arrive"] --> B["Wrong: sampling first"]
  B --> C["Only kept spans reach the connector"]
  C --> D["Metrics undercount and are biased"]
  A --> E["Right: connector sees every span"]
  E --> F["Metrics are accurate"]
  E --> G["Sampling decides which traces to store"]
    

The fix is simple in principle: let the connector see every span, and apply sampling only on the path to the trace backend. In practice, this means arranging pipelines so that spans first flow into the connector, and only afterwards into a separate path where a sampling processor decides what to store. One convenient way to do this is to use a pipeline that receives from the incoming spans, sends them to the connector, and forwards them to a second traces pipeline that contains the sampler. A simple forwarding connector exists for exactly this kind of wiring.

Sampling in the Application Is a Trap Too

A subtler version of the same problem happens before the Collector. Many applications use head sampling in their SDK, which means they decide at the start of each trace whether to record it, and unsampled spans are never even exported. Those spans never reach the Collector, so the connector never sees them. If you rely on the connector for accurate rates, your SDKs must send all spans to the Collector, and sampling should be done in the Collector afterwards. This increases the volume sent from applications to the Collector, which is a real cost, so weigh it against the value of accurate derived metrics.

Why Counting Everything Costs Something

Sending every span to the Collector is the price of accurate metrics, and it is worth understanding that price. Applications spend a little more processor time and memory creating and exporting spans. The network carries more data between applications and the Collector. The Collector spends more effort receiving, batching, and copying spans. If you also store every trace, the storage bill rises further, which is why sampling exists in the first place.

The good news is that this cost can be placed where you can control it. You can let all spans reach a Collector close to your applications, count them there, and then sample before sending the rest across expensive links to a distant trace backend. Because the metrics are tiny compared with traces, they travel cheaply. In this way you pay for full counting only on the short, cheap leg of the journey, and you save on the long, expensive one.

When Approximations Are Enough

If you are happy with approximate numbers, you can compensate for known sampling rates by multiplying counts by the inverse of the rate. This only works when the sampling is uniform and the rate is known and constant. Anything more complicated, such as keeping all errors and some successes, makes simple correction unreliable. For trustworthy dashboards and alerts, the cleaner approach is to count everything before sampling.

A single Collector is simple. A fleet of them needs a plan.

As traffic grows, one Collector instance cannot handle it all, so you run several behind a load balancer. This is normal and works well for most Collector features. But the spanmetrics connector keeps running totals in memory, and that raises two issues.

Issue One: Colliding Series

Suppose three Collector copies each receive a third of the spans for the checkout service. Each one builds its own counter for that service, and each one emits a metric with exactly the same name and labels. In a metrics backend, these three streams look like one series that keeps being overwritten by whichever copy reported last. The chart then jumps wildly or shows only a third of the traffic. This is a classic scaling mistake.

The simplest cure is to give every Collector copy a unique label, such as an instance identifier, added to the metrics it produces. Then the three streams become three separate series, and the backend can add them up to get the true total. The price is that the number of series is multiplied by the number of Collector copies, which is acceptable for a handful of copies but grows with larger fleets.

Issue Two: Splitting a Trace’s Spans

A different cure is to organize traffic so that all spans of one service always go to the same Collector copy. A load-balancing exporter in a first tier of Collectors can route spans by service name or by trace ID to a second tier, so that each service’s spans land on one specific copy. Then each series is produced by exactly one Collector, and no unique label is needed. The cost is extra complexity: you now run two tiers, and you must handle what happens when a copy fails and its share moves to another.

Memory and Capacity

The connector’s main resource cost is memory for the running totals, which grows with the number of series. Number of series is the product of resources, span names, span kinds, status codes, and extra dimensions. Planning capacity starts with estimating this number from your real data, and watching it as services and endpoints are added. CPU cost is modest, since each span only triggers a lookup and a few additions.

A chart tells you something is wrong. An exemplar helps you find an example.

Imagine your latency chart shows a spike in the slowest bucket. You know that something took too long, but not what. An exemplar is a small sample attached to a metric point that records the trace ID and span ID of one real span that contributed to it. With exemplars, you can click on the spike and jump straight to an actual slow request, with all its detail. This closes the gap between the summary view of metrics and the detailed view of traces.

The spanmetrics connector can attach exemplars to its histogram. Because it sees the spans themselves, it has the trace and span IDs on hand. When enabled, each histogram bucket may carry a recent exemplar from a span that fell in that bucket. This is one of the biggest advantages of deriving metrics from traces: the link between the two is built in, instead of being a separate effort.

What You Need for It to Work

Exemplars only help if the rest of your tools understand them. Your metrics backend must store and show exemplars, your dashboard tool must display them as clickable points, and your trace backend must be reachable from the link. Also, exemplars that point to spans that were later dropped by sampling lead nowhere. This is another reason to design sampling carefully, for example by keeping slow and failed traces, which are the ones exemplars most often point at.

Let us follow one request, and the metrics it creates, from start to finish.

A shopper presses “Place order” in a mobile app. The request reaches a checkout service that has been instrumented to send traces. The platform team runs a Collector gateway with the spanmetrics connector, set up to see every span before any sampling.

Notice what the application team never did. They wrote no metric code. They did not agree on names with other teams. They did not set up a metrics library. The platform team’s single Collector configuration produced consistent RED metrics for every service that sends traces, including old services and third-party ones that nobody could modify.

Several tools can give you metrics. Choosing the right one saves effort.

Application Metrics or Spanmetrics?

If you control the code and have the time, metrics recorded directly in applications are usually the gold standard. They count everything regardless of tracing and sampling, can include business measures such as items per order, and cost less because they skip the trace step. Spanmetrics is best when you want quick, uniform coverage across many services, when some services cannot be changed, or when you want the tight link between metrics and traces. Many teams use spanmetrics to get started and then add application metrics for the most important services.

Moving From the Old Processor to the Connector

Many teams started with the older spanmetrics processor and now need to move. The change is more than renaming a line, because the connector lives in a different part of the configuration and is wired differently. A calm, step-by-step migration avoids broken dashboards.

Spanmetrics or Service Graph?

The service graph connector answers a different question. It matches the client side of a call with the server side and produces metrics about the relationship between two services, such as how many calls service A made to service B and how many failed. This lets tools draw a map of which services depend on which. The spanmetrics connector, by contrast, describes each service’s own operations. The two are complementary, and some teams run both in the same Collector.

Spanmetrics or Count Connector?

The count connector counts telemetry items that match rules you write, such as “spans with this attribute”. It is good for custom counters. But it does not build latency histograms by default. If you want rate, errors, and duration for operations, the spanmetrics connector is purpose-built.

An honest look at what the connector gives and what it costs.

The Key Trade-offs

The first trade-off is convenience versus precision. The connector gives broad coverage with no code, but application metrics are more exact and flexible. The second is coverage versus cost. Counting every span gives true numbers, but sending all spans to the Collector costs bandwidth and processing, and may push you to store fewer traces. The third is detail versus cardinality. More dimensions answer more questions but multiply series, memory, and bills. Each of these is a dial, and good setups tune them with real data rather than guesses.

The connector is light per span, but a busy gateway sees a great many spans.

For each span, the connector builds a key, looks it up, and updates a counter and a histogram. This is a small, constant amount of work, but at hundreds of thousands of spans per second it adds up. Two things dominate its cost: the work to build keys from attributes, which grows with the number of dimensions, and the memory to hold the running totals, which grows with the number of series.

Habits That Keep It Efficient

• Use few dimensions. Each extra dimension makes keys longer and series more numerous.
• Normalize span names early. Cleaning names before the connector prevents series growth.
• Set a cardinality limit and expiration. They protect memory against surprises.
• Choose a sensible flush interval. Very short intervals increase output volume.
• Pick a histogram that fits. More buckets mean more series per histogram. Exponential histograms can be compact.
• Filter out noisy spans first. Health checks and internal chatter that nobody charts can be removed before the connector, saving work.
• Measure. Watch the Collector’s own memory, CPU, and the number of active series before and after changes.

Estimating Series Count

A practical way to plan is to estimate the number of series as the number of distinct services multiplied by the average number of operations per service, multiplied by the number of span kinds and status codes in use, multiplied by the number of values of each extra dimension. Then multiply by the number of histogram buckets for the duration metric, because each bucket becomes its own line in many backends. A modest setup might produce tens of thousands of series. A careless one can produce millions. Doing this arithmetic early, with real numbers, avoids unpleasant surprises.

Turning span fields into metric labels moves data into a new place with its own rules.

Traces are often stored with tight access controls and short retention, because they can contain detailed information about individual requests. Metrics, by contrast, are usually shared widely, kept for a long time, and shown on public dashboards. When the connector turns a span attribute into a metric dimension, that attribute’s values now live in the metrics system, with its broader audience and longer life. So the choice of dimensions is also a privacy decision.

Who Should Control the Configuration

Because the connector decides which span fields become long-lived metric labels, its configuration deserves the same care as a database schema. Treat changes as reviewed work, not quick edits. A simple habit is to keep the Collector configuration in version control, require a second person to review any added dimension, and record the reason for it. When someone later asks why a label exists, the answer is written down.

It also helps to give teams a safe way to ask for new dimensions. A short request that states the attribute name, the expected number of distinct values, whether it can contain personal data, and which chart or alert needs it gives the platform team everything required to say yes quickly or to suggest a safer alternative. This keeps the system open to improvement without letting the series count drift upward unnoticed.

Practical Rules

• Never use personal or secret values as dimensions. Email addresses, user names, account numbers, and tokens do not belong in metric labels. They also cause cardinality explosions.
• Check span names. If an application puts personal data into span names, it will appear as a dimension value. Normalize or remove it before the connector.
• Limit who can see metrics dashboards that expose operation names. Even names of endpoints can reveal information about how your system works.
• Clean attributes upstream. A processor that removes or hashes sensitive attributes before the connector protects both traces and metrics.
• Secure the Collector configuration. Anyone who can edit it can add a risky dimension and spread sensitive values into metrics.
• Use encrypted connections between the Collector and the metrics backend, as with any telemetry.

When the numbers look wrong, a calm, step-by-step approach finds the cause.

A Methodical Order

First, confirm the wiring: the connector must appear in the exporters of a traces pipeline and in the receivers of a metrics pipeline, and the metrics pipeline must have an exporter. Second, look at the metrics as they leave the Collector, using a debug exporter on the metrics pipeline, so you can see names, labels, and values before the backend changes anything. Third, compare counts against a trusted source, such as a load balancer’s request count, to check for missing spans. Fourth, count the series and look for labels with many values. Fifth, check for multiple Collector copies and whether their series collide. Finally, review any processors before the connector that might drop, rename, or sample spans.

Alerts and Service Level Objectives Built on Spanmetrics

One of the best uses of span-derived metrics is alerting. A service level objective, often shortened to SLO, is a promise about quality, such as “ninety-nine out of a hundred requests succeed” or “ninety-five out of a hundred requests finish within a set time”. The connector’s calls counter and duration histogram are exactly the raw material for such promises.

For a success promise, divide the number of calls with error status by the total number of calls over a window of time. For a speed promise, use the histogram to find the share of requests that finished under your chosen time. If you placed a bucket boundary at that exact time, this is a simple and precise reading rather than a guess. When the measured value drifts too close to the limit, your alerting tool can warn the on-call person before customers feel serious pain.

Two cautions apply. First, remember that the metrics are only as complete as the spans that reach the connector. If a sender fails and stops exporting spans, the metrics will fall quietly, and a request-rate alert is then the only clue. Second, low-traffic services produce noisy ratios: with ten requests in a window, a single failure is a ten percent error rate. Use longer windows or minimum-traffic conditions so that alerts do not fire on tiny numbers.

Watching the Connector Itself

The Collector exposes its own metrics, such as how many spans each component received and refused, how much memory it uses, and how full its queues are. Watch these after any change. Also track the number of active series coming out of the connector, because a sudden rise is the earliest sign of a cardinality problem. Alert on it, so you learn about it before your bill or your Collector does.

Habits that keep spanmetrics accurate and affordable, and habits that quietly spoil it.

Patterns That Work

• Count first, sample later. Wire the connector so it sees every span, and apply sampling only on the path to trace storage.
• Keep dimensions small and stable. Start with the defaults, add one at a time, and review each.
• Normalize span names. Ensure names use route templates, not raw values.
• Match buckets to goals. Put boundaries where your latency objectives sit.
• Plan for many Collectors. Use unique instance labels or route by service.
• Chart by server kind. Count each request once at its entry point.
• Enable exemplars with a good sampling policy. Keep error and slow traces so exemplar links work.
• Treat names as a contract. Pin the namespace, unit, and version, and test upgrades in a staging Collector.

Anti-Patterns to Avoid

A Short Checklist

• Does the connector see every span before any sampling?
• Is it listed as an exporter in a traces pipeline and a receiver in a metrics pipeline?
• Are span names low in cardinality, and are extra dimensions bounded and non-sensitive?
• Do bucket boundaries match your latency goals, and is the unit fixed?
• Is there a plan for multiple Collector copies?
• Are cardinality limits and expiration set, and is the series count monitored?
• Are dashboards built on server spans, with rates rather than raw totals?
• Have you pinned the version and tested upgrades before rollout?

Common situations that show the connector at work. These are illustrative patterns frequently seen in production systems.

Short answers to questions that appear again and again.

If you remember only a few things, remember these.

The spanmetrics connector is the Collector’s bridge between traces and metrics. It watches spans flow by, counts them, measures their durations, and emits request, error, and latency metrics, while the spans themselves continue on untouched. It gives broad, consistent, no-code coverage, and its power comes with responsibilities: see every span, choose dimensions carefully, tune histograms, and plan for scale.