Before we meet the exporter, we need to understand the problem it exists to solve.

A quick picture of the OpenTelemetry Collector

The OpenTelemetry Collector is a program that receives telemetry data, which means traces, metrics and logs, then processes it and sends it onwards. It has three kinds of building blocks. Receivers accept data from applications. Processors change or filter the data on its way through. Exporters send the data to its next destination, which might be a storage system, a vendor service, or another collector. These pieces are joined together in pipelines, and one collector can run several pipelines at once.

In a small system a single collector is enough. As the system grows, one collector can no longer cope, so teams run many collectors side by side. At that point a new question appears: when an application or an upstream collector sends data to this group, which member of the group should receive each piece?

When any destination will do, and when it will not

For many jobs the answer is easy. If a collector only needs to compress data and forward it, any member of the group can do the work, and you can spread the load evenly with an ordinary load balancer. This is called stateless work, because each piece of data is handled alone and nothing needs to be remembered between pieces.

Other jobs are stateful. A tail-sampling processor, for example, must see all the spans of a trace before it decides whether to keep that trace. A component that turns spans into metrics must see all the related spans together to produce correct numbers. A component that combines cumulative metrics needs all the points of one series. For these jobs, scattering related data across different collectors breaks the result.

How collectors are usually arranged

Teams arrange collectors in a few common shapes. In the agent shape, a small collector runs on every machine or beside every group of applications. It collects data locally, adds useful labels such as the machine name, and forwards onwards. Agents are cheap, numerous and mostly stateless. In the gateway shape, a smaller group of larger collectors receives data from many agents or applications, performs heavier processing, and sends the result to storage. Gateways are fewer, and some of them hold state.

As traffic increases, the gateway shape often needs to be split into two layers, because the stateless parts of the work and the stateful parts of the work scale in different ways. Stateless work grows by adding more identical members behind an ordinary balancer. Stateful work grows only if each member owns a clear share of the data. The load balancing exporter is the device that hands out those shares, and it is the reason a two-layer gateway can work at all.

A handful of terms will appear again and again in the rest of this guide.

Why a trace identifier is such a good key

Every span in a trace carries the same trace identifier, which is a long random number created when the trace begins. Because it is random, identifiers spread out evenly across any group of backends. Because it is shared by every span in the trace, using it as the routing key sends the whole trace to a single backend. These two properties together are why the trace identifier is the default and most popular routing key for traces.

More vocabulary that helps

These terms sound technical, but each one maps to something simple. OTLP is the common language. A pipeline is an assembly line. A batch is a box of items. A hot spot is one overloaded worker. A headless service is a phone book that lists every person rather than only the switchboard. Sticky routing is the habit of sending the same customer to the same clerk. Keeping these pictures in mind will make the rest of the guide much easier to follow.

Now we can state the definition plainly.

The load balancing exporter is an exporter in the contributed part of the OpenTelemetry Collector ecosystem. Its job is to take incoming telemetry, split it into groups according to a routing key, and send each group to the backend that the key maps to. It keeps a separate connection to every backend, and it keeps that list of backends up to date by using a resolver.

Internally it builds on the standard OTLP exporter, which is the usual way of sending data between collectors. For each backend it creates one OTLP exporter with the same settings, and it then acts as a clever front door to this family of exporters. This means features such as encryption, compression, retries and sending queues behave in the way you already know from the ordinary OTLP exporter.

What it is not

It is easy to mistake this component for a general traffic balancer, so it helps to say what it is not. It is not a replacement for a network load balancer. It does not measure the health or speed of backends. It does not guarantee perfectly equal shares. It does not store data. It does not process data in any way other than grouping and routing it. It is a routing exporter that gives you stickiness by key.

What you configure, described in words

Setting up the exporter involves a small number of decisions, and it helps to see them as a short list of questions. First, which signals will pass through it: traces, logs, metrics, or a combination. Second, which routing key applies to each signal. Third, how will the backends be found, which means choosing a resolver and supplying its details, such as the hostname and port for DNS or the service name for Kubernetes. Fourth, how will the exporter talk to each backend, which means the protocol settings that mirror the ordinary OTLP exporter, including encryption, compression, timeouts, retries and queue size.

Because the per-backend settings are shared, every backend is treated the same. You cannot easily give one backend special settings. This keeps the design simple, and it also means that your configuration must work for the slowest or most demanding backend in the group.

The exporter is small, but it has several moving parts that cooperate.

flowchart LR
  A[Applications and agents] --> R[Tier 1 collectors with load balancing exporter]
  R --> B1[Tier 2 collector 1]
  R --> B2[Tier 2 collector 2]
  R --> B3[Tier 2 collector 3]
  B1 --> S[Tracing backend]
  B2 --> S
  B3 --> S
        

The tier one collectors are normally stateless and numerous. They receive data from applications or from agents running next to applications, perhaps do light processing such as adding labels, and then pass everything to the load balancing exporter. The tier two collectors are fewer and stateful. They hold the data for as long as needed, run the tail-sampling or aggregation logic, and send the final results to storage.

Sizing the two tiers differently

The routing tier and the stateful tier have very different needs, so they should be sized by different rules. The routing tier mostly spends processor time receiving data, reading keys and sending batches onward. Its capacity should follow the total volume of incoming telemetry. Because it holds almost no state, you can add or remove members freely, and it recovers instantly from restarts.

The stateful tier mostly spends memory. A tail-sampling processor must keep all the spans of every trace that has not yet been judged. The amount of memory therefore depends on how many traces arrive per second, how many spans each trace holds, how large each span is, and how long the processor waits before deciding. Doubling the wait roughly doubles the memory needed. Increasing traffic does the same.

Failure domains

Think about what happens when something breaks. If a routing tier member fails, the data it was holding is small and the other members carry on, so the loss is brief. If a stateful tier member fails, every trace it was still gathering is lost, which may be a large number. This is why many teams run the stateful tier on stable machines with generous limits and treat its restarts as events to plan, while treating the routing tier almost as disposable.

The routing key decides which pieces of data stay together, so it deserves careful thought.

Routing keys differ by type of telemetry, and the available choices have grown over time. Check the documentation for the version you run, because exact names and options can change. The common ideas are described below.

Traces

For traces the default key is the trace identifier. All spans of a trace go to one backend, which is what tail sampling needs. A second option is the service name, in which case all spans produced by one service go to the same backend. This is useful for components that compute figures per service, such as request counts and latency for each service.

Logs

For logs the same ideas apply in simpler form. Routing by trace identifier keeps the logs that belong to one request together, and routing by service keeps one service’s logs together. Some versions offer further choices, such as routing by resource or by a chosen attribute.

Metrics

Metrics are more varied. Common choices include routing by service, by resource, by the metric name, or by a stream identity that represents one exact time series. Each choice keeps a different set of data together. Routing by metric name keeps every series of one metric on a single backend, while routing by stream identity keeps each individual series together, which spreads load more finely.

A simple way to choose the key

Ask one question: what must the receiving collector see together in order to do its job correctly? If the answer is “every span of a trace”, use the trace identifier. If the answer is “every span from one service, so I can count requests for that service”, use the service. If the answer is “every point of one metric series”, use the stream identity. The key is never an arbitrary choice; it follows directly from the job downstream.

Mixed pipelines

A single collector can handle traces, logs and metrics at once, and each signal can use a different key. For example, traces may be routed by trace identifier for tail sampling, while metrics may be routed by service for aggregation. The two do not need to agree, because they are separate pipelines. What matters is that every signal’s key matches the job its own downstream processor performs.

The central trick is a mapping that survives change.

Suppose you have four backends and a simple rule: take the key, turn it into a number, divide by four, and use the remainder to pick the backend. This works, until the day you add a fifth backend. Now the remainder is calculated differently, and almost every key moves to a different backend. All the traces currently being gathered at their old homes suddenly have their remaining spans sent elsewhere. This is disastrous for stateful work.

Consistent hashing avoids this. Imagine a circle marked with a huge range of numbers. Each backend is placed on the circle at several points, found by hashing its name together with a counter. To route a key, you turn it into a number, find that number on the circle, and walk clockwise until you meet the first backend marker. That backend receives the key. When a backend is added, it takes over only the stretches of the circle just before its markers, and everything else stays where it was. When a backend is removed, only the keys it owned move to its neighbours.

flowchart TD
  K[Key such as trace identifier] --> H[Hash the key to a number]
  H --> P[Find that point on the ring]
  P --> W[Walk clockwise to the next backend marker]
  W --> B[Send data to that backend]
        

Why several markers per backend

If each backend had only one marker, the stretches of the circle between markers would differ widely in size, and some backends would receive far more keys than others. Placing each backend at many points averages out these differences, giving a much more even spread. The cost is a slightly larger ring and a slightly longer lookup, both of which are very small.

What consistent hashing does and does not give you

It gives you minimal movement when the group changes, and an approximately even spread for well-mixed keys. It does not give you perfect fairness, and it does not move any keys for load reasons. If one backend is slower than the rest, the ring keeps sending it its share. This is a deliberate trade, because moving keys would break the very stickiness the exporter exists to provide.

A worked example in words

Suppose you have four backends. With the simple remainder method, a key whose number leaves remainder two goes to the third backend. Now add a fifth backend. The same key, divided by five, may leave remainder four, so it moves. In fact, roughly four out of every five keys end up with a different backend. With a ring, adding the fifth backend takes over only about one fifth of the circle in total, scattered in small stretches, and those are the only keys that move. Four out of five keys stay exactly where they were.

The difference is dramatic for stateful work. Under the remainder method, adding a backend causes nearly every trace in progress to be split. Under the ring, only a fifth of them are affected, and often less because many traces finish before the change matters.

Determinism across routers

Many routing collectors run side by side, and each builds its own ring. For stickiness to hold, every routing collector must build the same ring from the same list of backends and map every key to the same place. This works because the hashing is a fixed calculation: the same inputs always give the same result. It holds only if all routers see the same backend list. If two routers have different views of the list for even a short time, they will send the same trace to different backends during that time.

The ring is only as good as its list of backends, and the resolver supplies that list.

Choosing among them

A DNS subtlety that surprises beginners

For DNS discovery to work, the name must return the address of every backend, not a single virtual address that hides them. In Kubernetes this is usually done with a headless service, which publishes one address per pod. If you point the resolver at an ordinary service that returns only one virtual address, the exporter believes there is a single backend and sends everything to it, and the whole point of the arrangement is lost.

Tuning the DNS resolver

The DNS resolver has a few settings worth understanding. The refresh interval controls how often the resolver asks for the current list of addresses. A short interval notices change quickly but increases the number of DNS queries. A long interval is gentler on DNS but leaves the ring out of date for longer. The timeout controls how long the resolver waits for an answer before giving up and keeping the previous list. A sensible timeout prevents a slow DNS server from freezing the routing tier.

Also remember that DNS answers are sometimes cached by the operating system or by intermediate servers. Even when the resolver refreshes often, a cached answer may still be returned. Check the caching behaviour of your platform so that the effective delay matches your expectations.

Tuning the Kubernetes resolver

The Kubernetes resolver watches the endpoints behind a named service and updates the ring as pods become ready or disappear. It needs the name and namespace of the service and the ports to use. Some versions can return hostnames instead of raw addresses, which is helpful when certificates are issued for names rather than addresses. Because it watches directly, it reacts faster than DNS, but it needs permission to read the relevant objects, so an access role must be created for it.

When to use the static resolver

A static list is entirely suitable for small, stable setups, for example two stateful collectors on fixed machines. It has no moving parts to fail and is easy to reason about. Its weakness is that changes require editing the configuration. If you scale rarely and carefully, as recommended for the stateful tier, this weakness may be an acceptable price for simplicity.

Following one batch of spans through the exporter makes the whole mechanism concrete.

sequenceDiagram
  participant App as Application
  participant T1 as Tier 1 collector
  participant LB as Load balancing exporter
  participant R as Resolver
  participant B1 as Backend 1
  participant B2 as Backend 2
  R-->>LB: Current list of backends
  App->>T1: Batch of spans
  T1->>LB: Pass batch to exporter
  LB->>LB: Read trace identifier of each span
  LB->>LB: Look up each key on the ring
  LB->>B1: Spans for traces mapped to backend 1
  LB->>B2: Spans for traces mapped to backend 2
  B1-->>LB: Acknowledgement
  B2-->>LB: Acknowledgement
        

An important detail is that splitting a batch costs a little processing, because every item must be inspected. For this reason the load balancing exporter is usually placed in a lightweight tier whose only duty is routing, and it is sized to cope with the full volume of incoming data.

Connections and fan-out

Every routing collector keeps a connection to every backend. If you have twenty routing collectors and ten stateful ones, that is two hundred connections. This number grows quickly with the sizes of both tiers. Each connection consumes memory and a small amount of processor time, and each carries its own queue. At very large sizes, connection counts and queue memory become real considerations, and some teams shard the arrangement into several independent groups to keep each group manageable.

Batch size and efficiency

Splitting a batch into many small ones reduces efficiency. If a batch of one thousand spans is spread across ten backends, each backend receives roughly one hundred spans per batch. Smaller messages cost more per item than larger ones because each carries fixed overhead. For this reason it is wise to build large batches before the routing step, for example by using a batch processor earlier in the pipeline, so that after splitting each backend still receives reasonably sized messages.

Order inside a batch

The exporter groups items by backend, but the order in which different traces appear inside the original batch is not meaningful. Each backend receives its own group, and within it the items for each trace remain together. Downstream processors must not rely on a particular ordering across traces. This is normal for telemetry, but it is worth remembering when you design processing steps that look for patterns across neighbouring items.

The exporter became famous because it makes tail-based sampling possible in a scaled-out system.

Tail sampling means deciding whether to keep a trace after it has finished, using facts such as whether it contained an error or how long it took. To do this, a collector must receive every span of a trace. If the spans of one trace are spread over three collectors, each collector sees only a fragment. One may see a fast, healthy fragment and discard it, while another sees the failing fragment and keeps it. The result is broken, misleading traces.

The load balancing exporter in tier one fixes this. By routing on trace identifier, it guarantees that all the spans of a trace reach one tier two collector, where the tail-sampling processor waits for the trace to complete and makes a sound decision. Only the traces chosen by the policies are then forwarded to storage.

The decision wait: a central tuning knob

The tail-sampling processor must decide how long to wait after seeing the first span of a trace before judging it. If it waits too little, slow traces are judged before their later spans arrive, and the processor sees an incomplete and misleadingly fast trace. If it waits too long, memory fills with traces that are already finished but still waiting for the clock. The right value is slightly longer than your slowest ordinary request, and you can find it by looking at the distribution of trace durations in your system.

The routing tier plays a supporting role here. Because the routing tier only forwards data, it does not add meaningful delay. But network delays, retries and queues can make spans arrive late, so the decision wait should include a margin for these as well.

Policy ordering and cost

Tail-sampling policies are evaluated for each trace, and some are more expensive than others. A policy that checks the status of spans is cheap. A policy that inspects many attributes is more expensive. Placing cheap policies first and thinking carefully about which traces need the expensive ones helps keep the stateful tier’s processor use manageable.

Anything that needs related data in one place can benefit.

The shared theme is state. Wherever a processor needs to remember something about earlier items in order to handle later items correctly, the load balancing exporter provides the guarantee that the earlier and later items meet at the same place.

Choosing between routing and doing less

Before adding a routing tier for any of these uses, it is worth asking whether the stateful processing could be simplified or moved. Sometimes the same result can be reached by computing figures in the applications themselves, for example by recording request counts and latency as metrics at the source, instead of reconstructing them from spans later. Metrics recorded at the source are complete by nature and need no routing at all.

The routing tier earns its place when the logic truly needs to see data from many sources together, as tail sampling does. It is less compelling when the same information is available in a simpler form. Each extra tier carries ongoing costs in machines, monitoring and attention, so adopt it for clear reasons and review the reasons from time to time.

The hardest moment for any stateful design is when the group changes size.

flowchart TD
  A[Resolver notices a backend was added or removed] --> B[Ring is rebuilt]
  B --> C[Most keys keep their backend]
  B --> D[A small share of keys moves]
  D --> E[Traces in progress may be split across two backends]
  E --> F[Some traces are judged on fragments]
        

Consistent hashing keeps movement small, but small is not zero. When a backend is added, a fraction of keys, roughly one divided by the new number of backends, will move to the newcomer. For traces still in progress at that moment, the spans received before the change sit on the old backend, and the spans received afterwards go to the new one. The tail-sampling decision for those traces may then be made on incomplete evidence.

The same happens when a backend disappears, either through planned shutdown or sudden failure. In the sudden case the data held in that backend’s memory is also lost, because tail-sampling processors keep their working data in memory. Resolvers also differ in how fast they notice the loss, so for a short period the exporter may keep sending data to an address that no longer answers.

Ways to soften the impact

Scale tier two slowly and deliberately rather than letting an autoscaler add and remove members every few minutes. Prefer scaling during quiet periods. Give tail-sampling processors a decision wait long enough for normal traces to finish, but not so long that memory balloons. Make sure shutdown gives backends time to finish and flush work. Keep a metric of how many traces were judged on fragments so that you can see how much damage your scaling pattern causes.

A gentle scaling routine

Because scaling the stateful tier is the riskiest operation, it helps to follow a routine. The goal is to keep the number of split traces small and to be able to explain any that occur.

Accepting a small, known loss

No routine removes the effect entirely. Teams that are comfortable with sampling usually accept that a small share of traces around a scaling event will be incomplete, and they record the times of such events so that engineers can explain an odd trace. The aim is not perfection but predictability: scaling events should be rare, scheduled and understood.

Because the exporter sits in the middle of a data path, its failure behaviour matters a great deal.

Per-backend queues and retries

Each backend has its own exporter with its own sending queue and retry rules. This isolation is helpful: if one backend becomes slow or unreachable, its queue fills and its retries pile up, but the other backends continue to receive data normally. A problem with one tier two collector does not stop the others from working.

The same isolation has a cost. A single unhealthy backend still owns a share of the keys, so data for those keys keeps heading to it. Once its queue is full, new data for that share is refused or dropped, while data for the rest flows on. The overall effect is that part of your traces vanish while the remainder looks healthy, which can be hard to notice without good monitoring.

What the exporter does not do for you

The exporter does not actively probe each backend to see whether it is healthy, and it does not automatically move keys away from a struggling backend. Its view of the world is whatever the resolver reports. If a backend is alive according to the resolver but too slow in reality, the exporter keeps sending it data. Health must therefore be managed by the platform around it, for example through readiness checks that cause an unhealthy pod to be removed from the list the resolver sees.

Back-pressure and memory protection

When a backend slows down, the exporter’s queue for that backend grows. Left alone, the growth could consume all memory in the routing collector and take it down, which would affect traffic destined for healthy backends too. A memory limiter processor, placed early in the pipeline, prevents this by refusing new data when memory use crosses a threshold. The upstream senders then retry or drop, depending on their own settings. This is back-pressure: the problem at the destination is communicated backwards rather than hidden.

Back-pressure is uncomfortable but healthy. It tells the whole system that capacity has been reached. The worst designs are those that hide the problem by buffering without limit, because the failure then arrives suddenly, all at once, and with a large amount of data lost.

Persistent queues

Some versions allow queues to be stored on disk instead of memory. A disk queue survives restarts of the routing collector, so data waiting to be sent is not lost when the process stops. The cost is slower handling, extra disk use, and operational care for the storage. For many teams, telemetry is valuable but not irreplaceable, and an in-memory queue is an acceptable choice. For others, such as those with audit duties, disk queues provide extra assurance.

Like every design choice, the exporter trades one set of problems for another.

Is the extra tier worth it?

The honest answer depends on your needs. If you only need to receive and forward data, a simple setup with a standard load balancer is cheaper and easier. If you need tail sampling, span-derived metrics, or any other stateful processing across several collectors, the extra tier is the price of correctness. The exporter is therefore a tool you adopt when the requirement forces you to, not one you add for its own sake.

Cost in plain terms

Running a second tier is not free, and it helps to count the costs honestly. There are more machines to pay for, because the routing tier is additional to the stateful tier. There is more network traffic, because data crosses an extra hop, and in many cloud environments traffic between machines in different zones has a price of its own. There is extra work for the team, who must monitor, upgrade and secure one more layer.

There are also savings. Tail sampling reduces the amount of data stored, which can be a large share of an observability bill. Better decisions mean that engineers see the traces they need and not a flood of boring ones, which saves time during incidents. For many organisations the savings exceed the costs, but only for those that actually need the stateful behaviour.

Zone awareness

In a cloud with several zones, the exporter does not know which zone each backend lives in, and it does not prefer nearby ones. Keys are mapped by hash alone, so a large share of traffic will cross zones. If cross-zone traffic is costly or slow in your environment, consider placing both tiers within the same zone, or accept the cost as the price of consistent routing. Choosing a key by locality would break the very evenness and stickiness the ring provides.

A routing layer sees all of your telemetry, so it deserves the same care as any critical system.

Encrypting the hop

Because the exporter builds on the OTLP exporter, it supports encrypted connections and client certificates to each backend. Telemetry often contains sensitive details such as user identifiers, internal addresses and parts of queries, so traffic between tiers should be encrypted, and the identity of both sides should be verified where possible. Remember that encryption must be set up for every backend, and that certificates must match the names or addresses the resolver supplies.

Permissions for discovery

Some resolvers need permission to learn about backends. The Kubernetes resolver, for example, must be allowed to read information about services and their endpoints. Grant it only what it needs and nothing more, and review the permission as part of your regular security checks. A resolver with too little permission fails to discover backends, while one with too much permission widens the damage if the collector is compromised.

Who may change the routing

The configuration of the routing layer decides where data goes and how it is grouped. A mistake or a malicious change could send data to the wrong place, stop it from arriving, or cause overload. Keep the configuration under version control, require review for changes, and restrict who may deploy them.

Certificates and names

Encrypted connections rely on certificates, and certificates are tied to names. When the resolver supplies raw addresses, the router connects to an address and must still check that the certificate presented belongs to the expected service. If certificates are issued for hostnames, but the resolver provides addresses, the check may fail. Use resolver options that return hostnames, or issue certificates that cover the addresses or a shared service name, and test the setup before relying on it.

Certificates also expire. A routing tier whose certificates expire quietly will stop sending data, and the failure will look like a network problem. Automate renewal and monitor expiry dates as you would for any other critical service.

Limiting what leaves the routing tier

The routing tier sees raw data from many sources, including data that has not yet been scrubbed. If your design includes a step that removes sensitive attributes, place it before the routing step, so that sensitive values are removed as early as possible and never travel across the second hop. This also reduces the volume of data to be routed, which saves effort.

A routing layer that fails quietly can damage data without anyone noticing.

The exporter publishes internal measurements about its own behaviour. Names and details vary between versions, so check the documentation, but the ideas are consistent. Typical figures include how many times the resolver has run, how many backends the exporter currently knows about, how long exports to each backend take, and whether those exports succeeded or failed.

Also monitor the tier two collectors for the effects of routing, such as memory use of the tail-sampling processor and the number of traces dropped before a decision was made. Together these show whether the exporter and the stateful tier cooperate well.

Building a useful dashboard

A good dashboard for this layer tells a story from left to right. On the left, show how much data enters the routing tier. In the middle, show how that data is divided among backends, ideally as one line per backend so that imbalance is visible. On the right, show what the stateful tier does with it, such as traces kept and traces dropped before a decision. Under these, show the health numbers: backend count, failure rate, queue length and memory use.

Include annotations for deployments and scaling events. Many puzzling changes in the graphs turn out to coincide with a change that someone made, and annotations let you see that at a glance. Teach the on-call engineers to read the dashboard, so that during an incident they quickly decide whether the problem lies in discovery, in a single backend, or in overall capacity.

A few arrangements work again and again, and a few mistakes repeat just as often.

Helpful patterns

Harmful anti-patterns

A checklist in plain language, drawn from everything above.

A rollout approach that reduces risk

Introduce the second tier in a test environment first. Send a copy of real traffic through it, and check that traces arrive whole and that load is spread as expected across tier two. Then move a small slice of production traffic, compare results with your existing setup, and expand gradually. Keep the previous path available for a time so you can return to it if something looks wrong.

Tell the people who use traces what has changed. When sampling decisions move to a different place, the volume and shape of stored traces may change, and engineers will ask questions. A short explanation prevents confusion and builds trust in the new arrangement.

A review checklist before going live

Going through this list with the whole team before the first production deployment catches most of the problems described in this guide. It also gives everyone a shared picture of how the system should behave, which makes later incidents shorter and calmer. Revisit the list after any large change to traffic, to the number of services, or to the platform that hosts the collectors.

Some typical situations show where the exporter earns its place.

graph LR
  A[Steady traffic] --> B[Routing tier handles volume]
  C[Traffic spike] --> B
  B --> D[Stateful tier sized for peak]
  D --> E[Whole traces and stable sampling]
        

Lessons that repeat across scenarios

In every case the successful teams treat the two tiers differently. They let the stateless routing tier scale freely and the stateful tier scale carefully. They use random keys when they can. They watch the number of backends and the failure rates. They accept that scaling events cause small losses and design their routines to keep those events rare.

What the scenarios teach together

Each of these stories came down to one of a few simple causes: a key that did not spread well, a resolver that did not show every backend, or a scaling pattern that disturbed the ring. All were found by watching the exporter’s own signals and comparing backends with their peers. All were easy to prevent with a short checklist and a test environment.

Short, plain answers to the questions people ask most often.

No. It is part of the contributed components, which are distributed separately from the core. You need a collector build that includes it, which is usually done by choosing a contrib distribution or building a custom one.

No. It routes by key using a hash ring. Evenness comes from the keys being well mixed, not from measuring load.

Use the trace identifier when you need whole traces, for example for tail sampling. Use service name only when the job needs all spans of a service together, and accept the risk of hot spots.

The ring is rebuilt, and a small share of keys moves to the new backend. Traces in progress for those keys may be split between the old and new homes.

Most often the resolver returns a single address, for example because a DNS name points to a virtual address instead of listing every pod. Check the backend count the exporter reports.

Yes. Each signal has its own choices of routing key, so check the documentation for your version and choose the key that keeps together what your downstream processing needs.

No. A network load balancer spreads connections, while this exporter reads the data and routes by content. They solve different problems and are often used in different places.

Either a DNS resolver with a headless service or the Kubernetes resolver, which watches endpoints and notices change faster. The Kubernetes resolver needs permission to read endpoint information.

Exports to it fail until the resolver stops listing it. Data held in its memory is lost. Other backends keep working, but the crashed backend’s share of keys is affected until the ring is updated.

It adds the small cost of reading keys and splitting batches. In a properly sized routing tier this is modest, and the benefit of correct stateful processing usually outweighs it.

The exporter sends to other collectors, so a second tier is part of the idea. Without a stateful destination there is nothing for it to route to.

Estimate the incoming data volume and the time that tail-sampling policies wait before deciding. Memory need grows with both. Measure in a test environment and keep headroom for bursts.

With dynamic resolvers, such as DNS or Kubernetes, the list updates as the environment changes. With a static list you must change the configuration and reload or restart.

Some teams route through a message queue partitioned by trace identifier, or through a proxy with consistent hashing. These can work, but the load balancing exporter has the advantage of being built into the collector and aware of telemetry formats.

The load balancing exporter solves a specific and important problem: how to send related telemetry to the same collector when many collectors are available. It reads a routing key from each item, maps the key to a backend with a consistent hash ring, keeps one OTLP exporter per backend, and uses a resolver to keep the list of backends current. This makes it the natural front door for any stateful collector tier, most famously tail sampling.

Its strengths come with limits. It does not react to load, it cannot prevent every split trace when backends come and go, and it depends on the resolver supplying an honest list. Used with care, with a random key, an encrypted hop, a patiently scaled stateful tier and good monitoring, it provides the reliable foundation that scaled-out observability needs.