Before any technical detail, we need to see why these two worlds exist, why they keep meeting, and what the word “interoperate” honestly promises.

Two projects, two histories

Prometheus started at SoundCloud in 2012 as a system for watching the health of services. It joined the Cloud Native Computing Foundation in 2016 and became the default metrics tool of the Kubernetes era. Its big idea was simple. Every service shows a web page full of numbers. The Prometheus server visits that page on a timer, which is called a scrape. It stores what it finds and lets you ask questions using a query language named PromQL.

OpenTelemetry, usually shortened to OTel, arrived later. In 2019 two older projects, OpenTracing and OpenCensus, merged to form it. Its goal was bigger than metrics. It wants one vendor-neutral way to produce traces, metrics and logs from any programming language. It gives you code libraries called APIs and SDKs, a wire format called OTLP (the OpenTelemetry Protocol), and a program called the Collector that receives, changes and forwards telemetry. The metrics part of its specification reached stable status in 2023.

So one project is very strong at storing and querying metrics, and the other is very strong at producing and moving telemetry. Most real companies end up using both.

Why the two worlds must meet

Nobody wakes up wanting to translate between telemetry formats. The need appears because of everyday pressures inside a company. Here are the four that show up again and again.

What “interoperate” really promises

When people say OpenTelemetry works with Prometheus Remote Write, they are bundling three different promises together. It helps to separate them, because each one has a different level of difficulty.

• Delivery. The numbers arrive at the storage system. This is the easy part and it works reliably.
• Meaning. A counter stays a counter, a histogram stays a histogram, and units are not mangled. This needs careful translation rules.
• Queryability. You can write sensible PromQL against the data. This depends on naming, labels and a few habits that you control.

To follow any translation, you first need to know the shape of the thing being translated into. Here is the Prometheus data model in plain words.

Time series, samples and labels

A metric is a number that tells you something about a system, such as how many requests arrived or how much memory is in use. Prometheus does not keep one number. It keeps a long list of numbers over time. Each entry in that list is a sample, which is just a pair of a timestamp and a value. A full list of samples that belong together is called a time series.

Now imagine you run a shop with ten cashiers. You would not want one single “sales” line, because you could not tell the cashiers apart. Prometheus solves this with labels. A label is a small name and value pair, like cashier equals “anita”. A metric name plus a unique set of labels defines exactly one time series. Change any label value and you get a different series.

The four classic metric types

Prometheus describes a metric with a type. The type is mostly a hint for the person writing queries, but it also drives important rules about how numbers should behave over time.

A classic histogram is not stored as one series. It becomes several series: one per bucket limit, plus a running sum of all observed values and a running count of observations. The bucket series carry a special label named le, which means “less than or equal to”. Each bucket is cumulative, so a bucket for 0.5 seconds also includes everything that was faster than 0.1 seconds.

Native histograms

The classic bucket layout has a cost. Every bucket is its own series, so a histogram with many buckets and many label combinations creates a huge number of series. Newer Prometheus versions therefore support native histograms. A native histogram is stored as one series whose every sample carries a compact structure of buckets with automatically sized boundaries. This idea matters a great deal for us later, because it is the natural landing place for OpenTelemetry’s exponential histograms.

Cardinality: the quiet danger

The number of unique label combinations for a metric is called its cardinality. Each combination is a separate series, and each series uses memory and disk. A label such as “user id” or “request id” can create millions of series and bring a server to its knees. Remember this word. It will return in almost every production warning in this guide.

OpenTelemetry describes metrics with a richer vocabulary than Prometheus. Knowing these extra ideas explains almost every translation decision that comes later.

API, SDK and instruments

In your code you use the OpenTelemetry API to say what you want to measure. You create an instrument, which is a named tool for recording numbers. The SDK sits behind the API and does the real work of adding the numbers up and sending them out. This split lets library authors use only the API, while the application owner chooses the SDK and where the data goes.

The main instruments are a counter that only goes up, an up-down counter that can go up and down, a gauge for a current reading, and a histogram for distributions. Several of them also come in an “observable” form, where the SDK calls your code on a timer to read the value instead of you pushing each change.

Aggregation and temporality

The SDK does not send every single measurement. It aggregates them: it adds up counts and fills histogram buckets in memory, then exports the result on a timer. By default this export happens about once every sixty seconds.

Here comes a concept that Prometheus does not have as an option: temporality. When a counter is exported with cumulative temporality, each export says “the total since the start”. When it is exported with delta temporality, each export says “the change since the last export”. Prometheus only understands the cumulative style. This difference creates real work for the Collector, as you will see in Chapter 11.

Resource, scope and attributes

OpenTelemetry keeps three layers of descriptive information, and Prometheus flattens them into one flat set of labels.

• Resource. Describes the thing that produced the data: the service name, its version, the host, the container or the cloud region. It is attached once per batch, not per data point.
• Instrumentation scope. Names the library that created the metric, for example a web framework’s instrumentation package, along with its version.
• Attributes. Describe one single data point, such as the HTTP method or the status code. These are the closest cousins of Prometheus labels.

The instrument to wire-type map

When the SDK exports, each instrument turns into one of five OTLP data types. This table is the foundation for the mapping rules in Chapter 8.

A fifth OTLP type called Summary exists mainly so that data scraped from older Prometheus summaries can be carried through. Normal OpenTelemetry instruments do not produce it.

One catch with default buckets

The default bucket limits of an explicit-bucket histogram were designed for values measured in milliseconds, running from zero up to ten thousand. If your instrument measures seconds, almost every observation will land in the very first bucket, and your percentile estimates will be useless. Always set bucket limits that match the real range of your values, or switch to exponential histograms, which size their own buckets.

The two projects grew up with opposite habits about who starts the conversation. Understanding that difference removes a lot of confusion.

The pull habit

In classic Prometheus, the server decides when to collect. It keeps a list of targets, visits each one every fifteen to sixty seconds, and reads the numbers. If a target is dead, the visit fails, and Prometheus knows something is wrong immediately. That is a lovely side effect: the act of collecting also acts as a health check.

The push habit

In OpenTelemetry, the application decides when to send. The SDK pushes its aggregated results on a timer to a receiving endpoint, over gRPC or HTTP. This suits short-lived jobs, serverless functions and places where the server cannot reach into the network to scrape.

So where does Remote Write sit?

Here is a point many beginners miss. Prometheus Remote Write is itself a push protocol. It was created so that a Prometheus server can copy every sample it collects to a second system, usually one built for long-term storage and huge scale. Over time it became a general-purpose door that many storage systems offer for incoming time series data, even when no Prometheus server is involved at all.

That is what makes the interop possible. Remote Write is a push door that is already open on many backends. OpenTelemetry also pushes. So the Collector can take OTLP data, reshape it into the format that door expects, and push it through.

Remote Write looks mysterious from the outside, yet it is a small and clear contract. Learning what is inside a request explains why the translation rules exist.

What a request looks like

A Remote Write request is an ordinary HTTP POST. The body is a message written in Protocol Buffers, which is a compact binary format made by Google. That body is then compressed with the snappy method, which is fast rather than very small. A few headers tell the receiver what to expect, including the content encoding and a version string for the protocol.

Inside the message there is a list of time series. Each time series carries its own list of labels, and then one or more samples, where each sample is a timestamp in milliseconds and a number stored as a 64-bit float. A series can also carry exemplars, which are example trace links attached to a measurement, and in newer versions it can carry native histogram samples. The metric name itself is simply a label called __name__. Labels must be sorted by name, which lets receivers compare and hash them quickly.

How a sender works inside Prometheus

The most famous sender is the Prometheus server itself. It does not send straight from memory. As samples arrive, Prometheus first writes them into a write-ahead log, which is an on-disk journal used for crash recovery. A component called the WAL watcher reads new records from that journal and hands them to a set of in-memory queues. Each queue is served by a worker called a shard. Each shard collects samples into batches and sends them as separate HTTP requests in parallel.

All samples of one series are always sent by the same shard. This keeps the order of samples for that series correct, which matters because storage engines dislike samples that arrive out of order. Prometheus also watches how far behind the senders are and adds or removes shards automatically. If the remote side is slow, it adds shards. If it is fast, it removes them.

flowchart LR
  A["Scraped or ingested samples"] --> B["TSDB head in memory"]
  B --> C["Write ahead log on disk"]
  C --> D["WAL watcher"]
  D --> E["Queue per shard"]
  E --> F["Shard workers send in parallel"]
  F --> G["HTTPS POST with snappy protobuf body"]
  G --> H["Remote Write receiver"]
  H -->|"2xx OK"| I["Samples accepted"]
  H -->|"5xx or 429"| J["Retry with backoff"]
  H -->|"other 4xx"| K["Batch dropped"]
  J --> F

Read the diagram from left to right. The journal on disk is what makes the sender robust: if the remote system is down for a while, the data is still waiting in the journal. The three response paths at the end are crucial. A success removes the batch from the queue. A temporary failure triggers a retry. A permanent client error means the batch is thrown away, because retrying a request that is wrong will never fix it.

The rules for retrying

The protocol gives clear guidance. Servers should answer with a 2xx code when they accept data. A 5xx code means “my fault, try again”. A 429 code means “you are sending too fast, slow down and try again”. Other 4xx codes mean “your request is bad”, and senders should not retry them. This simple rule matters in production, because one malformed or too-old sample can cause a whole batch to be rejected without any retry.

Remote Write 2.0

The original version of the protocol, often called 1.0, worked well but repeated a lot of text. Every series repeated every label name and value in every request. It also had no good home for metadata such as the help text or unit of a metric.

Version 2.0 of the specification fixes these gaps. It uses a shared symbol table so each unique string appears once per request. It lets each series carry metadata like the metric type, help text and unit. It supports a created timestamp, which records when a counter started counting. It treats native histograms as first-class citizens. And it adds response headers so a receiver can say how many samples, histograms and exemplars it actually wrote, which helps with partial failures. Support for 2.0 is growing in recent Prometheus 3.x releases and in the OpenTelemetry Collector, but support differs by version and by backend, so check the release notes of every component in your path before relying on it.

A working interop setup is a chain of small parts. Each part has one job, and knowing the jobs helps you find problems quickly.

The whole chain at a glance

The path starts with application code that uses the OpenTelemetry SDK. The SDK sends OTLP data to a Collector running close by. That first Collector often forwards to a second layer of Collectors that act as a central gateway. The gateway converts the data and sends it out using Remote Write to a metrics backend. Finally, a dashboard tool such as Grafana asks the backend questions using PromQL.

flowchart LR
  subgraph APPS["Application layer"]
    A1["Service A with OTel SDK"]
    A2["Service B with OTel SDK"]
    A3["Legacy service with metrics page"]
  end
  subgraph AGENT["Agent tier"]
    C1["Agent Collector one per node"]
  end
  LB["Load balancer"]
  subgraph GATEWAY["Gateway tier"]
    G1["Gateway Collector replica 1"]
    G2["Gateway Collector replica 2"]
  end
  subgraph STORE["Storage and query"]
    R["Remote Write endpoint"]
    S["Long term metrics store"]
    Q["Query layer with PromQL"]
  end
  D["Dashboards and alert rules"]
  A1 -->|"OTLP"| C1
  A2 -->|"OTLP"| C1
  A3 -->|"scraped by Prometheus receiver"| C1
  C1 -->|"OTLP"| LB
  LB --> G1
  LB --> G2
  G1 -->|"Remote Write"| R
  G2 -->|"Remote Write"| R
  R --> S
  S --> Q
  Q --> D

In this picture, the agent tier is small and sits next to the applications. It absorbs bursts, adds basic labels and protects the application from slow networks. The gateway tier is where the heavier work happens: filtering, renaming, converting and finally exporting. The load balancer spreads incoming OTLP traffic across gateway replicas so that no single replica becomes a bottleneck or a single point of failure.

The parts inside a Collector

A Collector is built from four kinds of parts that you connect into a pipeline. Data enters through a receiver, passes through processors, and leaves through an exporter. Extensions run beside the pipeline and add helpers such as authentication.

The jobs of each processor you will meet

This is the heart of the guide. We follow one data point through the Remote Write exporter and watch every decision it makes.

The translation step by step

When a batch of OTLP metrics reaches the exporter, it is organized like a tree: resources at the top, then scopes, then metrics, then data points. The exporter walks down the tree and flattens each data point into one or more Prometheus time series. Here is the path it takes, in order.

flowchart TD
  A["OTLP metric arrives at exporter"] --> B{"Which type?"}
  B -->|"Gauge"| C["One gauge series"]
  B -->|"Sum"| D{"Cumulative?"}
  B -->|"Histogram"| E{"Cumulative?"}
  B -->|"Exponential histogram"| F["Native histogram series"]
  B -->|"Summary"| G["Quantile series plus sum and count"]
  D -->|"No, delta"| H["Dropped with warning"]
  D -->|"Yes"| I{"Monotonic?"}
  I -->|"Yes"| J["Counter series"]
  I -->|"No"| K["Gauge series"]
  E -->|"No, delta"| H
  E -->|"Yes"| L["Bucket series plus sum and count"]
  C --> M["Build name and labels"]
  J --> M
  K --> M
  L --> M
  F --> M
  G --> M
  M --> N["Add job, instance and target info"]
  N --> O["Convert time to milliseconds"]
  O --> P["Batch and send"]

The diagram shows an important truth: the first fork depends on the type, and the second fork depends on temporality. Everything after the fork is mostly about names and labels. If you keep this picture in your head, almost every translation surprise becomes easy to explain.

A worked example in words

Imagine a web service named “checkout” that records a histogram called http.server.request.duration with a unit of seconds. One data point has the attribute “http.request.method” equal to “GET”. After translation, you would find a family of series with names like http_server_request_duration_seconds_bucket, along with _sum and _count companions. The attribute becomes a label called http_request_method. The service name becomes the job label. The instance identifier becomes the instance label. Every bucket series also gets the le label.

Each OpenTelemetry metric type has its own rule for how it lands in the Prometheus model. Some rules are perfect matches. Others are careful compromises.

Gauges

This is the easiest case. A gauge is a number that can be read at any moment, and Prometheus gauges mean exactly the same thing. The name and labels are cleaned, and the value is passed along.

Monotonic cumulative sums

An OpenTelemetry counter that only increases becomes a Prometheus counter. By convention the translator adds a _total suffix, so a counter named “orders.processed” ends up as orders_processed_total. Counters work well with the PromQL rate function, because Prometheus can detect when the value drops and treat that as a restart.

Non-monotonic sums

An up-down counter can fall as well as rise, for example the number of items in a queue. Prometheus counters are not allowed to fall, so these are exported as gauges. The value is correct, but you should not use rate on it, and no total suffix is added.

Explicit-bucket histograms

OpenTelemetry stores the count for each bucket separately. Prometheus buckets are cumulative, so the exporter adds the counts up as it goes and finishes with a bucket whose limit is infinity. It then adds the sum and the count. OpenTelemetry histograms may also carry a recorded minimum and maximum. Prometheus classic histograms have no place for them, so they are dropped.

Exponential histograms and native histograms

Exponential histograms size their buckets automatically using a scale number, so they cover a very wide range with small relative error. Prometheus native histograms use a very similar idea, so the conversion is natural. There are limits, though. Prometheus only supports a certain range of scales, so very fine scales are reduced in resolution, and extremely coarse ones may not fit. Both the sender and the receiver need native histogram support switched on, and older Collector versions may drop exponential histograms completely. Test this path with your exact versions before depending on it.

Summaries

Summaries map one to one. Each reported quantile becomes a series with a quantile label, and the sum and count become their own series. OpenTelemetry instruments rarely create summaries, so you mostly meet them when scraping older Prometheus clients through the Collector.

Exemplars

OpenTelemetry measurements can carry exemplars, which are sample values linked to a trace and span. These can travel through Remote Write too, so a dashboard can show a dot on a latency graph that jumps straight to the trace. The receiving backend must be set up to store exemplars, and the exemplar’s trace link must be allowed to pass through any processors you configured.

Most “my dashboard shows nothing” problems are really naming problems. This chapter explains exactly how names and labels are rewritten, so you can predict the result before you deploy.

Why names need rewriting

OpenTelemetry names often contain dots, such as http.server.request.duration. Classic Prometheus names may contain only letters, digits, underscores and colons. So the translator replaces every illegal character with an underscore. Newer Prometheus versions can accept a much wider range of characters, including dots, but that needs matching support in every tool that reads the data, so many teams still choose the underscore style.

Suffixes and units

OpenTelemetry keeps the unit of a metric in a separate field, written in a standard notation. Prometheus has no such field in the original Remote Write format, so by convention the unit is placed at the end of the name. A duration measured in seconds ends up with a _seconds suffix. A size in bytes gets _bytes. A gauge with the plain unit “1”, which usually means a ratio between zero and one, may get a _ratio suffix. Units written in curly braces, such as a count of requests, are only annotations and are dropped. Compound units with a slash, like meters per second, are spelled out using the word “per”.

Counters get a _total suffix. This follows a long Prometheus habit that makes counters easy to recognize at a glance.

Translation strategies

Because needs differ, modern tools let you choose how aggressive the renaming is. The exact option names depend on the tool and its version, but the choices look like this.

Rewriting label names

Attribute names go through a similar cleaning. Illegal characters become underscores, so http.request.method becomes http_request_method. A name that starts with a digit receives a prefix so it remains valid. Names that start with a single underscore may also receive a prefix, because double underscores at the start are reserved for Prometheus internal use.

Sometimes two different attribute names clean up to the same label name. For example, “a.b” and “a_b” both become “a_b”. The translator then joins their values into one label separated by a semicolon, rather than letting one overwrite the other. If you ever see a label value with a semicolon in it, this is usually the reason.

Rewriting label values

OpenTelemetry attributes can be text, whole numbers, decimals, true or false values, and even lists. Prometheus label values are only text, so every attribute is converted to its text form. A numeric status code of 200 becomes the text “200”. Lists are converted to a text representation of the list. Because of this, avoid using list or object attributes on metrics.

OpenTelemetry’s resource idea has no direct twin in Prometheus. The way the two are bridged is clever, and it is also the place where cardinality mistakes are most often made.

Job and instance

Every Prometheus series normally has two labels that say where it came from. The job label names the logical service, and the instance label names the particular running copy. When Prometheus scrapes, it fills these in itself. With push, nobody scrapes, so the translation must fill them in from the resource.

The rule is straightforward. The job label comes from the service name, and if a service namespace exists, the job becomes the namespace, a slash, and the service name. The instance label comes from the service instance identifier. If a resource has no service name, a placeholder name for an unknown service is used, which is a clear sign that someone forgot to configure the SDK.

The target info series

A resource may hold thirty or more attributes: cloud provider, region, operating system, container image, version and more. Copying all of them onto every series would multiply storage cost. So the exporter writes them only once, into a special series named target_info. This series always has the value one, and its labels are the resource attributes. It also carries the job and instance labels, so it can be linked to other series.

When you want to filter or group a normal metric by something from the resource, such as the software version, you use a PromQL join. In words: take the request-rate series, match it to target_info using the shared job and instance labels, and copy the version label across. It is one extra step in a query, in exchange for a huge saving in storage.

Promoting selected attributes

Joins are powerful but slightly harder to write. A middle path is to promote a small number of important resource attributes into real labels on every series. The cluster name, the namespace and the environment are classic candidates, because you will filter by them in almost every dashboard.

The Remote Write exporter has an all-or-nothing switch that copies every resource attribute onto every series. That switch is convenient for small tests, but it can multiply cardinality in production. A safer pattern is to use a processor that copies only the few attributes you choose. The native OTLP receiver in recent Prometheus versions has a similar setting where you list exactly which resource attributes to promote.

What happens to scope information

The instrumentation scope, meaning the library name and version that produced a metric, is the piece most likely to be left out. Depending on the tool and its settings, it may become extra labels, it may become part of a separate info series, or it may be dropped. If you care about knowing which library produced a metric, check how your chosen path handles it and test with a real sample.

Numbers are only half of a metric. The other half is when they were measured and how a system knows a series has started, restarted or ended. Here the two worlds differ most.

Delta metrics need a converter

Some OpenTelemetry setups prefer delta temporality, where each export reports only the change since the last export. Teams like it because the SDK can forget old values after each export, which saves memory when there are many label combinations. Serverless functions also like it for the same reason.

Prometheus, however, expects running totals. To bridge this, the Collector offers a processor that turns deltas into cumulative values by keeping a running sum for every series in memory. This is powerful but it makes the processor stateful. That has three consequences you must plan for.

• Every data point of one series must reach the same Collector replica, otherwise each replica builds a different running total. A load-balancing exporter that routes by series identity solves this.
• If the replica restarts, the running total starts again from zero. Prometheus will read that as a counter reset, which is handled gracefully by rate calculations.
• Series that stop reporting must eventually be forgotten, so the processor has settings for how long to remember a silent series and how many series to track at most.

Counter resets and start time

OpenTelemetry attaches a start time to every cumulative point. It says exactly when counting began, so a new start time clearly signals a restart. Prometheus does not have this field in the original protocol. It detects a reset only when a value goes down compared with the previous sample.

This leads to a rare but real blind spot. If a process restarts and its counter climbs past its old value before the next sample is taken, Prometheus never sees a drop and underestimates the growth. The created timestamp in Remote Write 2.0 is designed to close this gap. Some exporters can also write an extra series that carries the start time, which helps storage engines that understand it.

Staleness: how a series ends

When Prometheus scrapes a target and a series disappears from the page, it writes a special “stale” marker. Queries then stop showing that series right away. With push, nobody notices that a series went silent. So instead, queries keep showing the last value until a lookback window passes, which is five minutes by default.

OpenTelemetry does have a way to say “no value recorded” on a data point. A well-behaved exporter translates that flag into the Prometheus stale marker, so a series can end cleanly. However, most SDKs simply stop sending, and so most series fade out slowly after the lookback window.

Choosing time windows that work

The rate function in PromQL needs at least two samples inside its window. If your SDKs export every sixty seconds, a window of one minute may contain only one sample and return nothing. A good rule of thumb is to make the window at least four times the export interval. For sixty-second exports, use windows of five minutes or more.

Let us now follow a single measurement on its whole journey, from the moment a line of code runs to the moment a person sees it on a graph.

sequenceDiagram
  participant App as Application code
  participant SDK as OTel SDK
  participant Col as Collector
  participant Que as Export queue
  participant Rec as Remote Write receiver
  participant DB as Time series store
  participant Gra as Dashboard
  App->>SDK: Record a measurement
  Note over SDK: Add to in memory aggregate
  SDK->>Col: Export aggregate over OTLP on timer
  Col->>Col: Memory check and batch
  Col->>Col: Convert to Prometheus time series
  Col->>Que: Place request in queue
  Que->>Rec: POST snappy protobuf
  Rec->>DB: Validate and write samples
  DB-->>Rec: Accepted
  Rec-->>Que: 2xx response
  Gra->>DB: PromQL query
  DB-->>Gra: Series and samples

The journey in seven stops

These three numbers tell you where the time goes. The Collector adds a fraction of a second. The network and storage add a few seconds at worst. The big delay is the SDK export interval, so a measurement may take up to a minute or two to show up on a graph. If an alert needs faster reaction, shorten the SDK interval, but remember that more frequent samples also mean more storage.

The Collector’s Remote Write exporter is the best-known path, but it is not the only one. The right choice depends on where your data starts and what your storage understands.

flowchart LR
  subgraph P1["Pattern A: Remote Write exporter"]
    A1["OTel SDK"] -->|"OTLP"| A2["Collector"]
    A2 -->|"Remote Write"| A3["Prometheus compatible store"]
  end
  subgraph P2["Pattern B: Native OTLP ingestion"]
    B1["OTel SDK"] -->|"OTLP"| B2["Collector or direct"]
    B2 -->|"OTLP"| B3["Store with OTLP endpoint"]
  end
  subgraph P3["Pattern C: Pull from SDK"]
    C1["OTel SDK with Prometheus exporter"] -->|"scraped"| C2["Prometheus server"]
    C2 -->|"Remote Write"| C3["Long term store"]
  end
  subgraph P4["Pattern D: Prometheus into OTel"]
    D1["Prometheus server"] -->|"Remote Write"| D2["Collector Remote Write receiver"]
    D2 -->|"OTLP"| D3["OTLP backend"]
  end

Pattern A: Collector with the Remote Write exporter

This is the pattern the rest of this guide describes. It works with a very wide range of storage products because Remote Write is almost a universal door. You also get full control over processing inside the Collector.

Pattern B: Native OTLP ingestion

A growing number of storage systems accept OTLP directly. Recent Prometheus versions can be started with a flag that opens an OTLP receiving endpoint. Grafana Mimir, and several other systems built on the Prometheus model, expose OTLP endpoints too. In this pattern the translation to the Prometheus model happens inside the storage system rather than in your Collector, using the same rules you have just learned. The benefit is one fewer moving part to run. The drawbacks are less flexibility and a dependence on the storage system’s version for features such as delta handling or exponential histograms.

Pattern C: Let Prometheus scrape the SDK

Many OpenTelemetry SDKs include a Prometheus exporter that exposes a classic metrics page inside the application. A normal Prometheus server scrapes it, and that server can then use Remote Write to copy data to long-term storage. This keeps the pull habit and its free health check, and it is a gentle migration path for teams with a lot of existing Prometheus setup. It does not carry every OpenTelemetry concept, and the maturity of this exporter varies between languages.

Pattern D: From Prometheus into OpenTelemetry

The reverse direction is also useful. The Collector’s Prometheus receiver can scrape classic targets using standard scrape configuration, so a Collector can replace a Prometheus server as the thing that scrapes. In Kubernetes, the OpenTelemetry Operator includes a component called the Target Allocator, which discovers scrape targets from the same service monitor definitions many teams already use, and shares them fairly among several Collector replicas. There is also a Remote Write receiver for the Collector, which lets an existing Prometheus server send its Remote Write stream straight to a Collector that then forwards it as OTLP. At the time of writing, that receiver is newer and less settled than the exporter, so confirm its maturity for your version.

A Remote Write door is open on many products. The door looks the same on all of them, but the rooms behind it differ. Knowing those differences saves you from painful surprises.

The common backends

Questions to ask before you choose

Because Remote Write is a loose standard, two backends can both say “we accept Remote Write” and still behave differently. Before connecting your fleet, run through the following checklist with each backend.

Metrics look harmless, yet they can reveal how your systems are built, and an open endpoint can be abused to flood your storage. Security should be designed in from the start.

Encrypt everything in transit

Every hop in the chain should use TLS: from the application to the Collector, between Collector tiers, and from the Collector to the storage endpoint. For machine-to-machine links inside your own network, many teams go one step further and use mutual TLS, where both sides present certificates, so each side knows exactly who it is talking to.

Prove who is sending

Remote Write backends do not accept anonymous strangers in production. The Collector supports several ways to prove identity through extensions.

Protect the front doors

The OTLP receiver listens on well-known ports, commonly 4317 for gRPC and 4318 for HTTP. If these are reachable from the open internet, anyone can send you junk metrics or try to overwhelm you. Bind receivers to private addresses, place them behind internal load balancers, require authentication when traffic crosses trust boundaries, and use network rules to allow only known senders. Recent Collector versions are moving toward listening on the local machine by default, which is a safer starting point.

Keep secrets out of files

Do not write tokens and passwords into configuration files that live in version control. Feed them through environment variables or a secret store, and rotate them on a schedule. For cloud-signed requests, prefer role-based identity attached to the workload so there are no stored keys at all.

Watch what labels reveal

Labels are data. If a developer adds an attribute holding an email address, a customer name or an internal host name, it will be copied into every series, stored for a long time and shown on shared dashboards. Use processors at the gateway to delete or hash sensitive attributes before export, and use review rules so new attributes are checked before they ship.

A bridge that works for ten services must also work for ten thousand. Here is how to size it, scale it and keep it alive when parts fail.

Size the load first

The load on Remote Write is easy to estimate. Count your active series and multiply by how often each series gets a new sample. One million active series sampled once a minute produce about sixteen thousand seven hundred samples per second. That number, not the number of services, decides how many Collector replicas and how many backend ingesters you need.

Batching and request size

Many small requests waste time on connection and header overhead. Larger batches are far more efficient. The batch processor groups data by count and by time, and the exporter splits very large payloads so each request stays under a maximum size, which is around a few megabytes by default. Always check that this limit is below the request size limit of your backend.

Queues, workers and backpressure

The exporter holds data in an in-memory queue and sends it with several parallel workers. The queue absorbs short slowdowns. If the backend stays slow longer than the queue can hold, the queue fills up and new data is rejected, which means metrics are lost. The names of these settings differ between Collector versions, so read the documentation for your version, but the ideas are always the same: how many items can wait, how many workers send, and how often to retry.

A memory limiter placed first in the pipeline is your safety valve. When memory use climbs too high it refuses new data early, which pushes pressure back to the senders instead of letting the Collector crash and lose everything at once. For stronger durability, the Collector can keep its queue on disk so that waiting data survives a restart, at the cost of some speed.

Scaling the Collector tier

Stateless pipelines scale by adding replicas behind a load balancer. Any replica can handle any request. Stateful pipelines, such as the delta-to-cumulative converter, need every point of a series to land on the same replica. In that case place a load-balancing exporter in an earlier tier that routes by a stable identity, such as the series, so each series always lands on the same replica in the next tier.

High availability and the single-writer rule

The golden rule of Remote Write is that each series should have exactly one writer at any time. Two senders writing different values at the same timestamp to one series cause conflicts. For pushed OpenTelemetry data, high availability usually comes from a load balancer that spreads and fails over traffic between healthy gateway replicas, so there is still only one writer per series at a time.

When you scrape targets with the Collector, high availability is often built by running two identical scrapers. In that case both replicas send the same series. Systems such as Mimir and Cortex offer a deduplication feature that elects one replica as the leader for each cluster, using two special labels, one naming the cluster and one naming the replica, and drops the other copy. Thanos solves a similar need by deduplicating at query time using a replica label.

flowchart LR
  T["Scrape targets"] --> R1["Collector replica A"]
  T --> R2["Collector replica B"]
  R1 -->|"Remote Write with replica label A"| D["Distributor with HA tracker"]
  R2 -->|"Remote Write with replica label B"| D
  D -->|"Leader copy kept"| I["Ingesters"]
  D -.->|"Duplicate dropped"| X["Discarded"]
  I --> S["Object storage and queriers"]

In Figure 6, both Collectors scrape the same targets and send identical data. The receiver looks at the replica label, picks one leader and quietly drops the copy. If the leader dies, the tracker waits a short time and promotes the other replica, so there is only a brief gap rather than a long outage.

Old and out-of-order data

Scrape-based setups send samples in neat time order. Push setups may deliver samples late, for example after a network pause or when a queue drains after an outage. By default, storage engines reject samples that arrive older than the newest sample of that series. Most modern backends have a setting for an out-of-order time window that accepts slightly late data. Turn it on if your pipeline can deliver late data, and size it to your realistic delays.

A pipeline that carries your monitoring data needs monitoring of its own. This chapter shows what to watch and how to recover when things break.

Watch the bridge itself

The Collector reports its own health as metrics. A few of them tell you almost everything. Watch the number of metric points successfully sent and the number that failed. Watch the queue size compared with the queue capacity, because a queue that is always nearly full means the backend cannot keep up. Watch the Collector’s memory use and the count of data points refused by the memory limiter. In recent versions many of these names carry a total suffix, so check the exact names in your version.

The receiving side also tells its own story. Storage systems count the samples they reject and give the reason: too old, out of order, duplicate, too many labels, label too long, or series limit reached. Send these counters to a dashboard and alert on any sustained rise. Many silent data gaps come from rejections nobody was watching.

Symptoms, causes and fixes

Failure scenarios and recovery

After you know how the pieces work, the remaining skill is arranging them well. These patterns and warnings come from the mistakes that teams repeat most often.

Patterns that work

Anti-patterns to avoid

Best practices in one place

• Standardize resource settings. Make sure every service sets a service name and a unique instance identifier.
• Agree on one naming strategy before building dashboards, and document it.
• Prefer cumulative temporality for Prometheus-compatible destinations unless you have a strong reason to use delta.
• Put the memory limiter first and the batch processor near the end of the pipeline.
• Limit cardinality at the gateway with filters, attribute deletion and backend limits.
• Test version upgrades of the Collector and the backend on real samples, because translation behavior and defaults do evolve.
• Monitor the monitoring. Alert on failed exports, a full queue, and rising rejection counters on the backend.
• Keep one writer per series and keep clocks synchronized.

Common mistakes beginners make

Once the basics are running, three bigger questions decide how smooth your next two years will be: how you store distributions, how rich your metadata should be, and which integration path you commit to.

Why distributions deserve special care

Counters and gauges are cheap. A histogram is the opposite, because it is really a small table of numbers sent again and again. With classic buckets, each bucket is a separate series. If a service has fifteen buckets and a hundred label combinations, one histogram creates about sixteen hundred series, counting the sum and the count. Multiply that by hundreds of services and the bill becomes serious. This is the pressure that drives teams toward native histograms.

A native histogram keeps all buckets inside one sample. The bucket boundaries follow a mathematical pattern controlled by a single number called the schema, so the boundaries do not need to be written out each time. Because buckets are created only when observations land in them, empty ranges cost nothing. The result is a big drop in series count and, usually, better accuracy for percentile estimates.

Where exponential histograms meet native histograms

OpenTelemetry exponential histograms and Prometheus native histograms were designed with the same mathematical idea, so they line up closely. Each bucket is a fixed percentage wider than the one before it, so a single set of buckets can describe values from microseconds to hours with the same relative accuracy. When the schemas differ, the exporter reduces the resolution to the nearest supported level, merging neighbouring buckets. This loses a little detail but never invents data.

To use this path end to end, four things must all be true. The SDK must be set to produce exponential histograms. The Collector version must convert them rather than drop them. The backend must be started with native histogram support. And your dashboard tool must know how to query them. If any link is missing, the data either disappears or falls back to something less useful, so test the whole chain with a small service first.

What Remote Write 2.0 changes in practice

Think of version 2.0 as a better envelope. It does not change how you design metrics, but it removes several old compromises. Strings are sent once and referred to by number, so requests shrink noticeably when many series share long label names. Each series can say what type it is, what unit it uses and what it means in plain words, so you may no longer need to rely on suffixes alone. A created timestamp lets counters declare when they began, which closes the restart blind spot from Chapter 11. And response headers tell the sender exactly how many items were stored, so partial failures can be understood instead of guessed.

The cost is coordination. Sender and receiver must both speak version 2.0, and some products in between, such as gateways or proxies that inspect requests, may not understand it yet. Many teams therefore run version 1.0 for now, watch the release notes of their Collector and storage vendor, and switch when both ends are ready. Because the receiver tells the sender what it supports in many implementations, a careful rollout can often be done one destination at a time.

Choosing a path with confidence

Because there are several ways to connect the two worlds, it helps to decide using a few plain questions rather than habit. Ask yourself where your data starts, what your storage understands and who will operate the pieces.

Cost thinking for architects

Every design choice in this guide has a price tag, even if it is hidden. More series means more memory in ingesters and more disk in object storage. More frequent samples means more network traffic and more compute for queries. Longer retention multiplies storage. Extra labels multiply series. For this reason, mature teams track cost per thousand active series and review it each quarter, just as they review server costs. They also use recording rules, which pre-calculate expensive queries, so dashboards stay quick even when raw data is large.

A useful habit is to estimate the cost of a new metric before adding it. Multiply the number of label combinations by the number of services and by the sample interval, and compare the result with your budget. A metric that is cheap in one service can be expensive when copied into a thousand. Treat this estimate as part of normal design review, in the same way you would review a new database table.

The ideas in this guide were not invented in a lab. They grew out of real scaling pain at real companies, and they show up again in the typical situations below.

How the ecosystem grew around Remote Write

Everyday situations where the bridge appears

A sensible rollout plan

Quick answers to the questions that come up most when teams first connect these two worlds.

You started with two systems that speak different languages. You now know how a small, careful translator lets them share one picture of your software.