Before we meet any new protocol, we need to feel the cost that it tries to reduce.

Telemetry is a lot of data

Telemetry is the information that software produces about itself: traces that follow requests, metrics that count and measure, and logs that record events. A modern system with hundreds of services can produce millions of these records every minute. Each record carries many details: names, labels, timestamps, identifiers, and values. When you add them all up, the volume is enormous.

All of that data must travel. It leaves applications, passes through collectors and gateways, crosses networks, and reaches a backend where it is stored and queried. Every step of that journey costs something.

Where the cost shows up

Why simply compressing is not the whole answer

The standard way to shrink data on the network is compression, using general-purpose tools that look for repeated patterns in a stream of bytes. Compression helps a great deal and is already widely used. But a general-purpose compressor sees only a stream of bytes. It does not know that one stretch of bytes is a service name and another is a timestamp. If the data is arranged in a way that scatters similar values far apart, even a good compressor misses many chances to shrink it. Arranging the data so that similar values sit side by side gives the compressor a much better starting point, and that is the opportunity columnar encoding creates.

A short, friendly refresher on the pieces of OpenTelemetry that OTel Arrow builds on.

OpenTelemetry in one paragraph

OpenTelemetry, often shortened to OTel, is an open standard and a set of tools for producing and moving telemetry. Applications use SDKs to create traces, metrics, and logs. A separate program called the Collector receives that telemetry, processes it, and forwards it to backends. The standard format used between these pieces is called the OpenTelemetry Protocol, or OTLP.

The building blocks

How OTLP lays out data: a nested tree

OTLP organises telemetry as a tree. At the top is a resource, which describes where the data came from, such as a particular service running on a particular machine. Under each resource are one or more scopes, which describe the library that produced the data. Under each scope are the actual records: spans for traces, data points for metrics, and log records for logs. Each record carries its own list of attributes, which are named labels such as “http method” or “customer region”.

flowchart TD
  R["Resource: service and host details"] --> S1["Scope: instrumentation library A"]
  R --> S2["Scope: instrumentation library B"]
  S1 --> A1["Span 1 with its own attributes"]
  S1 --> A2["Span 2 with its own attributes"]
  S2 --> A3["Span 3 with its own attributes"]
        

This tree is natural and easy to understand, and it already saves some repetition, because details about the resource and the scope are written once instead of on every record. But inside each record, the attribute names and many values still repeat again and again.

The whole story of OTel Arrow rests on one simple choice: how to arrange a table of information.

Row-oriented layout

Picture a spreadsheet of spans. Each row is one span, and the columns are fields such as service name, operation name, start time, duration, and status. A row-oriented layout writes the data row by row: all the fields of span one, then all the fields of span two, then span three, and so on. This is how most people naturally think about records, and it is how OTLP encodes data inside each batch.

Column-oriented layout

A column-oriented layout writes the same table column by column: first every service name from every span, then every operation name, then every start time, then every duration. Nothing about the information changes. You can rebuild any row by reading the same position in each column. But the bytes sit in a very different order.

flowchart LR
  subgraph ROW["Row layout"]
    R1["span 1: checkout, pay, 10:00, 120ms"]
    R2["span 2: checkout, pay, 10:01, 95ms"]
    R3["span 3: checkout, cart, 10:01, 40ms"]
  end
  subgraph COL["Column layout"]
    C1["service: checkout, checkout, checkout"]
    C2["operation: pay, pay, cart"]
    C3["start: 10:00, 10:01, 10:01"]
    C4["duration: 120, 95, 40"]
  end
        

Why columns compress so well

Look at the column layout above. The service column contains “checkout” three times in a row. The operation column contains “pay” twice. The start times are close together. Values in the same column tend to be similar to each other, because they all describe the same kind of thing. When similar values sit side by side, simple tricks can shrink them enormously.

Columns also help computers read faster

Columns are friendly to computer hardware as well. Processors work best when they handle long runs of the same kind of value, because they can process several values in one step. Many analytical databases store data by column for exactly this reason. OTel Arrow borrows that long-proven idea and applies it to telemetry in motion.

The “Arrow” in OTel Arrow is a well-known open standard for columnar data. Knowing what it is makes the rest easy.

What Apache Arrow is

Apache Arrow is an open standard for representing tables of data in columnar form. It defines exactly how each column is laid out in memory, how missing values are marked, how text and nested values are stored, and how groups of columns are packaged together. It is used widely in data analysis tools, databases, and data engineering pipelines, and it has libraries in many programming languages.

Key ideas in Arrow

Why Arrow suits streaming telemetry

Three Arrow features line up well with telemetry. The streaming format lets a sender transmit the description of the columns once and then keep sending batches that follow it. Dictionary arrays let repeated names and values be sent once and reused. And the whole design favours long runs of similar values, which is exactly what telemetry from a busy service looks like.

Now we can state the idea precisely and place it in the OpenTelemetry world.

A working definition

OTel Arrow is the name of an OpenTelemetry project that defines and implements a way of sending the same traces, metrics, and logs as OTLP, but encoded with Apache Arrow in columnar form and sent over long-lived streaming connections. The protocol is often called OTAP, the OpenTelemetry Protocol with Apache Arrow. It is designed as a companion to OTLP, not a replacement for it.

What stays the same and what changes

A bridge, not a rewrite

The most important design choice is that OTel Arrow is meant to live alongside OTLP. Applications keep using the normal SDKs and OTLP. The savings are collected at the hops between Collectors, especially long or expensive ones. A sender converts OTLP data into Arrow form, ships it compactly, and a receiver at the other end converts it back to OTLP. Every other part of your system sees ordinary OTLP and never needs to know that Arrow was used in the middle.

flowchart LR
  APP["Applications with standard SDKs"] -->|"OTLP"| E["Edge Collector"]
  E -->|"OTel Arrow, compact"| G["Gateway Collector"]
  G -->|"OTLP"| BE["Backend"]
        

Where the project came from

The work began as a collaboration between engineers at observability companies who were looking for ways to cut the cost of moving very large volumes of telemetry. They published the design and the first implementations openly, and the work was contributed to the OpenTelemetry community, where it is developed in its own repository alongside the main project. The first implementation focuses on Collector components written in Go. Later work explores a more native Arrow-based processing engine, so the project continues to evolve.

The tree of OTLP becomes a small family of related tables. Understanding this step explains most of the savings.

From a tree to related tables

OTLP stores telemetry as a nested tree. Columnar formats like Arrow work with flat tables, so OTel Arrow first reorganises the tree into several tables that point to each other. This is the same idea that relational databases use: instead of copying details into every row, you keep each kind of detail in its own table and connect them with short identifiers.

For traces, the main table holds one row per span, with columns such as trace identifier, span identifier, name, start time, duration, and status. Separate tables hold the pieces that vary in size: the attributes of the resource, the attributes of the scope, the attributes of each span, the events that happened inside spans, the links between spans, and the attributes of those events and links. Each of these side tables refers back to its parent row with a small number.

flowchart TD
  SP["Spans table: one row per span"]
  RA["Resource attributes table"]
  SA["Scope attributes table"]
  PA["Span attributes table"]
  EV["Span events table"]
  EA["Event attributes table"]
  LK["Span links table"]
  LA["Link attributes table"]
  SP --- RA
  SP --- SA
  SP --- PA
  SP --- EV
  EV --- EA
  SP --- LK
  LK --- LA
        

Why side tables help

Take the attributes of a span, such as the HTTP method, the route, and the status code. In OTLP, each span carries its own list of named attributes, so the name “http.method” is written again for every span. In the columnar form, all attributes go into one attributes table with three kinds of columns: the owner (which span), the key (the attribute name), and the value. The key column then contains the same few names repeated, which dictionary encoding turns into tiny numbers. The value column can be stored by type, with separate columns for text, whole numbers, decimals, and true or false values, so that each column holds values of one kind.

Sorting to create runs

The sender can also reorder rows within a batch before encoding, because telemetry records within a batch do not depend on their order. Sorting by fields such as resource, scope, or name puts identical values next to each other, lengthening runs of repeats. Longer runs mean better compression. This is a classic trick from the world of analytical databases, applied here to telemetry in transit.

The same idea for metrics and logs

Metrics follow the same pattern with their own tables. There is a table of metric descriptions, and tables for the different kinds of data points, such as plain numbers, histograms, and summaries, each with its own attribute table and, where relevant, a table of exemplars, which are sample traces linked to a metric value. Logs use a main table of log records and an attribute table. The principle is identical: keep each kind of detail in a tidy column, and refer to shared pieces by number.

OTel Arrow does not just change the layout. It also changes how batches travel, using long-lived streams that remember what has already been sent.

From many requests to one stream

With classic OTLP over gRPC, each batch is a separate request and response. Every request is independent, so nothing is remembered between them. OTel Arrow instead opens a bidirectional stream: a single long-lived connection over which the sender keeps pushing batches, and the receiver keeps sending back acknowledgements. Because the stream stays open, both sides can keep shared state.

Schemas and dictionaries are sent once

The first message on a stream carries the description of the columns, the schema, and the initial contents of the dictionaries, the lists of distinct values such as service names and attribute keys. Later messages carry only what is new: the actual rows, plus any dictionary entries that were not seen before. A value that has already been sent is referred to by its number. This is the delta idea, and it is where much of the long-run saving comes from.

sequenceDiagram
  participant S as Sender
  participant R as Receiver
  S->>R: Open stream
  S->>R: Batch 1 with schema and full dictionaries
  R-->>S: Acknowledge batch 1
  S->>R: Batch 2 with rows and only new dictionary entries
  R-->>S: Acknowledge batch 2
  S->>R: Batch 3 with rows and few new entries
  R-->>S: Acknowledge batch 3
  Note over S,R: Each later batch is smaller because shared values are already known
        

Acknowledgements keep delivery honest

For every batch the sender transmits, the receiver replies with a status that says whether the batch was accepted, whether it failed temporarily so that it can be retried, or whether it failed permanently. This gives the sender the same kind of reliability signal it gets from separate request and response calls in classic OTLP, while still using one long connection. The sender also learns when the receiver is overloaded and can slow down.

State has a price

Because both ends remember dictionaries and schemas, each open stream occupies memory on the receiver. If the stream breaks, that remembered state is lost, and the sender must open a new stream and start again with a full schema and dictionaries. Well-designed senders also close streams on purpose after a chosen lifetime and open fresh ones, which keeps the remembered state from growing without limit and helps spread load across receivers.

Graceful fallback to OTLP

A sender that tries to use OTel Arrow with a receiver that does not understand it must not simply fail. The design allows a sender to detect that Arrow is not supported and fall back to standard OTLP, so that data keeps flowing, only at the normal size. This makes it safe to roll out gradually, and it protects you from outages caused by mismatched versions.

Numbers make the idea concrete. The figures below are illustrative, chosen to show the mechanism rather than to promise a particular result.

The batch

Suppose a batch holds 1,000 spans from one service. The spans share the same resource and scope. There are only 5 different service operations in the batch. Each span has 8 attributes, and the names of those attributes, such as the method, the route, the region, and the status code, are the same set of 8 names for every span. The start times are all within a one-second window, and the durations are small whole numbers.

Row by row

In a row-oriented layout, the operation name is written 1,000 times, once per span. The 8 attribute names are written 8,000 times in total. Every start time is written as a full-size number. Even with a good general compressor, which will notice many repeats, the compressor must discover these patterns while scanning a mix of different kinds of bytes.

Column by column

The effect grows with scale and repetition

Savings rise with the size of the batch, because the one-off cost of the dictionary is spread over more rows. They also rise with repetition: the more the same names and values appear, the more the dictionaries pay off. And they continue to improve over the life of a stream, because later batches can reuse dictionary entries that were sent earlier.

When the saving is small

In practice, you meet OTel Arrow as a pair of Collector components: one that sends and one that receives.

The two components

Both components handle traces, metrics, and logs. They are distributed in the community contributions collection, so they are present in the contrib distribution and can be added to a custom build. Their stability level has been evolving, so check the component documentation for your version.

Where they sit in a pipeline

flowchart LR
  subgraph EDGE["Edge Collector"]
    R1["OTLP receiver"] --> P1["Processors and batching"] --> X1["OTel Arrow exporter"]
  end
  subgraph CENTRAL["Gateway Collector"]
    R2["OTel Arrow receiver"] --> P2["Processors"] --> X2["OTLP exporter"]
  end
  X1 -->|"Arrow streams"| R2
  X2 --> BE["Backend"]
        

The settings that matter, in plain words

Working with the rest of the Collector

The batch processor, placed before the Arrow exporter, decides how large the batches are. Bigger batches give the encoder more repetition to exploit, so tuning batch size is one of the most effective ways to improve the saving. The memory limiter protects the Collector from running out of memory, and works alongside the Arrow receiver’s own limits. If you use a persistent queue on the exporter, the queued items are stored in the Collector’s ordinary form and converted to Arrow as they are sent.

The savings appear where data crosses long or expensive links. Choosing the right place for the Arrow hop matters more than anything else.

Pattern A: Edge to gateway over a limited link

Collectors at remote sites, branch offices, or factories send to a central gateway across a slow or costly connection. An Arrow hop on that link reduces the bytes crossing it, leaving more capacity for other traffic and making the telemetry arrive sooner during busy periods.

Pattern B: Across regions or clouds

Cloud providers often charge for data that moves between regions or leaves the cloud. A Collector in each region can compress telemetry with Arrow before sending it to a central region, cutting the charge for the cross-region hop.

flowchart LR
  subgraph R1["Region 1"]
    A1["Services"] --> C1["Regional Collector"]
  end
  subgraph R2["Region 2"]
    A2["Services"] --> C2["Regional Collector"]
  end
  C1 -->|"OTel Arrow across regions"| CEN["Central gateway"]
  C2 -->|"OTel Arrow across regions"| CEN
  CEN -->|"OTLP"| BE["Backend"]
        

Pattern C: The last mile to a backend

If your backend or vendor offers an endpoint that speaks OTel Arrow natively, the final hop from your gateway can also use it, saving egress to the outside world. This depends on the backend’s support, which is still uncommon, so check with your vendor before planning around it.

Pattern D: Tiered fan-in

Large estates often have several layers of Collectors, each aggregating many senders. Where the layers are far apart or numerous, Arrow can be used on each long hop, while the short hops near applications remain plain OTLP.

Where not to use it

The hop between an application’s SDK and a Collector on the same machine is short and cheap, and the batches are usually small. Arrow adds encoding effort without saving anything meaningful there. Keep that hop on ordinary OTLP.

flowchart TD
  Q1{"Is the hop long, costly, or bandwidth limited?"} -->|"No"| N["Keep plain OTLP"]
  Q1 -->|"Yes"| Q2{"Are batches large and repetitive?"}
  Q2 -->|"No"| N
  Q2 -->|"Yes"| Q3{"Can both ends run Arrow components?"}
  Q3 -->|"No"| N
  Q3 -->|"Yes"| Y["Use OTel Arrow on that hop"]
        

OTel Arrow trades processor effort for network savings. A little arithmetic tells you whether the trade is good for you.

The trade in one sentence

Converting data to columns, building dictionaries, and encoding them takes more processor time than simply writing out OTLP, and the receiver must do matching work to rebuild the data. In exchange, fewer bytes cross the network. Whether that is a good deal depends on how expensive your bytes are compared with your processor time.

A worked example of the bandwidth saving

Suppose a regional Collector sends 40 megabytes per second of OTLP traffic, after general compression, across a link where data is charged by volume. Over a day, that is about 3.4 terabytes. Imagine that, for this particular data, OTel Arrow reduces the compressed size by about a third. The same telemetry would then cross the link at roughly 27 megabytes per second, or about 2.3 terabytes per day, saving more than one terabyte every day.

Multiply by thirty days and by several regions, and the saving becomes a significant line on the monthly network bill. Now compare it with the cost of the extra processor time: perhaps a few additional processor cores across the senders and receivers. If a terabyte of cross-region traffic costs more than those cores, the trade is favourable. If your link is free and fast, the trade may not be worth the added complexity.

The knobs that affect the result

Memory for stream state

Each open stream keeps dictionaries and schemas in memory on both the sender and the receiver. With a few streams per sender and thousands of senders, the receiver’s memory can become a real consideration. The receiver’s memory limit setting guards against exhaustion, but you should still estimate the total in advance: number of streams multiplied by the typical state per stream, plus the memory needed for batches in flight.

Latency

Encoding adds a small delay to each batch, and larger batches add the time needed to fill them. For telemetry this delay is usually unimportant, since people look at dashboards and traces after the fact, but if you rely on near-instant alerts from the data, measure the end-to-end delay before and after the change.

Long-lived streams are the source of most of the savings, and also of most of the operational surprises.

The long connection problem

Traditional load balancers spread work by sending each new request to a different server. With plain OTLP over gRPC, each batch can be a separate request, so the work is spread evenly. With OTel Arrow, a stream is a single long connection. Once it is attached to a receiver, every batch on it goes to that same receiver for as long as the stream lives. If a few streams carry most of the traffic, one receiver can end up with a heavy load while others sit idle.

flowchart LR
  subgraph BAD["Without rotation"]
    S1["Sender A"] -->|"stream stays forever"| RA["Receiver 1: overloaded"]
    S2["Sender B"] -->|"stream stays forever"| RA
    S3["Sender C"] -->|"stream stays forever"| RB["Receiver 2: nearly idle"]
  end
        

Stream lifetime to the rescue

The cure is for senders to close each stream after a chosen lifetime and open a new one. Each new stream is a fresh chance for the load balancer to choose a receiver, so over time the load spreads out. The lifetime is a balancing act: too long and load stays uneven, too short and the senders keep paying the cost of resending schemas and dictionaries, which weakens the compression.

Use a load balancer that understands gRPC

A basic connection-level balancer sees only a single long connection per sender and cannot spread the batches inside it. A balancer that understands gRPC can sometimes balance at the level of streams or requests. Whichever you use, test the distribution of load across receivers under realistic traffic, and watch for a slow drift toward imbalance after rolling updates.

Reliability when a stream breaks

Backpressure and queues

The acknowledgement messages carry the receiver’s view of its own load. When the receiver is busy, it can respond with a temporary failure, and the sender slows down and retries. A sending queue in the exporter absorbs short bursts. For longer outages, a persistent queue on the sender holds data on disk until the receiver returns. The behaviour is similar to what you would plan for plain OTLP, with the extra detail that a restarted stream briefly costs more bytes.

What about tail sampling?

If your gateway applies tail sampling, spans of one trace must reach the same Collector instance. Arrow streams do not change that rule. Arrange trace-aware routing in the tier that needs it, and remember that long-lived streams can concentrate load on one instance, so keep an eye on balance at that tier in particular.

OTel Arrow is powerful in the right place and wasteful in the wrong one. A balanced view tells you which is which.

OTLP with compression versus OTel Arrow

Other ways to reduce telemetry cost

OTel Arrow is one tool among several, and often the best results come from combining them. The table shows how they differ in what they change.

When it is not worth it

If your telemetry volume is small, your network is fast and free, or your biggest costs are in storage and querying rather than transport, the extra complexity of OTel Arrow may not pay off. Likewise, if your data is mostly free-form text or unique identifiers, there is little repetition to exploit. In such cases, plain OTLP with compression and thoughtful sampling is simpler and good enough.

A new encoding is a new thing to secure and a new thing to watch.

Security considerations

Compliance

Because the meaning and content of the telemetry are unchanged, your obligations are the same as with OTLP. If your rules require encryption in transit, enforce it on the Arrow hop. If your rules forbid certain data from crossing a border, remember that compression does not change what is being sent, only how many bytes it takes.

Monitoring OTel Arrow

A troubleshooting guide

Teams keep rediscovering the same good habits and the same expensive mistakes. Here they are, named.

Patterns worth copying

Anti-patterns to avoid

The habits that separate a smooth OTel Arrow rollout from a frustrating one, gathered in one place.

Start with the bill, not the technology

The strongest reason to adopt OTel Arrow is a measurable cost: a cross-region transfer charge, a metered egress link, or a constrained edge connection. Write down the number before you begin. Without it, you cannot tell whether the project succeeded, and you cannot defend the extra complexity when someone asks why the change was made.

Best practices at a glance

Common mistakes and their cures

A short vocabulary cheat sheet

People new to the topic often get lost in words. Here is a compact list you can keep beside you while reading documentation or talking to colleagues.

Four typical situations, described in general terms, showing where OTel Arrow helps, where it does not, and what to ask before you adopt it.

Questions to ask before you adopt

A calm, step-by-step path from “this sounds interesting” to “this is running quietly in production”.

Maturity levels

Short, direct answers to the questions people ask most often.

It is a way of sending OpenTelemetry data in columns, using the Apache Arrow format over long-lived streams, so that repeated values are stored and sent only once and the data compresses much better than plain record-by-record OTLP.

No. It is a companion. Applications and SDKs continue to speak OTLP, and Collectors can translate to Arrow for a hop and back to OTLP at the other end.

Instead of storing each record as a bundle of fields, the data is stored one field at a time. All timestamps sit together, all names sit together, and so on. Similar values placed side by side are easy to shrink.

An open, language-neutral format for tabular data held in columns. It is designed so that different programs can share the same data cheaply, and OTel Arrow uses it as its container.

General compression looks for repetition in a stream of bytes. Columnar layout puts similar values next to each other and uses dictionaries and runs that remember repetition across batches, so there is far more repetition for any compressor to exploit.

It depends on your data. Highly repetitive telemetry with many shared labels can shrink a lot, while data full of unique values shrinks less. Measure with a replay of your own traffic rather than trusting any quoted figure.

More processor time to convert and encode, more memory to hold stream state, and more operational care around load balancing and upgrades. You pay in effort and compute to save on network bytes.

Usually not. The best fit is between Collectors on expensive hops. The link between an application and a local agent is cheap, so the savings would not justify the added work.

The Collector components have covered traces, metrics, and logs. Check the current documentation for the exact support and stability level in the version you use.

In the contrib collection of components. If you use a custom distribution, include the Arrow receiver and exporter in your manifest.

With fallback enabled, the sender notices and sends plain OTLP instead. Nothing is lost, but you also do not get the saving, so watch for fallback events.

Both sides lose their remembered schema and dictionaries. Batches that were not acknowledged are retried, a new stream is opened, and compression recovers as dictionaries are rebuilt.

Because a stream is long-lived, it stays with whichever receiver it first reached. Without rotation or a gRPC-aware balancer, some receivers end up busy while others sit idle.

No. It changes the shape of the data, not who can read it. Keep encryption, authentication, and redaction exactly as you would with OTLP.

Compare bytes sent before and after, check that fallback counters are low, and confirm that stream churn and retries are calm. If the bytes do not fall, one side is probably not using Arrow.

Not if you convert back to OTLP at the end of the Arrow section. Many teams use Arrow only between their own Collectors and send OTLP onward to the backend.

That depends on the component’s stability level in the release you run, and the answer has changed over time. Read the current documentation, test with your own traffic, and start with one hop.

It can add a small delay while a batch fills and is encoded. Keep batches large enough to compress well but small enough to meet your freshness needs, and measure end-to-end delay.

Yes, and it is a good pairing on unreliable links. The queue protects data during outages, and Arrow helps the queue drain faster when the link returns.

A better algorithm still works on the same record-by-record layout. OTel Arrow changes the layout itself, which exposes patterns that no byte-level compressor can see, and keeps dictionaries across batches.

Pick your most expensive hop, replay a sample of its traffic through a test pair with Arrow on, and compare bytes and processor use against plain OTLP.

When the hop is cheap, volumes are small, processors are already stretched, or nobody can own the operation. In those cases plain OTLP with general compression is simpler and good enough.

Everything in this guide, gathered into a few ideas you can carry with you.

Telemetry is repetitive. The same service names, metric names, labels, and resource details appear again and again, and plain OTLP writes them out record after record. OTel Arrow takes the same data, arranges it into columns, replaces repeated values with small references, and sends it over long-lived streams that remember what has already been shared. The result is a much smaller flow of bytes on the hops where bytes are expensive, paid for with some extra processor time and memory.

It is a companion to OTLP and not a replacement. You use it where it pays, usually between Collectors across regions or from the edge to a gateway, and you leave the rest of your pipeline alone. With limits, rotation, fallback, and good dashboards, it can run quietly. Without them, it can surprise you. The wise path is to measure first, start small, and grow from evidence.