Before comparing two roads, we need to know what is being carried and why anyone cares how it is carried.

Telemetry in one minute

Telemetry is the data your software produces to describe itself. It comes in three main flavours, which OpenTelemetry calls signals. Traces follow one request as it hops through many services, like a parcel tracking page. Metrics are numbers measured over time, such as requests per second or memory used, like the dashboard in a car. Logs are timestamped text records of things that happened, like a ship’s diary. Together they let engineers answer the question “what is my system doing and why is it slow?”

What OpenTelemetry and OTLP are

OpenTelemetry, often shortened to OTel, is an open project under the Cloud Native Computing Foundation. It gives developers one standard way to create telemetry, no matter which programming language or which monitoring vendor they use. Before OTel, every vendor had its own agent and its own format, so changing vendors meant rewriting code. OTel removed that lock-in.

OTLP stands for OpenTelemetry Protocol. It is the standard “envelope format” and delivery contract for telemetry. It defines what the data looks like (a precise schema) and how it is sent (the transports). Think of it as the postal standard: the shape of the envelope, the address layout, and the delivery rules. Anything that speaks OTLP can talk to anything else that speaks OTLP, whether it is an SDK inside your app, an OpenTelemetry Collector, or a commercial observability backend.

Why the transport matters

If both transports carry the same data, why does the choice matter? Because the data does not travel through a vacuum. It passes through your application’s network library, a sidecar or agent, one or more proxies, a load balancer, firewalls, maybe a service mesh, a cloud gateway, and finally the backend. Each of those parts has opinions. Some love long-lived HTTP/2 connections. Some choke on them. Some can inspect plain HTTP requests easily but treat gRPC as an opaque stream. Some cloud services only accept one of the two. The transport decides how well your telemetry survives that journey.

It also matters because telemetry is a high-volume, never-ending firehose. A large e-commerce platform can produce tens of millions of spans (single timed steps inside a trace) every minute. Small percentage differences in CPU use, bytes on the wire, or connection overhead turn into real money when multiplied by that volume.

Five small ideas explain almost every trade-off later in this guide. Read them slowly once and the rest will feel obvious.

HTTP: the universal request and reply

HTTP is the language of the web. A client sends a request (“here is some data for this address”) and the server sends back a reply with a status number. A status of 200 means success, 400 means the request was bad, 429 means “slow down, you are sending too much”, and 503 means “I am overloaded right now”. HTTP is everywhere, so every proxy, firewall, load balancer, and monitoring tool understands it. That universality is its greatest strength.

HTTP/1.1 versus HTTP/2

HTTP/1.1 is the older version. On one connection, requests are mostly handled one after another, like a single-lane road. To send many things at once, a client opens many connections, which wastes effort. HTTP/2 is the newer version. It lets many requests share one connection at the same time, a feature called multiplexing, like turning a single-lane road into a motorway with many lanes. It also compresses the repeated header text and keeps the connection open for a long time. gRPC requires HTTP/2. OTLP over HTTP can use either version, depending on the client and server.

Protocol Buffers: the compact packing format

Protocol Buffers, or protobuf, is a way of turning structured data into very small binary bytes. Instead of writing “service name” in full words for every record, protobuf uses a tiny numeric tag. The result is much smaller than text and quicker for computers to read. The price is that humans cannot read it directly, like Morse code. OTLP defines all its messages in protobuf, which is why both transports share one data model.

JSON: the readable text format

JSON is a plain-text format that people can read and edit. It is larger and slower to process than protobuf, but it is easy to inspect with simple tools and easy to produce from any language, including web browsers. OTLP over HTTP allows JSON as an alternative to binary protobuf. OTLP over gRPC does not use JSON; it always uses binary protobuf.

gRPC: a remote phone call between programs

gRPC is a framework, created at Google, for calling a function on another computer as if it were a local function. You define the available “calls” in a protobuf file, a tool generates client and server code, and then programs talk to each other using HTTP/2 underneath. gRPC adds conveniences on top of raw HTTP/2: standard error codes, deadlines, metadata, compression, and built-in support for streaming. Its calls are described as services with named methods. For OTLP there is one service per signal, and each has one main method called Export.

The OTLP specification separates what is said from how it is carried. That separation is the reason you have a choice at all.

The shared data model

OTLP defines a request message for each signal: one for traces, one for metrics, one for logs. Each request has a nested shape. At the top sits a resource, which describes who produced the data (for example, the service name, the version, the cloud region, the pod). Under the resource sit groups called scopes, which identify the instrumentation library that produced the data. Under the scope sit the actual records: spans, metric data points, or log records. This nesting is deliberate. Describing the resource once and then listing thousands of spans under it avoids repeating the same labels in every record, which saves bytes.

The three official transports

The specification describes OTLP/gRPC, OTLP/HTTP with binary protobuf, and OTLP/HTTP with JSON. In practice, people talk about the first two as “the two choices”, with JSON as a flavour of HTTP. In all three cases the client sends an Export request and receives an Export response that says whether everything was accepted.

Where the choice is made

You make the choice in two places. First, on the exporter inside your application’s OpenTelemetry SDK. The exporter is the component that packages telemetry and sends it. Most SDKs let you pick the protocol through a setting, commonly an environment variable that takes values such as grpc, http/protobuf or http/json. Second, on the receiver inside the OpenTelemetry Collector or your backend. A Collector’s OTLP receiver can listen on both ports at the same time, so one Collector can serve gRPC clients and HTTP clients together. Defaults differ between language SDKs and versions, so never assume. Check what your SDK actually does, and set the protocol explicitly.

Where do the two transports actually sit in a real telemetry pipeline? Here is a typical production layout.

The cast of components

An application uses an OpenTelemetry SDK, which creates spans, metrics and logs. A batch processor inside the SDK groups many records together so they are sent in bulk rather than one by one. The OTLP exporter sends each batch over the chosen transport. On the receiving end, an agent Collector often runs close to the application, for example as a sidecar container or a daemon on every host. Many companies then add a gateway Collector, a shared pool of Collectors behind a load balancer that does heavy work such as filtering, sampling (keeping only a fraction of traces), and adding labels. Finally the data goes to a backend: a tracing store, a metrics database, or a vendor service.

flowchart LR
    APP["Application with OTel SDK"] -->|"OTLP gRPC 4317"| AGENT["Agent Collector on same host"]
    WEB["Browser app with OTel JS"] -->|"OTLP HTTP 4318 JSON or protobuf"| EDGE["API Gateway or Edge Proxy"]
    EDGE --> GWLB["Layer 7 Load Balancer"]
    AGENT -->|"OTLP gRPC or HTTP"| GWLB
    GWLB --> GW1["Gateway Collector 1"]
    GWLB --> GW2["Gateway Collector 2"]
    GWLB --> GW3["Gateway Collector 3"]
    GW1 --> BACKEND["Observability Backend"]
    GW2 --> BACKEND
    GW3 --> BACKEND
    

Reading the diagram

Start on the left. The backend application sits next to an agent Collector on the same machine. The distance is tiny and the network is trusted, so gRPC works beautifully there. A browser application is a different story: it lives on a stranger’s laptop, travels over the public internet, and cannot speak raw gRPC. It sends OTLP over HTTP, passing through an edge proxy that checks who is calling. Both streams then arrive at a layer 7 load balancer, which understands application-level protocols and can spread requests across several gateway Collectors. The gateways finally forward everything to the backend.

Notice that the transport can change at every hop. The agent could receive gRPC from the app and forward HTTP to the gateway. The Collector acts as a translator, which is one of its most valuable features. You are not locked into one transport for the whole pipeline; you choose the best transport per hop.

Let us open the hood on the first road, step by step.

Step 1: the connection

When the exporter starts, it opens a single TCP connection to the endpoint, usually secured with TLS (the encryption layer behind the padlock in your browser). On top of it, both sides agree to speak HTTP/2. This setup costs a few network round trips, so gRPC clients try to do it once and keep the connection open for hours. A single connection can then carry many requests at the same time, each in its own numbered lane called a stream.

Step 2: the call

Each export is a unary call, which simply means one request followed by one response. The client builds a request containing one batch of spans, metric points or log records, and calls the Export method of the matching service. OTLP defines a trace service, a metrics service and a logs service, each with that one Export method. The spec describes these calls as unary even though gRPC supports streaming; this choice keeps things simple and makes retries easy to reason about.

Step 3: the response

The server answers with a response and a gRPC status code. A normal reply means the batch was received. The response can also carry a partial success section, which tells the client that some records were rejected and why, while the rest were accepted. Error cases use standard gRPC status codes such as UNAVAILABLE (try again later), RESOURCE_EXHAUSTED (server is overloaded or enforcing a limit), or INVALID_ARGUMENT (the data is malformed and retrying will not help).

What gRPC gives you for free

What gRPC costs you

The same machinery that makes gRPC neat also brings burdens. The client library is heavier: in several languages the gRPC package pulls in large native components or many dependencies, which can slow startup, enlarge container images, and complicate building on unusual platforms. The long-lived connection interacts awkwardly with some load balancers, which we explore in a later chapter. And gRPC needs real HTTP/2 end to end; anything in the middle that downgrades or buffers HTTP/2 can break it in confusing ways.

The second road trades some built-in comfort for simplicity and reach.

One address per signal

With OTLP/HTTP, each signal has its own path on the server. Traces go to a path ending in /v1/traces, metrics to /v1/metrics and logs to /v1/logs, all sent as ordinary HTTP POST requests. A base address such as a hostname and port 4318 plus these fixed paths forms the full destination. Many people forget that SDKs usually append these paths for you when you set a general endpoint, but not when you set a signal-specific one. Mixing those up is a classic source of “404 not found” errors.

Choosing the body format

The request body holds the same Export message used by gRPC. The sender labels it with a content type header: one value for binary protobuf and another for JSON. The server replies in the same format. JSON has a few special rules defined by the OTLP spec. For example, trace and span identifiers are written as hexadecimal text rather than the base64 text common in other protobuf-to-JSON conversions, and enumerations are written as numbers. These details matter when you hand-craft test messages, and they explain why a generic protobuf-to-JSON converter sometimes produces data a backend rejects.

Status codes, the web way

OTLP/HTTP speaks in ordinary HTTP status codes. A 200 means the batch was accepted, possibly with a partial success note in the body. A 400 means the request was malformed and the client must not retry the same bytes. Codes 429, 502, 503 and 504 mean “temporary trouble”, and well-behaved clients retry them after waiting, honouring a Retry-After header when the server sends one. Because these codes are universal, every generic tool in your network understands them. A load balancer’s dashboard can show you the error rate without knowing anything about OpenTelemetry.

Connections and versions

OTLP/HTTP clients typically use keep-alive, so connections are reused between requests, though less aggressively than gRPC. Depending on the language library, the connection may be HTTP/1.1 or HTTP/2. The specification does not demand HTTP/2 here, which is precisely why it is more forgiving. The exporter also supports gzip compression of the request body, a flag that often cuts bandwidth dramatically for repetitive telemetry.

Choosing a transport is only one of several exporter settings. Understanding the full set of knobs helps you avoid the most common misconfigurations.

The standard settings

OpenTelemetry defines a family of standard environment variables so that every language SDK can be configured the same way. You can set them without touching code, which is perfect for containers. The same settings can also be written in code or in a configuration file. The important ones are listed below in plain words.

General versus signal-specific settings

Each setting usually has a general form that applies to all signals and a signal-specific form for traces, metrics or logs only. The signal-specific forms win when both are present. There is one famous difference in how endpoints are treated: when you supply the general endpoint for HTTP, the SDK appends the standard signal path for you, but when you supply a signal-specific endpoint, the SDK uses your value exactly as given and does not add anything. Mixing the two styles is the number one reason people see “not found” errors on an HTTP setup that looks perfectly fine.

Why defaults deserve suspicion

The specification recommends one default protocol, but individual SDKs have shipped with different defaults over the years, and some changed their default between versions. A service that moves from one language runtime or SDK version to another can silently change transport. That is harmless when everything accepts both, and a disaster when only one port is open. Setting the protocol and endpoint explicitly in every deployment is cheap insurance.

Headers and secrets

Authentication keys often travel in the headers setting. Because these values are secrets, treat them like passwords: inject them from a secret store rather than writing them into images or version control. Remember also that they are sent on every request, so a leaked configuration exposes the key permanently until it is rotated.

Follow one batch of spans from the moment your code finishes a task to the moment the backend stores it. The journey is nearly identical for both transports, with a few important differences in the middle.

sequenceDiagram
    participant App as Application SDK
    participant Exp as OTLP Exporter
    participant LB as Load Balancer
    participant Col as Collector Receiver
    App->>Exp: Hand over finished batch
    Exp->>Exp: Encode as protobuf and compress
    Exp->>LB: Send Export request
    LB->>Col: Forward request
    Col->>Col: Decode and push into pipeline
    Col-->>LB: Success or error reply
    LB-->>Exp: Pass reply back
    alt Success
        Exp->>App: Batch done, free memory
    else Retryable error
        Exp->>Exp: Wait with backoff and resend
    else Permanent error
        Exp->>App: Drop batch and count the loss
    end
    

Where the two transports diverge

In step 4, the difference becomes real. A gRPC exporter sits on one connection and multiplexes requests on it. If the load balancer in the middle works at the connection level (layer 4), every request from that exporter lands on the same Collector for as long as the connection lives. An HTTP exporter, even with keep-alive, tends to open more connections and a smart layer 7 balancer can send each request somewhere different. This single detail causes much of the “uneven load” pain people report with gRPC, and we return to it in chapter 13.

In step 6, gRPC hands the client a tidy status code, while HTTP hands the client a web status code. Both let the exporter decide what to do, but the details of “which codes are retryable” differ, and an exporter that is configured wrongly for one transport may retry too much or not at all.

A single table cannot decide for you, but it shows where the two roads differ most.

Performance arguments about transports are often loud and rarely precise. Let us break the cost into four honest pieces.

Piece one: bytes on the wire

Binary protobuf is much more compact than JSON, often several times smaller before compression, because field names are replaced by tiny tags and numbers are packed tightly. Both gRPC and HTTP-with-protobuf send these same compact bytes, so on this point they tie. Only HTTP-with-JSON pays the extra cost. After gzip compression, which loves repetitive data, the gap between JSON and protobuf narrows but does not vanish, and compression costs CPU on both ends.

Piece two: header and framing overhead

Every request has housekeeping bytes: addresses, content types, authentication tokens. HTTP/2, used by gRPC, compresses these repeated headers across requests on a connection, so the overhead per request becomes tiny. Plain HTTP/1.1 resends them each time. For telemetry, where each request carries a big batch, header overhead is usually small compared with the payload, which is why this advantage rarely decides anything unless batches are tiny. Large batches mean this piece barely matters. Small batches sent very frequently make it matter more.

Piece three: connection setup

Opening a secure connection costs several network round trips and some CPU for encryption setup. gRPC wins when connections are held open for a long time. HTTP with keep-alive reaches nearly the same result. The difference shows up in short-lived processes. A serverless function that starts, sends one batch, and exits pays the setup cost every time on either transport, but gRPC’s heavier client library also adds startup time, which is why many teams choose HTTP for functions and short jobs.

Piece four: CPU and memory of the client

The SDK exporter lives inside your application. Whatever CPU and memory it uses is stolen from your business logic. gRPC clients can be heavier, but they are also well optimised in languages such as Java and Go. HTTP clients are lighter to load but may be less tuned for large volumes in some languages. Measure in your own environment instead of trusting generic claims. Different language runtimes behave very differently.

Scaling to millions of requests per minute

At that scale, the transport on its own is rarely the bottleneck. The bottlenecks are almost always the batch settings, the Collector’s processing pipeline, the memory limits, and the backend’s ingestion rate. A well-tuned HTTP-protobuf pipeline can handle enormous volume, and a badly tuned gRPC pipeline can fall over at modest volume. The practical scaling advice is the same for both transports: send larger batches less often, compress, run several Collector replicas behind a balancer that spreads the load fairly, and add a memory limiter in each Collector so overload causes polite refusals instead of crashes.

Numbers make trade-offs concrete. Let us size a pipeline step by step, using friendly round figures. The figures are illustrations, not benchmarks, so replace them with measurements from your own system.

The SDK’s batching defaults

Most OpenTelemetry SDKs batch spans with a few well-known defaults. The queue holds up to 2,048 spans. The exporter wakes up about every five seconds. Each export carries at most 512 spans. When the queue fills faster than the exporter can drain it, new spans are dropped silently, with only a counter to hint at the loss. These defaults come from the specification and work well for modest traffic, but you should know them because they control how often your chosen transport is used and how much slack you have during trouble.

An imaginary fleet

Suppose a shop runs 400 application instances. Each instance finishes about 500 spans per second. That is 200,000 spans per second across the fleet. Assume each span takes about 400 bytes once encoded, a plausible but invented figure, because attributes vary widely. The raw rate is then 200,000 times 400, which is 80 million bytes per second. In network terms that is roughly 640 megabits per second before compression, a serious amount of traffic that costs real money to move.

How many requests is that?

At 500 spans per second, one instance fills a 512-span batch in a little over one second, so it sends about one request every second. Across 400 instances, the Collectors receive roughly 400 export requests per second. Each request carries about 512 times 400, or around 205 kilobytes, far under any size limit. This tells us something reassuring: even a busy fleet produces only hundreds of requests per second at the transport layer. A modern Collector handles that comfortably over either gRPC or HTTP. The transport’s per-request overhead is a tiny slice of the total.

How much buffer do you have?

The default queue of 2,048 spans divided by 500 spans per second gives a cushion of about four seconds. If the backend or network stalls for longer than that, the instance starts dropping spans. Four seconds is not much. Raising the queue size buys time, at the cost of memory inside your application, so the right setting is a business decision about how much telemetry loss you accept versus how much memory you can spare. A local agent Collector that buffers on disk or in a larger memory queue is the usual way to buy minutes of protection without bloating every application.

What if someone raises the batch size too far?

Imagine a well-meaning engineer sets the batch to 10,000 spans for “efficiency”. At 400 bytes each, that is four million bytes, about 3.8 mebibytes. The default maximum message size accepted by a Collector’s gRPC receiver is about four mebibytes. A slightly bigger average span, or a busy moment with long attributes, pushes a batch over the limit, and the whole batch is rejected as too large. A rejected batch is not retried successfully, because the same oversized bytes fail again. This is a real and common production incident, and it appears as a mysterious rise in dropped data that no network graph explains.

Compression is the cheapest way to shrink telemetry traffic, but it is not free and not equally available on every hop.

What compression does

Compression finds repeated patterns in data and replaces them with shorter references, like writing “see page 3” instead of repeating a paragraph. Telemetry is highly repetitive: the same service names, the same attribute keys, the same operation names appear in thousands of spans. That is why compression works so well on it. Even binary protobuf, which is already compact, shrinks further with gzip because of this repetition. JSON shrinks even more in relative terms, because its text is so wordy, although it still ends up larger than compressed protobuf in most cases.

The price

Compression spends CPU on the sender to shrink and CPU on the receiver to expand. On a slow or expensive link, that is a bargain. Inside a data centre where bandwidth is cheap and plentiful, the CPU may cost more than the bandwidth saves. The break-even point depends on your hardware and your traffic. As a rule of thumb, compress anything that crosses availability zones, regions, the internet or a paid gateway, and think twice on a loopback or same-host link.

Batch size and compression ratio

Compression works better on bigger inputs because there are more repeated patterns to exploit. A tiny batch of ten spans compresses poorly; a batch of five hundred compresses well. This is another reason to avoid sending very small batches over a costly link. It also means the best savings usually show up on high-volume pipelines, exactly where cost matters.

Support by transport

Both OTLP transports support gzip, and gzip is the one every implementation is expected to understand. Some Collector builds also accept other algorithms such as zstd and snappy on their receivers, and zstd often gives a better balance of speed and size, but SDK support for those algorithms is patchier, so verify your exact versions before relying on them. For gRPC, compression is negotiated per call through the library. For HTTP, it is signalled by a standard content-encoding header, which means ordinary HTTP intermediaries understand it. One subtle point: a proxy that terminates and re-sends the request may decompress and re-compress, or may pass the compressed bytes through unchanged. Know which your proxy does, because it affects both CPU and the limits it applies.

This is where most real-world gRPC pain lives, so it deserves a careful, slow explanation.

The sticky connection problem

Suppose you put four Collectors behind a simple layer 4 load balancer. A layer 4 balancer only sees connections, not individual requests. It decides which backend gets each new connection and then leaves it alone. With plain HTTP/1.1, clients open and close connections often, so over time the connections spread nicely across the four Collectors. With gRPC, a client opens one connection and keeps it for hours, sending everything over it. The balancer picked one Collector at the start, so all that client’s traffic goes there. If you have only a handful of clients, you can easily end up with one overloaded Collector and three idle ones.

Fixes for gRPC

Cloud load balancers and gRPC

Cloud balancers differ. On AWS, for example, the Application Load Balancer can route gRPC when a target group is configured for the gRPC protocol version, and it understands gRPC health check codes. A Network Load Balancer works at layer 4, so it passes the connection through without request-level spreading. Some API gateway products do not support gRPC at all. Always check the exact capabilities of every component between your exporters and your Collectors before you commit to gRPC, and test with realistic traffic, not a single curl command.

Rolling deployments and connection draining

When you update a Collector, the old instance must finish its work and say goodbye. With long-lived gRPC connections, a graceful goodbye needs the server to tell clients to move on, usually by sending a go-away signal when the connection reaches a maximum age or the server shuts down. If the shutdown is abrupt, clients see errors and the in-flight batches may be lost or retried. HTTP connections are more disposable, so abrupt shutdowns hurt less, though they still cause some failed requests.

Long-lived connections are wonderful until some device between you and the server quietly forgets about them.

The idle connection trap

Networks keep a small memory of every open connection, in firewalls, balancers and address translators. To save memory, they forget connections that stay silent for too long. A connection that carried no data for a while may be removed without anyone telling the two ends. The client believes the connection is alive and the server believes so too, but the middle has already thrown it away. The next request goes into a void and fails after a long timeout. This is a classic cause of mysterious occasional export errors that vanish when you retry.

Typical idle limits

On AWS, an Application Load Balancer closes connections that stay idle for 60 seconds by default, and a Network Load Balancer drops idle TCP connections after about 350 seconds. Other balancers, firewalls and gateways each have their own numbers, often between half a minute and several minutes. Telemetry exporters can be quiet for a while, such as on a nightly low-traffic service, so these limits do get hit in real life. Remember that these figures are defaults at the time of writing and can be changed, so always check your own configuration.

Keepalive pings

The remedy for gRPC is keepalive: the client sends a tiny ping every so often, which keeps the middle from forgetting the connection and quickly reveals a dead connection. The ping interval must be shorter than the shortest idle timeout in the path, but not so short that it annoys servers. Servers can enforce a minimum allowed ping interval and will close connections from clients that ping too often, a protection that surprises people who set aggressive values. Tune both ends together.

HTTP is more forgiving

An HTTP exporter that uses keep-alive faces the same idle drop, but HTTP client libraries typically detect a closed connection when they try to reuse it and simply open a fresh one, often retrying the request transparently. The failure is quieter and less costly. gRPC has built-in reconnection logic too, with its own backoff between attempts, but during the reconnection delay exports can fail or queue, and that delay can be several seconds.

Timeouts that protect you

Every export call should have a deadline. If the server takes too long, the exporter must give up, free the batch for a retry or a drop, and move on. Without deadlines, a stuck request holds memory and slows the queue behind it. Pick a deadline longer than your normal worst-case response but short enough that a stall is noticed in seconds, and align it with any timeouts configured in balancers so the shortest one is not an unpleasant surprise.

Telemetry pipelines fail in ordinary ways: backends slow down, networks blink, deployments restart things. Good transport behaviour decides whether you lose a little data or a lot.

What counts as retryable

The OTLP specification tells clients to retry on temporary failures and to give up on permanent ones. For gRPC, the retryable list includes codes such as UNAVAILABLE and DEADLINE_EXCEEDED, and RESOURCE_EXHAUSTED only when the server explicitly signals that retrying later is useful. For HTTP, the retryable list is 429, 502, 503 and 504, and clients should respect the Retry-After header when it is present. A bad-request answer should never be retried because the same bytes will fail the same way.

flowchart TD
    START["Export attempt"] --> SEND["Send batch to endpoint"]
    SEND --> RESULT{"Reply received?"}
    RESULT -->|"Success"| DONE["Batch accepted and forgotten"]
    RESULT -->|"Partial success"| PART["Log rejected count and move on"]
    RESULT -->|"Retryable error"| WAIT["Wait with exponential backoff and jitter"]
    RESULT -->|"Permanent error"| DROP["Drop batch and count the loss"]
    WAIT --> LIMIT{"Retry budget left?"}
    LIMIT -->|"Yes"| SEND
    LIMIT -->|"No"| DROP
    

Backoff and jitter

Exponential backoff means that each retry waits longer than the one before it, for example one second, then two, then four. Jitter means adding a small random amount to each wait. Without jitter, a thousand clients that failed at the same moment would all retry at the same moment, hitting the recovering server like a stampede. Both transports expect this behaviour from the exporter. In practice, you tune it in the SDK, not in the transport.

Partial success

Sometimes a server accepts most of a batch and rejects a few records, perhaps because a span is too old or has an invalid field. OTLP handles this with a partial success reply that includes the number of rejected items and a message. This is a success from the transport’s view: do not retry the whole batch, because the accepted records would be duplicated. Both gRPC and HTTP carry this field identically. Many dashboards ignore it, so watch for it in your Collector logs.

Backpressure: saying “slow down”

Backpressure is how a busy server asks senders to ease off. Over HTTP, a 429 or 503 reply with a Retry-After note does that clearly and every tool in the middle understands it. Over gRPC the equivalent is a RESOURCE_EXHAUSTED or UNAVAILABLE status. Collectors can also apply backpressure by refusing data when their memory limiter trips. Make sure your exporters queue a limited amount, because endless queues turn a short outage into an out-of-memory crash in the application itself.

Failure scenarios

Telemetry often contains sensitive details: user identifiers, internal host names, database statements, even fragments of customer data. Transport security is not optional.

Encryption in transit

Both transports support TLS, the encryption standard that keeps eavesdroppers from reading traffic. Always use it outside a fully private network. For gRPC, TLS also enables negotiation of HTTP/2 in the common setup. For HTTP, use an https address. A telltale beginner mistake is leaving the insecure option switched on from a local test and shipping it to production.

Mutual TLS

Mutual TLS, or mTLS, means both sides show certificates, so the server knows exactly which client is calling, not only the client knowing the server. Collectors support it on both receivers. It is a common choice for agent-to-gateway links inside a company, because the certificates can be handed out automatically by a service mesh or certificate authority.

Authentication tokens

Sending a secret key with each request is the other common method. Over HTTP this is the familiar Authorization header, which gateways and web application firewalls can inspect and rate-limit easily. Over gRPC, the same secret travels as metadata, which works equally well, but generic web tools may not understand it. Many vendors expect an API key header on every request. Some cloud services ask for signed requests; these signing schemes are defined for HTTP and fit OTLP/HTTP much more naturally. For example, AWS-hosted OTLP endpoints that need signed requests work over HTTP, and a Collector sits in between to sign on the application’s behalf.

Abuse and limits

A public receiver can be flooded or fed huge payloads. Set limits on message size, rate-limit by client, and put authentication in front. HTTP gives you a rich ecosystem for this: web application firewalls, API gateways and standard rate limiters. gRPC support in those tools exists but is patchier. The browser case is the sharpest: any internet user can send data to your public telemetry endpoint, so treat it like any public API.

Observability bills often surprise teams. Part of the bill is the backend and part is simply moving the bytes, and the transport settings influence that second part.

Where the money goes

In a cloud environment, moving data between availability zones, between regions or out to the internet usually costs a small amount per gigabyte. Traffic through a managed network address translation service typically adds its own per-gigabyte processing charge. Telemetry that flows across these boundaries all day therefore appears as a steady line on the invoice. Compute costs also rise with compression and serialisation work. Finally, backends often charge by the amount of data ingested, which is a separate matter that the transport does not change.

A rough illustration

Take the imaginary fleet from the sizing chapter, producing about 80 megabytes per second before compression. If gzip shrinks the payload to about one fifth, a reasonable guess for repetitive telemetry that you should verify, the wire rate falls to about 16 megabytes per second. Over a day that is about 1.4 terabytes. If cross-zone transfer cost about one cent per gigabyte in each direction, which is the approximate order of magnitude on some clouds at the time of writing, moving that volume across zones would cost on the order of 28 dollars a day, or roughly 830 dollars a month. Without compression the same traffic would be five times larger, and the cost would be roughly 4,000 dollars a month. The numbers are illustrative only, but they show why compression and smart placement matter more than which transport you pick.

Cheap wins

• Place agents on the same host or zone as the applications, so the chatty hop stays local and free.
• Sample and filter early in the agent, so fewer bytes cross expensive boundaries.
• Compress on paid links and use bigger batches so compression works better.
• Keep traffic off NAT paths when you can, by using private endpoints for cloud-native destinations.
• Prefer protobuf over JSON for any hop that carries real volume.

Does the transport itself change the bill?

Only a little. gRPC and HTTP with protobuf send nearly the same number of payload bytes. Small differences arise from headers, connection overhead, and how well load is spread across zones. Layer 7 balancers, which are needed for good gRPC distribution, are often billed by capacity units based on connections, requests and bytes, so a more complex balancing setup can cost more than a simple one. Compare total cost of ownership rather than payload size alone.

The right transport often depends less on raw speed and more on the strange environments your telemetry must cross. Here are the places where the choice is practically forced.

Web browsers

A web page runs inside a sandbox that allows ordinary HTTP requests but does not give JavaScript access to the low-level HTTP/2 features gRPC needs. A browser application therefore cannot speak standard gRPC directly. It can use a translation layer called gRPC-Web, which needs a special proxy, but the OpenTelemetry browser tooling avoids that complication by sending OTLP over HTTP. The Collector’s HTTP receiver also offers cross-origin settings, the web rules that decide which websites may send data to it. Forgetting these settings gives you mysterious blocked requests that only appear in the browser’s developer console. If you collect front-end telemetry, OTLP/HTTP is effectively the only sensible choice for that hop.

Corporate proxies and strict firewalls

Many companies force outbound traffic through inspecting proxies. Those proxies are built for ordinary web traffic. Some break HTTP/2 features, some buffer entire responses, and some refuse unknown content types. gRPC is more likely to hit one of those walls than plain HTTP. If your exporters live in a locked-down customer network, as with software installed on customer premises, OTLP/HTTP is the polite guest that gets through the door.

Serverless functions and short-lived jobs

Serverless platforms run your code only while a request is being handled and may freeze or destroy the environment right afterwards. In this world, the key is to flush telemetry before the function finishes. A lighter HTTP exporter starts faster and has fewer native dependencies to package, which matters because deployment bundles in serverless are small. Many teams therefore send function telemetry over HTTP to a nearby Collector or directly to a vendor, reserving gRPC for long-running services.

Mobile and edge devices

Phones switch between networks, go to sleep, and run on batteries. HTTP is far better supported by mobile networking libraries, and a lightweight exporter drains less battery. Compression becomes especially valuable because mobile data can be slow and expensive. Telemetry from devices should also be batched aggressively and sent when the device is on a good connection.

Service meshes and sidecars

If your platform runs a service mesh, a sidecar proxy already handles HTTP/2 and gRPC awareness, mutual TLS, and request-level balancing. In that environment gRPC to a Collector works smoothly, because the mesh solves the connection problems for you. HTTP also works, and the mesh gives you the same metrics for both. This is the place where teams most often decide based on whichever protocol their client libraries support best.

Once the basics feel comfortable, these deeper ideas explain behaviours that puzzle even experienced engineers.

Head-of-line blocking

Head-of-line blocking is a traffic jam where one slow item holds up everything behind it. HTTP/1.1 suffers from it at the request level: on one connection, a slow response blocks the next. HTTP/2 solves that at the request level through multiplexing, so many streams proceed independently. But HTTP/2 still runs over a single TCP connection, and TCP delivers bytes in strict order. If one packet is lost, TCP pauses all streams on that connection until the packet is resent. So a gRPC client with one hot connection can see all its in-flight exports stall together on a lossy network. A newer web protocol, HTTP/3, avoids this by running on a different foundation called QUIC, but OTLP’s specification does not currently define gRPC over HTTP/3, so do not plan around it unless your libraries and balancers explicitly support it.

At-least-once delivery and duplicates

OTLP gives at-least-once delivery in practice. If a request reaches the server but the reply is lost on the way back, the client sees a timeout and retries, and the server receives the same batch twice. OTLP has no built-in duplicate detection. Traces and logs can show duplicates in rare cases, and counters in metrics can be double counted unless the backend handles it. This is true for both transports. It is another reason to retry only a limited number of times and to design backends to tolerate repeats.

Streaming and the Arrow experiment

Standard OTLP makes one unary call per batch. gRPC supports streaming, where one call stays open and carries many batches. A related community project, OpenTelemetry Arrow, explores streaming telemetry in a columnar format designed for strong compression, and it reports substantial bandwidth savings for large deployments. It relies on gRPC streaming, so this is one area where gRPC has options HTTP does not. It is a more specialised path and has more moving parts, so treat it as something to evaluate when bandwidth cost is a dominant problem, not as a starting point.

Message size and memory

Every receiver reads a request fully into memory before decoding it. With many large requests in flight at once, memory use can spike. That is why Collectors offer a memory limiter and limits on message size and concurrency. Setting these thoughtfully matters more at high volume than anything in the transport itself.

Ordering is not guaranteed

With multiple retries and multiple Collectors, batches can arrive out of order. Backends sort records by their timestamps, so this rarely matters, but any custom processing that assumes arrival order will be surprised. The transport promises delivery of each batch, not the order of batches.

Partial success revisited

The partial success mechanism is easy to ignore. A backend may accept your batch but reject, say, spans with timestamps too far in the past. If the client treats every successful reply as full success and never looks at the rejected count, you may lose data for weeks without noticing. Make sure someone reads those counts, in the SDK logs or the Collector metrics.

flowchart LR
    CL["Client sends batch"] --> SRV["Server receives and processes batch"]
    SRV --> LOST{"Reply lost in transit?"}
    LOST -->|"No"| OK["Client sees success"]
    LOST -->|"Yes"| TIMEOUT["Client times out"]
    TIMEOUT --> RESEND["Client resends same batch"]
    RESEND --> DUP["Server stores batch a second time"]
    

A pipeline that watches everything else needs someone watching it. Here is how to see inside the transport layer.

What to measure

The OpenTelemetry Collector publishes its own metrics about itself. The most useful ones count accepted and refused items per receiver, items sent and failed per exporter, current queue size against queue capacity, and memory usage against its limit. If accepted items stay high while sent items drop, something downstream is stuck. If refused items rise at the receiver, upstream senders are being told to back off. Watch these four numbers on a dashboard and alert on them before your engineers notice missing traces.

Transport-specific signals

Debugging tips by transport

HTTP is friendlier to humans. You can capture a request, look at the headers, and, with the JSON flavour, even read the body. When something breaks, a quick manual test with a command-line HTTP tool often tells you whether the endpoint, path, and authentication are right. For gRPC you need tools that understand gRPC framing, or you can enable detailed logging in the Collector and the SDK. A useful universal trick is to add a debug exporter in the Collector temporarily so it prints what it receives, which isolates whether the problem is before or after the Collector.

Another helpful habit is to monitor the monitoring: run a tiny synthetic service that emits a heartbeat span every minute and alert if it ever stops showing up in the backend. This catches protocol mismatches, expired certificates and broken routes much faster than waiting for engineers to complain.

The Collectors are the workhorses of the pipeline. Planning their capacity well turns transport tuning from guesswork into engineering.

What consumes Collector resources

A Collector spends CPU on decompression, decoding, processing steps such as sampling and attribute changes, re-encoding and exporting. It spends memory on request buffers, queues waiting to be exported and any state needed by processors, such as keeping spans of a trace together until the trace finishes. The transport affects the first few items only slightly. The processing pipeline typically dominates. A pipeline full of regular-expression rewriting or heavy tail sampling can cost many times more CPU than receiving the data did.

Scale on the right signal

Many teams scale Collectors on CPU alone, but CPU can lag behind overload. Better indicators include queue utilisation, the rate of refused items at the receiver and memory approaching its limit. If the exporter queue is 80 percent full and still rising, the backend or the Collector is the bottleneck, and adding replicas helps only if the bottleneck is Collector CPU or memory. Adding replicas in front of a slow backend simply moves the queue.

Stateless versus stateful tiers

Simple Collectors that batch and forward are stateless: any replica can handle any request, so scaling out is easy and balancing is simple. Collectors that perform tail sampling, which means deciding to keep or drop a whole trace after seeing all of its spans, are stateful: every span of a trace must reach the same Collector. That requires routing by trace identifier, usually through a first-tier Collector that spreads spans consistently to a second tier. The transport matters here, because the routing tier should hold connections to every second-tier Collector and balance by trace, not by connection. Keep the stateless and stateful tiers separate so each can scale on its own signal.

flowchart LR
    APPS["Applications"] -->|"OTLP"| T1["Tier 1 stateless Collectors"]
    T1 -->|"Route by trace ID"| T2A["Tier 2 tail sampling Collector A"]
    T1 -->|"Route by trace ID"| T2B["Tier 2 tail sampling Collector B"]
    T1 -->|"Route by trace ID"| T2C["Tier 2 tail sampling Collector C"]
    T2A --> BE["Backend"]
    T2B --> BE
    T2C --> BE
    

Headroom and failure planning

Plan for the loss of a whole availability zone. If one zone’s Collectors disappear, the others must absorb the extra traffic without tipping over. A common rule is to run at no more than about half of capacity in normal times when you have two zones, so a zone failure is survivable. Test it by deliberately removing instances during a quiet period and watching refusal counts and queue depths.

Teams that run telemetry at scale keep rediscovering the same handful of designs. Here are the helpful ones and the traps.

Pattern: agent plus gateway

Run a small agent Collector next to every application host or pod, and a pool of gateway Collectors in the middle. The application sends to its local agent over the fastest, simplest path, usually gRPC on the same machine. The agent batches, adds host information, and forwards to the gateway. The gateway applies sampling and routes to backends. This isolates applications from backend problems, because they only talk to something extremely close and reliable.

Pattern: protocol translation at the edge

Accept whatever the clients can speak, such as HTTP from browsers and partners and gRPC from internal services, in an edge Collector. Then convert to the single protocol your backend likes best. Clients get flexibility, and the backend sees consistency. Because the Collector can listen on both ports at once, this costs almost nothing to set up.

Pattern: separate pipelines per signal

Traces are bursty and large, metrics are steady and small, logs can be enormous. Giving each signal its own pipeline, and sometimes its own Collector pool, stops a log storm from starving your traces. The transport does not need to be the same for each pipeline either.

Half-remembered advice travels fast. Here are the most common myths about this topic, corrected.

A short, practical checklist you can apply regardless of which transport you choose.

Do these things

• Set the protocol explicitly. Never trust SDK defaults, because they differ between languages and versions.
• Match ports to protocols. 4317 for gRPC, 4318 for HTTP, in both client and receiver.
• Prefer binary protobuf for production, whichever transport you use.
• Turn on compression between networks, and skip it on very fast local links where CPU is more precious than bandwidth.
• Tune batching so batches are large enough to be efficient but small enough to stay under size limits.
• Add a memory limiter to every Collector so overload produces polite refusals instead of crashes.
• Bound exporter queues and retries so a long outage cannot harm the application.
• Use TLS everywhere outside a fully trusted private network, and mTLS between your own tiers when you can.
• Run a synthetic heartbeat to detect silent breakage fast.
• Test with realistic load and the real load balancer before going live.

Common mistakes

Different organisations land on different answers because their constraints differ. Here are some recognisable shapes.

Bigger organisations run telemetry across regions, accounts and teams. The transport choice interacts with these boundaries in useful ways.

Regional gateways

Instead of sending every region’s telemetry to one distant place, run a regional gateway tier and forward summaries or sampled data onward. Applications talk to a nearby regional endpoint with low latency, and only the reduced volume crosses the long, expensive link. For that long link, HTTP with compression often gets through assorted network devices more reliably, while gRPC between well-controlled sites works equally well when the path is private and HTTP/2 aware.

Multi-account organisations

On AWS, larger companies usually spread workloads over many accounts. A central observability account typically hosts the shared Collectors and backend access. Applications in other accounts reach it through private connectivity, such as a private link to an endpoint service, which exposes a network load balancer in the central account. Because that front door is often a layer 4 balancer, gRPC clients meet the sticky-connection problem there, while HTTP clients balance more evenly. If you choose gRPC, plan for connection age limits on the central Collectors.

Multi-tenancy

Multi-tenancy means many teams or customers share one pipeline while their data stays separate. Typical needs are tenant identification, per-tenant limits and per-tenant routing. HTTP makes tenant identification easy: a header can carry the tenant, and standard gateways can route and rate-limit on it. gRPC carries the same information as metadata, which a Collector processor or gRPC-aware proxy can read, but fewer generic tools understand it. A commonly chosen shape is a front-door gateway that authenticates each tenant, tags the data with the tenant’s identity, and enforces quotas before the shared Collectors see anything.

Data residency

Some regulations require telemetry to stay within a geographic area. Regional gateways, with filtering that removes sensitive attributes before forwarding, help satisfy such rules. The transport itself is neutral here. What matters is that the routing tier is under your control and that unfiltered data never crosses a boundary by accident.

Instead of arguing in the abstract, walk each hop of your pipeline through a few simple questions.

flowchart TD
    Q1{"Is the sender a browser, mobile app or locked-down network?"} -->|"Yes"| HTTP1["Choose OTLP over HTTP with protobuf"]
    Q1 -->|"No"| Q2{"Is it a short-lived function or job?"}
    Q2 -->|"Yes"| HTTP2["Choose OTLP over HTTP"]
    Q2 -->|"No"| Q3{"Does every hop support HTTP/2 and request-level balancing?"}
    Q3 -->|"No"| HTTP3["Choose OTLP over HTTP or fix the balancer first"]
    Q3 -->|"Yes"| Q4{"Do you need maximum throughput from long-running services?"}
    Q4 -->|"Yes"| GRPC["Choose OTLP over gRPC"]
    Q4 -->|"No"| EITHER["Either works, pick what your team knows best"]
    

A staged rollout

Sooner or later most teams change a hop from one transport to the other, often because a new balancer arrives or a new client type appears. Here is a careful way to do it without losing visibility.

When telemetry goes missing, start from the symptom, then follow the likely causes in order. This table-driven approach saves hours.

A reliable debugging order