The short answer first, then the details that make it useful.

OpenTelemetry, often shortened to OTel, is a free, open source framework hosted by the Cloud Native Computing Foundation. It gives developers one standard way to produce and send telemetry. The OpenTelemetry Collector is a separate, standalone program that sits between your applications and your monitoring tools. It receives telemetry, can change or filter it, and then forwards it onward.

The Collector is flexible, which means it can be placed in several positions. Gateway mode is the pattern where one or more Collector instances run as a central, shared service. Many applications, and often many smaller Collectors, send their data to this service. The gateway then processes everything in one place and sends it to the final destinations, which are called backends. In simple terms, a gateway is a Collector that acts as a single front door for a whole cluster, data center, or region.

It is important to understand that gateway mode is not a separate product or a special download. It is the same Collector program, simply deployed in a certain way. The word “mode” describes where it runs and what job it does, not a different piece of software.

What a gateway usually does

• Receives. It accepts telemetry from many applications and from smaller Collectors.
• Cleans. It removes secrets, personal data, and noisy details before data leaves your network.
• Enriches. It adds helpful labels, such as the environment or the team that owns a service.
• Samples. It decides which traces to keep and which to drop, so costs stay under control.
• Batches. It groups data into efficient bundles before sending.
• Routes. It sends different data to different destinations, or copies to several at once.
• Protects. It holds the passwords to the backends, so applications never need them.

What a gateway is not

A gateway is not a backend. It does not store telemetry for you to search, and it does not draw dashboards. It is also not an API gateway in the web sense, which manages requests from customers to your business services. It handles only telemetry about your software, never the customer traffic itself. Finally, it is not mandatory. Small systems can work happily without one, as we will see.

Why teams reach for a gateway

Most teams do not choose a gateway because it sounds elegant. They choose it because something hurt. Perhaps a monitoring bill doubled and nobody could say which service caused it. Perhaps a security review discovered that secret keys were copied into forty configuration files. Perhaps a change of vendor looked like six months of work. In every one of these stories, the missing ingredient was a single place where decisions about telemetry could be made.

A gateway supplies that place. It gives the platform team a lever they can pull to change what the whole company sends, and it gives application developers the comfort of knowing that sending telemetry is as simple as posting a letter. Everyone does the part they are good at, and nobody needs to understand the whole machine.

Every good pattern begins as a response to a painful problem. Gateways are no exception.

In the early days of tracing and monitoring, applications talked straight to the monitoring product. A developer added a small library, pointed it at the vendor’s address, and pasted in a secret key. For one program this was perfectly fine. For fifty programs written by ten teams, it became a headache.

Every program held its own copy of the secret key, so changing the key meant changing fifty programs. Every program had its own sending rules, so costs were unpredictable. Switching to a different monitoring product meant rewriting instrumentation across the company. And nobody had a single place to say, “never send customer email addresses out of this network.”

Early tracing tools such as Jaeger solved part of the problem by offering two helper programs: an agent that ran next to each application, and a collector that gathered data from many agents. The OpenCensus project built a similar service. When OpenCensus and OpenTracing merged into OpenTelemetry in 2019, the idea of a single vendor-neutral Collector, able to play both the agent and the collector roles, became part of the new standard.

Three pressures pushed companies toward the middle layer. The first was cost, because telemetry volume grows faster than the business itself, and the bills follow. The second was privacy, because laws and customer promises limit what personal data may leave a network. The third was freedom, because nobody wants to be locked into a single vendor for life. A central Collector answers all three at once, which explains why the pattern spread so quickly once the tooling matured.

A handful of words explain nearly everything in the Collector world. Let us meet them one at a time.

The three signals

The Collector handles three kinds of telemetry, each called a signal. Traces follow a single request through many services, like tracking a parcel from the warehouse to your door. Metrics are numbers measured over time, like a patient’s temperature chart. Logs are written records of events, like a ship’s diary. A good gateway treats all three, usually in separate pipelines, so that rules for one signal do not accidentally break another.

OTLP in plain words

OTLP stands for OpenTelemetry Protocol. It is the standard language that applications and Collectors use to hand telemetry to one another. It travels over two common transports: gRPC, a fast, efficient method that keeps a long-lived connection open, and HTTP, the same basic method that web pages use. By convention, gRPC listens on port 4317 and HTTP on port 4318. A port is simply a numbered door on a computer through which a particular kind of traffic enters.

Two Kubernetes words

Because gateways are very often run on Kubernetes, a system for managing many containers, two of its words appear often. A Deployment runs a chosen number of identical copies of a program somewhere in the cluster. A DaemonSet runs exactly one copy on every machine. Gateways are typically run as Deployments, because they are shared services that can live anywhere. Agents are typically run as DaemonSets, because they should live on every machine.

Let us look at the building from the street, and then step inside.

From the outside, a gateway deployment has four parts. First are the producers: your applications, jobs, and smaller Collectors, which create and send telemetry. Second is a front door, usually a load balancer or a Kubernetes service, which gives all senders one stable address. Third are the gateway replicas, which are several identical Collector copies sharing the work. Fourth are the backends, the places where telemetry is finally stored and shown.

flowchart LR
    subgraph CLIENTS["Telemetry producers"]
        S1["Checkout service"]
        S2["Payment service"]
        S3["Inventory service"]
        S4["Batch jobs"]
    end
    LB["Load balancer or Kubernetes service"]
    subgraph GW["Gateway Collector replicas"]
        G1["Gateway replica 1"]
        G2["Gateway replica 2"]
        G3["Gateway replica 3"]
    end
    subgraph BE["Backends"]
        B1["Trace store"]
        B2["Metrics store"]
        B3["Log store"]
    end
    S1 -->|OTLP| LB
    S2 -->|OTLP| LB
    S3 -->|OTLP| LB
    S4 -->|OTLP| LB
    LB --> G1
    LB --> G2
    LB --> G3
    GW --> B1
    GW --> B2
    GW --> B3

Read the diagram from left to right. Many producers send OTLP to one shared address. The load balancer shares the traffic among three identical replicas, so no single copy has to carry the whole load. If one replica fails, the others continue. The replicas then forward the data to whichever backends the team has chosen.

Inside one replica

Each replica is an ordinary Collector, and inside it every signal flows through the same three stages: receivers, processors, and exporters. The order of the processors matters a great deal, as the diagram shows.

flowchart LR
    subgraph REC["Receivers"]
        R1["OTLP receiver: gRPC 4317 and HTTP 4318"]
    end
    subgraph PRO["Processors in order"]
        P1["Memory limiter"]
        P2["Filter and redaction"]
        P3["Attributes and resource labels"]
        P4["Sampling"]
        P5["Batch"]
    end
    subgraph EXP["Exporters"]
        E1["OTLP exporter to backend"]
        E2["Second backend copy"]
    end
    R1 --> P1
    P1 --> P2
    P2 --> P3
    P3 --> P4
    P4 --> P5
    P5 --> E1
    P5 --> E2

The memory limiter goes first, so that the Collector can refuse new data before it runs out of memory and crashes. Filtering and redaction come next, so unwanted or sensitive data is removed as early as possible. Labels are added after that. Sampling decides what to keep. Finally, the batch processor groups the surviving data into efficient bundles just before export. Placing batch last means the bundles are formed from the final data, not from data that will be dropped a moment later.

One more detail helps beginners feel at home. A Collector is controlled by a single configuration that does two jobs. First, it lists the receivers, processors, exporters, and extensions that should exist, along with their settings. Second, it wires them together into pipelines, saying which pieces take part in each path and in what order. Because a piece can be defined once and used in several pipelines, a team can keep the file short and readable even when the gateway does a lot of work.

Gateway mode is one of three common choices. Understanding all three helps you pick wisely.

There are three common ways to send telemetry. In the first, no Collector is used, and applications export directly to a backend. In the second, an agent Collector runs beside every application or on every machine. In the third, a gateway Collector acts as the shared central service. Many organizations combine the second and third, with agents forwarding to a gateway.

The two-tier combination

The most common arrangement on Kubernetes is a two-tier design. A small agent Collector runs on every node and gathers local data, including data that only a node-level program can see, such as machine metrics and container logs. Each agent then forwards what it has to the gateway service, which applies the shared rules and talks to the backends.

flowchart TB
    subgraph N1["Node 1"]
        A1["App pods"] --> AG1["Agent Collector: DaemonSet pod"]
    end
    subgraph N2["Node 2"]
        A2["App pods"] --> AG2["Agent Collector: DaemonSet pod"]
    end
    subgraph N3["Node 3"]
        A3["App pods"] --> AG3["Agent Collector: DaemonSet pod"]
    end
    AG1 --> GWS["Gateway service"]
    AG2 --> GWS
    AG3 --> GWS
    GWS --> GW1["Gateway pod A"]
    GWS --> GW2["Gateway pod B"]
    GW1 --> BK["Observability backend"]
    GW2 --> BK

The benefit of the combination is a clean split of work. Agents are light and close to the programs, so they are good at collecting and adding local details, such as which container produced the data. The gateway is shared and central, so it is good at decisions that need the big picture, such as sampling, redaction, and routing. Neither tries to do the other’s job.

Follow one bundle of telemetry on its journey, from the moment it leaves an application to the moment it reaches storage.

Why the queue matters

Backends are not perfect. They may be slow, restarting, or briefly unreachable. If the gateway tried to send each bundle and simply gave up on failure, data would be lost every time the backend hiccuped. So the exporter keeps a sending queue, a waiting line of bundles, and a retry rule that tries again after a growing pause. The queue turns a short outage from a disaster into a small delay.

What the sender is told

OTLP is a conversation. After the sender transmits a bundle, the receiver answers that it was accepted, or that something is wrong. When the exporter queue is enabled, the gateway usually answers “accepted” as soon as the data is safely in its queue, not after the backend confirms. This keeps applications fast, because they never wait for the slow last mile. The trade-off is that data sitting only in memory could be lost if the gateway crashes at that moment, which is why a disk-backed queue is worth considering for important data.

Partial success and bad data

Sometimes only part of a bundle is acceptable. A few items might be malformed, or too old, or rejected by a rule. OTLP allows the receiver to say that the bundle was accepted partially and to report how much was rejected. Well-built senders log this and carry on, rather than resending the whole bundle forever. If you ever see steady partial-rejection messages, that is a hint that something upstream is producing bad data, and it deserves a friendly conversation with the owning team.

A single mailroom clerk cannot serve a thousand employees. Here is how gateways grow, and the traps along the way.

A gateway is only useful if it can keep up with the telemetry arriving. Fortunately, a basic gateway is stateless, which means each replica handles a bundle without needing to remember anything about earlier bundles. Stateless programs are easy to scale: just run more copies and share the traffic among them. Many teams let the platform do this automatically, adding replicas when processor or memory use climbs and removing them when it falls.

The long-lived connection trap

Remember that OTLP over gRPC keeps one connection open for a long time and sends many bundles through it. A simple load balancer shares connections, not bundles. If ten applications each open one connection, and three replicas exist, the connections might be shared evenly at first. But if an application restarts or a replica is added later, the new replica may receive almost nothing while old ones stay busy. The result is lopsided load, even though everything looks balanced on paper.

Scaling signals worth watching

Automatic scaling needs a trigger, and the best triggers are the ones that reflect real strain. Processor use is the simplest: when replicas are working near their limit, add more. Memory use is equally important, because the memory limiter will begin refusing data as it fills. A third, often forgotten signal is the queue: if exporter queues keep growing, the gateway is receiving faster than it can send, and more replicas may help only if the backend can accept the extra traffic. Always check that the backend is not the real bottleneck before adding copies.

When replicas must cooperate

Some jobs are stateful: they need to see related data together. The best example is tail sampling, where the Collector waits until a whole trace has arrived and then decides whether to keep it. If the spans of one trace are spread among different replicas, no replica sees the whole trace, and the decision is wrong. The same problem affects any step that counts or compares related data, such as turning spans into per-service metrics.

The usual cure is a two-tier gateway. The first tier receives data from everywhere and uses a special load-balancing exporter that picks the next collector by looking at the trace ID. All spans with the same trace ID go to the same second-tier collector, which can then make a correct decision.

flowchart LR
    SVC["Services and agents"] --> T1["Tier 1 gateway: load balancing exporter"]
    T1 -->|"hash of trace ID"| T2A["Tier 2 collector A: tail sampling"]
    T1 -->|"hash of trace ID"| T2B["Tier 2 collector B: tail sampling"]
    T1 -->|"hash of trace ID"| T2C["Tier 2 collector C: tail sampling"]
    T2A --> BK["Backend"]
    T2B --> BK
    T2C --> BK

Think of the first tier as a sorting desk that looks at the trace label on each piece of mail and always puts the same label into the same bin. The second tier workers each own a few bins and can see every page of the stories they are responsible for. When a bin’s story is finished, the worker decides whether it is interesting enough to keep.

The real value of a gateway is the work it does in the middle. Here are the jobs that teams hand to it most often.

All of these jobs have one thing in common: they are policy. They express what the organization wants to happen to its telemetry. When policy lives in a gateway, a security team or a platform team can change it in one place, and the change takes effect everywhere without asking dozens of application teams to rebuild and redeploy their programs.

It also helps to think about who owns each rule. A rule that masks personal data belongs to the security team. A rule that drops noisy health checks belongs to the platform team. A rule that adds the owning team as a label belongs to whoever manages the service catalog. Writing the owner next to each rule, and reviewing changes together, prevents the gateway from becoming a mysterious pile of settings that nobody dares to touch. A tidy rulebook is a quiet gift to your future self.

What the gateway cannot know

A gateway sits far from the applications, so it sees only what the data tells it. Details that come from the local machine, such as which Kubernetes pod produced a span, are best added by an agent that runs next to the program. If the data passes through a proxy before reaching the gateway, the gateway may see the proxy’s address instead of the true sender, which breaks any enrichment that depends on the sender’s address. This is one more reason why the agent and gateway combination is so popular: each adds what it is best placed to know.

A central service is a tempting place for failure. Good design makes sure a bad day does not become a lost week of data.

Because every sender depends on the gateway, an unreliable gateway is dangerous. The good news is that the Collector has several protections built in, and they stack on top of each other. Let us look at what happens when the backend is unavailable, which is the most common trouble.

sequenceDiagram
    participant S as Service SDK
    participant G as Gateway Collector
    participant Q as Export queue
    participant B as Backend
    S->>G: 1. Send batch of spans
    G->>Q: 2. Place batch in queue
    G-->>S: 3. Acknowledge receipt
    Q->>B: 4. Try to export
    B--xQ: 5. Backend unavailable
    Note over Q: 6. Wait and retry with growing delay
    Q->>B: 7. Retry export
    B-->>Q: 8. Backend accepts the batch
    Note over Q: 9. If the queue fills, drop data and count it

In steps one to three, the service sends a bundle, the gateway places it in its queue, and the service is told that it was received. In step four the exporter tries to deliver the bundle, but in step five the backend is unavailable. Instead of dropping the bundle, the exporter waits a little and tries again, waiting longer each time, as step six shows. When the backend recovers in steps seven and eight, the waiting bundles are delivered. Step nine is the final safety valve: if the outage is so long that the queue becomes full, the oldest or newest data is dropped, and the drop is counted so you can see it.

Layers of protection

Placing the replicas well

Spread replicas across different availability zones, which are separate buildings or power areas in a cloud region, so one local failure cannot remove every replica. Tell the platform not to stop too many replicas at once during maintenance. Also keep each replica’s memory limit comfortably below the container’s own limit, because if the container is killed for using too much memory, any data in flight is lost without ceremony.

Upgrading without drama

New Collector versions arrive regularly and carry fixes, features, and sometimes changes that require configuration updates. Plan upgrades the way you plan any change to a shared service. Read the release notes, try the new version in a test environment with realistic traffic, then upgrade one replica and watch its health numbers before moving on to the rest. Because several replicas share the work, a rolling upgrade can finish without any visible gap, provided each old replica is allowed to finish what it is doing before it stops.

A gateway sees everything and holds the keys to your backends. That makes it a very valuable target, and worth protecting carefully.

A gateway has two special features that matter for security. It sees the telemetry of the whole organization, which may contain details nobody planned to expose. It also holds the credentials for the backends, so anyone who takes control of it can send or read data as if they were the company. Treat it the way you would treat any important gate: lock it, watch it, and give it only the powers it needs.

Multiple teams, one gateway

In bigger organizations, several teams or even several customers share a gateway. In that case, the gateway can use an identity or a label on each request to decide where the data should go, and to keep one team’s data out of another team’s backend. The more teams share the gateway, the more important clear rules and review become.

Least privilege and an audit trail

Give the gateway only the permissions that it truly needs. It should be able to receive telemetry and write to the backends, and nothing more. It should not be able to read your databases, install software, or reach unrelated systems. Keep a record of who changed the configuration and when, so that if a strange rule appears, you can trace it back to a person and a reason. This habit rarely feels urgent on a quiet day, and it becomes priceless after an incident.

You rely on the gateway to watch everything else. Who watches the gateway?

If a gateway quietly drops data, every dashboard downstream looks calm while problems go unseen. That is far worse than an obvious outage, because nobody knows to look. Fortunately, the Collector publishes internal telemetry about its own health, including metrics with names that begin with otelcol, along with its own logs. You should collect these and alert on them, ideally in a place that does not depend on the gateway you are measuring.

Health checks and diagnostics

The Collector has a health check extension, a tiny web address that answers “am I well?” so the platform can restart a broken replica and avoid sending traffic to one that is not ready. It also offers diagnostic pages that show what is happening inside, such as which spans are currently moving through, which are handy during an investigation. Keep these pages private, because they reveal details about your system.

Alerts that earn their keep

Do not only look at today’s numbers. Watch the trend over weeks. If incoming data grows by ten percent every month, you can predict when your replicas will run short and add capacity in a calm, planned way. If the share of dropped data creeps upward, you can investigate before anyone complains. A gateway that is checked regularly stays small and boring, while a gateway that is only discussed during emergencies tends to surprise everybody at the worst moment.

Every design has a price. An honest view helps you decide whether a gateway is right for you.

Common trade-offs in plain language

Patterns are tried-and-true recipes. Habits are the small routines that keep a gateway calm and boring, which is exactly what you want.

Habits that pay off

• Run at least two replicas. Even a small system should survive the loss of one copy.
• Always include the memory limiter. Make it the first processor, and set its limits below the container’s own limit.
• Batch last. Place the batch processor just before the exporters so bundles contain only surviving data.
• Size from numbers, not hunches. Estimate the spans, metrics, and logs per second, test one replica under load, and plan headroom.
• Turn on the exporter queue and retries. Add a disk-backed queue for data you cannot afford to lose.
• Use a managed way to deploy. Helm charts and the OpenTelemetry Operator for Kubernetes make it easier to run gateways consistently across clusters.
• Roll out changes slowly. Test new rules on one replica or a test environment first. A bad rule at a gateway affects everyone.
• Pin and update versions. Choose a specific Collector version, read the release notes, and plan regular upgrades so security fixes arrive.
• Document who owns what. Platform teams own the gateway, and service teams own their instrumentation. Write it down.

A gentle path to adopting a gateway

Learning from other people’s mistakes is much cheaper than making your own.

An anti-pattern is a habit that looks sensible at first but causes trouble later. These four appear again and again in real gateway deployments.

Myths and realities

Let us size a gateway the way a platform engineer would, using simple arithmetic.

Imagine an online store whose platform runs 120 services. Together they handle about 20,000 customer requests per second at the busiest hour. Each request touches several services and produces about 12 spans on average. The team wants to keep roughly one tenth of the traces after sampling. The numbers below are for illustration only; real systems must be measured.

Reading the table from the top, the gateway must absorb 240,000 spans every second, even though only 24,000 will be stored. This is the important lesson: the gateway pays for the data it receives, not only for the data it keeps. Nine replicas, spread across three zones, give both capacity and protection against a zone failure. The last row shows why queues must be sized deliberately: a one-minute outage already needs room for well over a million spans, so the team must decide how much memory or disk it is willing to spend to hold them.

Real-world scenarios

The worked example hides one more cost that surprises many teams: network traffic between zones. Cloud providers often charge for data that crosses a zone boundary, and a large telemetry stream can add up quickly. If each agent prefers a gateway replica in its own zone, most of the bulk traffic stays local, while the other zones remain available as a backup. That one routing choice can save real money without any loss of safety, and it is a good example of how a thoughtful gateway design pays for itself.

The whole story in a few simple ideas you can carry with you.