Before any tool is chosen, it helps to see the pain that a pipeline is designed to remove.

What observability actually means

Observability is the ability to understand what is happening inside a running system by looking at the data it produces. If a customer says the checkout page is slow, observability lets an engineer find out which service, which database call and which deployment caused the slowdown, without guessing and without logging into individual machines. The data that makes this possible is called telemetry, and it is usually grouped into signals: traces, metrics and logs.

Observability exists because modern systems are made of many small services that call each other over a network. One user click may touch twenty components. No single person can hold the whole picture in their head, so the system itself must tell its story in a structured way.

The world before shared pipelines

In the early days, each team wired its own telemetry. One team used a vendor agent for metrics. Another wrote logs to files and shipped them with a custom script. A third added a tracing library that only worked with one backend. Every application ended up talking directly to one or more backends, which are the systems that store and display telemetry.

This direct wiring seems simple at first, but it creates a tangle. If the company decides to change its metrics vendor, every application must be changed, rebuilt and redeployed. If a backend slows down, applications may block or drop data. If a developer accidentally logs a customer’s email address, it is already sitting in a third-party system before anyone notices. If one service suddenly emits ten times more data, the bill grows, and nobody knows whom to ask.

A short, plain tour of the building blocks that the platform will later organize.

What OpenTelemetry is

OpenTelemetry, often shortened to OTel, is an open-source, vendor-neutral standard for producing, collecting and exporting telemetry. It is hosted by the Cloud Native Computing Foundation, the same foundation that hosts Kubernetes. It provides specifications, language libraries, a wire protocol and a standalone data-handling program called the Collector. It deliberately does not provide a storage backend or a dashboard. Its job is to get good data out of applications and to the place you choose, in a format everyone agrees on.

The signals

The API, the SDK and instrumentation

Inside an application, OpenTelemetry has two layers. The API is a thin set of calls that code uses to say “a request started” or “record this number”. The SDK is the implementation that decides what to do with those calls: how to sample, how to batch, and where to send the result. Splitting them lets library authors add instrumentation using only the API, without forcing any particular behavior on the people who use their libraries.

Instrumentation is the act of adding those calls. It can be automatic, where an agent or library hooks into common frameworks without code changes, or manual, where developers add spans and measurements around business logic. Most real systems mix both.

Resources, attributes and semantic conventions

Every piece of telemetry carries attributes, which are key and value pairs such as the HTTP method or the database name. A special group called the resource describes who produced the data: the service name, the version, the cloud region, the Kubernetes pod. Semantic conventions are the shared naming rules that say which attribute names mean what. They are the reason that two different teams, using two different languages, still produce data that a platform can understand uniformly.

OTLP, the common language

OTLP, the OpenTelemetry Protocol, is the wire format used to move all signals between SDKs, Collectors and compatible backends. It can travel over gRPC or over HTTP. Because most observability products now accept OTLP directly, it has become the default handshake of the ecosystem, and it is the glue that makes a pipeline possible.

The Collector in one paragraph

The OpenTelemetry Collector is a standalone program that receives telemetry, optionally changes it, and sends it onward. It is written in Go, it runs as a single binary or container, and it is configured by a text file that declares what to receive, what to do and where to send. It is the engine room of everything that follows in this tutorial.

flowchart LR
  A["Applications with OTel SDK"] -->|"OTLP"| B["Collector"]
  B --> C["Metrics backend"]
  B --> D["Trace backend"]
  B --> E["Log backend"]
  B --> F["Archive storage"]
        

The phrase is an architectural idea, not a product name, so it is worth defining carefully.

Two ideas joined together

An observability pipeline is the path telemetry takes from where it is produced to where it is stored, including every step of receiving, enriching, filtering, sampling, routing and exporting along the way. A platform, in the engineering sense, is a shared internal service that many teams use through a simple, self-service interface, run by a dedicated team that treats the service like a product with users, documentation, reliability targets and a roadmap.

Put together, an observability pipeline as a platform means one centrally operated, OpenTelemetry-based data path that every application team plugs into with minimal effort, while a small platform team owns its quality, cost, security and uptime. It is important to be honest about vocabulary here: OpenTelemetry does not ship a thing called “the Observability Platform”. The project provides the standard parts, and organizations assemble them into a platform by adding operating practices around them.

The four promises of a good platform

Platform layers

It helps to see the platform as layers. At the bottom are the producers, which are applications and infrastructure carrying the SDK. Above them is the collection layer, made of agents close to the workloads. Above that is the processing layer, made of central gateways where heavier decisions happen. At the top are the destinations. Wrapped around all of these is the control layer: configuration management, access rules, cost reporting and self-monitoring.

flowchart TB
  subgraph Producers
    P1["Services with SDK"]
    P2["Kubernetes and hosts"]
    P3["Third-party systems"]
  end
  subgraph Collection
    A1["Agent Collectors"]
  end
  subgraph Processing
    G1["Gateway Collectors"]
  end
  subgraph Destinations
    D1["Metrics store"]
    D2["Trace store"]
    D3["Log store"]
    D4["Cold archive"]
  end
  subgraph Control
    C1["Config as code"]
    C2["Access and tenancy rules"]
    C3["Cost and quality reports"]
    C4["Pipeline self-monitoring"]
  end
  P1 --> A1
  P2 --> A1
  P3 --> G1
  A1 --> G1
  G1 --> D1
  G1 --> D2
  G1 --> D3
  G1 --> D4
  Control -.-> A1
  Control -.-> G1
        

The Collector is built from a small number of component types that snap together like building bricks.

The five component types

Pipelines are declared per signal

The configuration has a section where pipelines are defined. Each pipeline belongs to exactly one signal type, so there is a traces pipeline, a metrics pipeline and a logs pipeline, and a Collector may run several of each. A pipeline lists which receivers feed it, which processors run and in what order, and which exporters it sends to. A component can be defined once and reused across several pipelines.

Order matters. Processors run one after another in the order listed, so a memory protection step normally comes first, enrichment and filtering come in the middle, and batching comes last, right before data leaves.

The fan-in and fan-out behavior

If two receivers feed one pipeline, their data is merged into one stream. If one pipeline has two exporters, every item is copied to both. This simple rule is what makes dual-writing to two backends during a migration almost trivial, and it is also what makes careless configuration expensive, because every extra exporter multiplies the outgoing traffic.

flowchart LR
  R1["OTLP receiver"] --> M["Memory limiter"]
  R2["Host metrics receiver"] --> M
  M --> E["Enrichment and redaction"]
  E --> F["Filtering"]
  F --> B["Batch"]
  B --> X1["Exporter to primary backend"]
  B --> X2["Exporter to archive"]
  B --> CN["Connector to metrics pipeline"]
        

Useful processors, explained simply

Knowing what happens to a single item inside the Collector explains almost every tuning decision later.

The journey of one batch of data

Back-pressure, the pressure valve

Back-pressure is what happens when a downstream step cannot keep up and the slowness travels backward toward the source. In a well-configured Collector, the memory limiter notices high memory use and starts refusing data, and the refusal is returned to senders as a retryable error. Well-behaved SDKs and upstream Collectors then wait and try again rather than flooding the system.

Queues: memory and disk

By default, the sending queue lives in memory, which is fast but is lost if the process dies. A persistent queue stores waiting data on disk through a storage extension, so that a restart does not lose what was in flight. The trade-off is slower throughput and the need for a reliable volume. Choosing between the two is a decision about how much loss the business can tolerate for a given signal.

sequenceDiagram
  participant App as Application SDK
  participant Agent as Agent Collector
  participant GW as Gateway Collector
  participant BE as Backend
  App->>Agent: Send batch over OTLP
  Agent->>Agent: Memory check, process, batch
  Agent->>GW: Export batch
  GW->>BE: Export batch
  BE-->>GW: Slow or unavailable
  GW->>GW: Hold in queue and retry
  GW-->>Agent: Refuse when full, retryable
  Agent->>Agent: Queue and retry
  Agent-->>App: Refuse when full, retryable
  BE-->>GW: Recovered
  GW->>BE: Drain queue
        

Logs are a special case

Traces and metrics usually arrive pushed by an SDK. Logs often start as plain files written by an application or container runtime. The Collector can read those files, parse them into structured records, and attach the same resource attributes used by traces and metrics. This is one of the most valuable platform features, because it lets old applications participate in the common data model without being rewritten.

Where you place Collectors decides your reliability, your cost and how much each team must know.

The four common shapes

flowchart LR
  subgraph Node1["Kubernetes node A"]
    S1["Service 1"] --> AG1["Agent"]
    S2["Service 2"] --> AG1
  end
  subgraph Node2["Kubernetes node B"]
    S3["Service 3"] --> AG2["Agent"]
  end
  AG1 --> LB["Load balancer"]
  AG2 --> LB
  LB --> GW1["Gateway 1"]
  LB --> GW2["Gateway 2"]
  LB --> GW3["Gateway 3"]
  GW1 --> V1["Vendor backend"]
  GW2 --> V2["Open-source stores"]
  GW3 --> V3["Archive"]
        

Agent flavors on Kubernetes

On Kubernetes, an agent can run as a sidecar, one extra container per pod, or as a node agent that runs once per machine. Sidecars give strong isolation and per-application tuning but multiply resource use. Node agents are cheaper and are the natural place to collect node metrics and container logs. Many platforms use node agents by default and allow sidecars for special workloads.

Comparing the shapes

Tiers beyond two

Very large organizations add more tiers. A regional gateway may forward to a global one, or a message queue may sit between tiers to absorb bursts. Each tier adds resilience and control, but also adds latency and operational effort. A good rule is to add a tier only when a clear problem demands it and to be able to name that problem in one sentence.

A platform earns its name by doing useful jobs for every team, once, in the middle of the road.

Capability 1: Enrichment and normalization

Enrichment means adding missing context to data as it passes through. A trace span from a small service might only say “database call took 300 milliseconds”. After enrichment it also says which Kubernetes namespace, which cloud region, which team owns the service and which version was running. Normalization means making equivalent things look identical, for example turning three spellings of an environment name into one agreed value.

Doing this in the pipeline rather than in each application is powerful because the platform team can fix a naming problem in one place and every service benefits on the next config rollout, with no code change by anyone.

Capability 2: Routing

Routing sends different data to different places. Production traces may go to a premium trace store, while development traces go to a cheaper one. Security-relevant logs may be copied to a compliance archive, while debug logs are kept for only a few days. A routing connector inspects an attribute, such as the tenant or environment, and picks the pipeline that should handle the item.

Routing also enables safe migrations. During a move from one vendor to another, the pipeline can send everything to both for a trial period, compare the results, and then switch off the old one. Applications never notice.

Capability 3: Filtering and volume shaping

Some telemetry is noise. Health-check requests, which orchestrators fire every few seconds, may account for a large fraction of all spans while telling nobody anything. Filtering drops such items according to simple rules. Volume shaping goes further and reduces detail, for example by aggregating a high-detail metric into fewer series, or dropping a verbose log level in production.

Capability 4: Redaction and privacy

Applications sometimes put sensitive information in telemetry by accident: an email address in a URL, a card fragment in a log line, a token in a header attribute. A redaction step in the pipeline can mask, hash or delete values that match patterns or that sit under known attribute names. This acts as a safety net, and it is much cheaper to maintain a single net than to audit every codebase.

Capability 5: Derived signals

Connectors can turn one kind of signal into another. A well-known example converts spans into metrics, producing request counts, error counts and latency distributions for every service and operation automatically. This is valuable because metrics are cheap to store for a long time, while detailed traces are expensive. A platform can therefore keep a complete statistical picture even when it only stores a small fraction of the individual traces.

Another common use is counting. A connector can count log lines or spans that match a rule and emit the count as a metric, which can then drive alerts without anyone storing every line.

Capability 6: Format bridging

Not every system speaks OTLP. Older applications may expose metrics in the Prometheus format, send logs through syslog, or emit traces in older open formats. Receivers for these formats let the pipeline accept them all and convert them into the common internal representation. On the way out, exporters can speak whatever the destination requires. The pipeline thus becomes a translator that protects the organization from format churn.

flowchart LR
  IN["Incoming traces"] --> ENR["Enrich and normalize"]
  ENR --> RED["Redact sensitive values"]
  RED --> RT{"Route by tenant and environment"}
  RT -->|"Production"| P1["Premium trace store"]
  RT -->|"Development"| P2["Low-cost store"]
  RT -->|"Security tagged"| P3["Compliance archive"]
  RED --> SM["Span to metrics connector"]
  SM --> MET["Long-term metrics store"]
        

Traces are the most expensive signal, so deciding which ones to keep is one of the platform’s biggest levers.

Why sample at all

A busy system can generate millions of traces per minute. Storing all of them is usually unaffordable, and most of them are boring successful requests that look identical. Sampling is the practice of keeping only a subset while preserving the ability to answer important questions. The art is keeping the interesting traces, such as errors and slow requests, while discarding most of the ordinary ones.

Head sampling

Head sampling makes the keep-or-drop decision at the very start of a trace, before anything is known about how the request will turn out. It is cheap and simple, because dropped traces are never generated or sent. Its weakness is that it is blind. It cannot know that the request that just started will later fail or take ten seconds, so it may throw away exactly the traces you wanted. Head sampling usually happens in the SDK, and the decision is passed along to downstream services so that a trace is kept or dropped as a whole.

Tail sampling

Tail sampling waits until a trace is finished, or nearly finished, and then decides using everything that happened. It can keep all traces with errors, all traces slower than a threshold, all traces touching a sensitive service, and a small random share of the rest. This is far more useful, but it has a cost: the Collector must hold every span of every in-progress trace in memory until the decision time, and all spans of one trace must arrive at the same Collector instance.

The trace affinity problem

Imagine a trace whose spans are produced by five different services, which send their spans to five different agents, which forward them to a pool of three gateways. If the spans land on different gateways, no single gateway sees the whole trace, and any tail decision made on a partial view will be wrong. The solution is trace-aware load balancing: a first tier of Collectors uses an exporter that hashes the trace identifier and sends every span with that identifier to the same second-tier Collector. The second tier then runs the tail sampling policies with a complete view.

flowchart LR
  A1["Agent 1"] --> LB1["Load-balancing tier"]
  A2["Agent 2"] --> LB1
  A3["Agent 3"] --> LB1
  LB1 -->|"Trace IDs 0 to 33 percent"| T1["Tail sampler A"]
  LB1 -->|"Trace IDs 34 to 66 percent"| T2["Tail sampler B"]
  LB1 -->|"Trace IDs 67 to 100 percent"| T3["Tail sampler C"]
  T1 --> S["Trace store"]
  T2 --> S
  T3 --> S
        

Layering policies

A mature platform combines policies instead of choosing one. A typical pattern keeps every trace that contains an error, keeps every trace slower than a set latency, always keeps traces from a short list of critical business flows, and keeps a small random percentage of everything else so that baselines remain statistically honest. Because sampling discards the raw evidence of the ordinary traces, the derived span metrics from Chapter 7 are what keep rates and percentiles accurate.

A platform that falls over during an incident is worse than no platform, because that is exactly when people need it.

Stateless and stateful parts

Most Collector work is stateless: each item can be handled alone, so you can add more identical instances behind a load balancer and capacity grows roughly in line. Some work is stateful, which means an instance needs memory of earlier items. Tail sampling is stateful because it must remember spans until a trace is complete. Certain metric conversions are stateful too, because they need previous values to compute rates or to merge cumulative counters. Separate stateless from stateful work into different tiers so that scaling rules for each can differ.

Horizontal scaling signals

To scale well, watch the right numbers. The key ones are CPU use, memory use, the depth of sending queues, the count of refused or dropped items, and the delay between data arriving and data leaving. Queue depth rising while CPU stays low usually means a slow destination, not a lack of Collector capacity, so adding more Collectors will not help.

Buffering with a message queue

For high-value data or for very bursty traffic, many platforms put a durable message queue between the gateway tier and the final writers. Gateways publish to the queue, and a separate pool of consumers reads from it and writes to backends at their own pace. This decouples the speed of arrival from the speed of storage. If a backend is down for an hour, the queue holds the data, and when the backend recovers the consumers drain the backlog.

High availability

Run at least two instances of every tier, spread across failure zones. Use health checks so the load balancer removes unhealthy instances. Roll out configuration changes gradually, one slice at a time, so that a bad change affects a small part of the fleet first. For persistent queues, use storage that survives the loss of a single machine.

Protecting the applications

The strongest reliability rule of a telemetry platform is that it must never harm the systems it observes. SDKs should use bounded in-memory buffers and drop data rather than block user requests when the pipeline is slow. Agents should have strict resource limits. Gateways should refuse excess load early rather than accept it and collapse. A monitoring system that causes an outage is a failure in design, not in luck.

When there are hundreds of Collectors, the question changes from “how do I configure one?” to “how do I keep all of them correct?”

The OpenTelemetry Operator

On Kubernetes, the OpenTelemetry Operator is a controller that manages Collectors and instrumentation for you. You describe the Collector you want as a Kubernetes resource, including whether it should run as a sidecar, a node-level agent, a standard deployment or a stateful set, and the Operator creates and maintains the underlying objects. A second resource type lets you describe automatic instrumentation for supported languages, so that pods can be instrumented by adding an annotation rather than rebuilding images.

Scraping at scale with the Target Allocator

Metrics in the Prometheus style are often pulled by scraping endpoints rather than pushed. If many Collectors each scraped everything, work would be duplicated. The Operator’s Target Allocator discovers scrape targets and divides them fairly among the Collector instances, so each target is scraped by exactly one instance, and it rebalances as pods come and go.

Fleet control with OpAMP

OpAMP, the Open Agent Management Protocol, is a standard that lets a central server talk to a fleet of agents. Through it, the server can see which agents are running, which configuration each one has, whether the last change was applied, and what their health is, and it can push new configuration or request updates. It is the foundation for a real control plane because it replaces ad hoc scripts with a common, observable conversation between the platform and its Collectors. Support in tools and distributions is growing but uneven, so verify what your chosen distribution implements.

Configuration as code

Treat pipeline configuration like application code. Keep it in version control, require review for every change, validate it automatically before rollout, and deploy it through the same automated delivery process as other infrastructure. Validation steps can check that the file loads, that required processors exist and are in the right order, that no forbidden exporter is used, and that sample data passes through producing the expected output.

Layered configuration

A useful technique is to build each Collector’s configuration from layers: a base layer owned by the platform team that contains non-negotiable safety items, and optional layers that tenant teams may select or tune within limits. The Collector supports merging several configuration sources, which makes this layering practical. The base layer guarantees memory protection, redaction and self-monitoring; the optional layers add things like extra attribute rules for a particular team.

This chapter is where a pipeline stops being infrastructure and becomes a product that people choose to use.

What a tenant is

A tenant is a group of users and workloads that the platform serves as a unit, such as a business unit, a product team or an environment. Multi-tenancy means many tenants share the same pipeline while being kept apart in the ways that matter: whose data is whose, who may see what, who pays for what, and who may be affected if another tenant misbehaves.

Identifying tenants

The platform needs a reliable tenant marker on every item of data. Common choices are a resource attribute set by the agent, a header on the incoming connection that is tied to a credential, or a naming convention that is verified by the gateway. The key principle is that tenant identity should be established by the platform from trusted information, not claimed freely by the application, otherwise one tenant could pretend to be another.

Isolation and fairness

Cardinality governance

Cardinality is the number of distinct combinations of attribute values for a metric. A metric labelled by user identifier can create millions of separate series, and each series costs storage and query time. High-cardinality mistakes are the most common reason for surprise bills. The platform can limit damage by removing or hashing risky attributes, by capping the number of series per tenant, and by publishing a list of the heaviest offenders so their owners can fix them at the source.

Data contracts and golden paths

A data contract is a written agreement between the platform and its users about what telemetry should look like: required resource attributes, naming rules, allowed attribute keys and size limits. A golden path is the easiest, best-supported way to do something, such as a starter template or a shared library that already has the SDK configured correctly. When the golden path is genuinely easier than the alternatives, most teams follow it voluntarily, and governance becomes a matter of convenience rather than of policing.

Showback and chargeback

Showback reports how much each tenant used, without charging. Chargeback actually bills them. Even showback alone changes behavior, because teams that can see their own volume tend to clean it up. The pipeline is a natural place to count, since all data passes through it before storage.

Telemetry often contains more sensitive information than people realize, so the pipeline is a security boundary and should be designed as one.

Encrypt and authenticate every hop

Every connection between SDK, agent, gateway and backend should be encrypted in transit. Where possible use mutual authentication, in which both sides prove their identity with certificates, so that only legitimate senders can submit data and only legitimate receivers can obtain it. Authentication extensions in the Collector can validate bearer tokens, certificates or other credentials at the receiving edge. Rotate credentials on a schedule and store them in a secrets manager rather than in plain configuration files.

Least privilege and a small attack surface

Run Collectors as unprivileged users, with the smallest set of permissions needed. A node agent that reads container logs needs read access to log paths, not full control of the host. Use custom builds that include only the components you need, because every extra component is more code that could contain a flaw. Expose diagnostic and health endpoints only on internal interfaces.

Privacy engineering

Treat telemetry as potentially personal data. Define which attributes may never leave the region where they were produced, and use routing rules to keep such data inside the right geographic boundary. Mask or hash identifiers where the raw value is not needed. Limit retention periods per data type. Provide a process for deleting data when a legal request requires it, which means knowing where each tenant’s data physically lives.

Auditability

Because the pipeline can change what data looks like, changes to it must be traceable. Keep a history of who changed which rule and when, record which version of configuration each Collector is running, and keep records of access to sensitive archives. When an auditor or an incident investigator asks “what happened to this data on its way here?”, the answer should be available from records, not memory.

The pipeline carries the evidence everyone relies on during an incident, so it needs its own instruments and its own promises.

The Collector reports on itself

The Collector exposes internal telemetry about its own behavior. This includes how many items each receiver accepted, how many each processor dropped, how many each exporter sent or failed to send, how full the sending queues are, and how much memory and CPU the process is using. These internal signals can be sent to a separate, simple, highly reliable path, deliberately different from the main one, so that you can still see problems with the pipeline when the pipeline itself is struggling.

The numbers worth watching

Service level objectives for a pipeline

A service level objective, or SLO, is a measurable promise about quality. For a pipeline, useful SLOs include the share of telemetry that is delivered rather than dropped, the delay between generation and availability for querying, and the availability of the ingestion endpoint. Publishing these numbers lets tenants plan around them and tells the platform team when to invest in reliability instead of features.

Canary data and end-to-end checks

Counters can look healthy while data is silently wrong. A strong technique is to inject synthetic telemetry with known content at the edge, then verify that exactly that content arrives at each destination within the expected time. This tests the whole path, including processors and routing, and catches subtle breakage that internal counters cannot see.

Patterns are shortcuts to decisions other teams already paid to learn; anti-patterns are the bills they paid.

Patterns that work

Anti-patterns to avoid

No architecture is free. A clear view of both sides helps decide whether, and how far, to go.

Build, buy or blend

Some organizations assemble the platform entirely from open-source parts. Others buy a managed pipeline product, often built on or compatible with OpenTelemetry. Many choose a blend: open-source Collectors at the edge, a vendor or managed service for the hard central pieces. The right choice depends on team size, appetite for operating infrastructure, regulatory needs and how strongly the organization values independence from any one supplier.

The following are composite scenarios, built from common patterns reported across the industry, to show how the ideas combine in practice.

Nobody builds the whole platform at once. A staged path keeps each step useful and affordable.

flowchart LR
  S1["Stage 1 Standard data"] --> S2["Stage 2 Agents"]
  S2 --> S3["Stage 3 Gateway"]
  S3 --> S4["Stage 4 Shape and sample"]
  S4 --> S5["Stage 5 Govern and automate"]
  S5 --> S6["Stage 6 Optimize and extend"]
        

A short checklist of habits that separate calm platforms from chaotic ones.

Upgrades and stability levels

OpenTelemetry components carry stability labels that range from experimental to stable. A platform should prefer stable components for the critical path and confine experimental ones to limited, well-watched areas. Because individual component names and settings sometimes change between releases, read the release notes before each upgrade, test in a staging environment that carries realistic traffic, and upgrade the fleet gradually. The pace of change in the project is a strength, but it requires discipline from anyone who depends on it.

Capacity planning in plain terms

Estimate how many items per second arrive at each tier at normal and peak times, how large each item is on average, how long you want to survive a destination outage, and how much headroom you want. From those four numbers you can size memory for queues, disk for persistent buffers and instance counts for each tier. Revisit the estimates after every large product launch, because traffic and telemetry volume rarely grow at the same rate.

Quick answers to the questions that usually come up first.

No. It is an architectural and organizational approach. OpenTelemetry provides the standard building blocks, such as the specification, OTLP, SDKs and the Collector. The platform is what an organization creates by combining those blocks with shared operations, governance and a dedicated team.

Not for small experiments, where an SDK can send straight to a backend. As soon as you need central policy, several backends, reliable buffering, privacy controls or cost control, a Collector becomes the natural place to provide them.

An agent runs close to workloads, on the same node or inside the same pod, and handles local jobs such as buffering and host data. A gateway is a central pool that receives from many agents and applies shared policy such as routing, sampling and multi-backend export.

Because all spans of one trace must reach the same sampler. A plain load balancer spreads spans across instances, so each sees only part of each trace. A trace-aware balancing tier hashes by trace identifier to keep traces together.

It should not. SDKs buffer in memory and send in the background, and agents on the same node add very little delay. The design rule is that if the pipeline is overloaded, data is delayed or dropped rather than user requests being blocked.

Use sending queues with sensible sizes, consider persistent queues on disk for important signals, and for demanding cases place a durable message queue between tiers. Also decide in advance which data may be dropped first, since no buffer is infinite.

It is an excellent safety net but not a complete defense. Combine pipeline redaction with developer guidance, code review, tests and periodic checks of real stored data to confirm that rules actually catch what they should.

It depends on the scale and ambition. A small team can run a modest agent-and-gateway design with strong automation. Large multi-tenant, multi-region platforms need more people for reliability, support and governance. Automation, templates and self-service are what let a small team serve many.

Often yes. The Collector has receivers for many older formats and exporters for many destinations, so it can translate between them. This lets you adopt the common pipeline first and migrate application instrumentation gradually.

Look at both sides. Technically, track delivery rate, drop counts, delay and availability against published objectives. For users, track how long a new service takes to get its first useful trace, how often teams bypass the platform and how satisfied they say they are.

Component stability levels, the status of newer signals such as profiles, support for fleet management and configuration features all evolve quickly. Treat the official OpenTelemetry documentation and release notes as the source of truth for current names, settings and maturity before you commit to a design.

An observability pipeline as a platform turns telemetry from a private chore of each team into a shared, reliable service. OpenTelemetry supplies the common language and the Collector that moves and shapes data, and the platform adds the human side: a team that owns quality, cost, security and uptime, and an experience that makes the right thing the easy thing. Like the postal service in the opening picture, its value is not in any single letter but in the fact that every letter, from every household, can travel by the same trusted route, be tracked and arrive where it should.

The journey is incremental. Start by standardizing how data is produced, add agents for local safety and context, introduce a gateway for central policy and flexibility, then layer on sampling, governance and automation as real needs appear. At each step, measure the benefit and watch the operating cost, and always make sure the pipeline observes itself.