To understand a custom version of something, first remember what the something does.

Applications constantly produce facts about themselves: how long a request took, how much memory they use, and what errors occurred. These facts are called telemetry, and they come in three main kinds. Traces follow one request through many services. Metrics are numbers measured over time. Logs are text records of events. The OpenTelemetry Collector is a standalone program that receives this telemetry, optionally changes it, and sends it on to the tools where people study it.

The Collector is built from small, swappable pieces called components. There are five kinds, and you will meet these words in every section below.

Here is a key point that surprises many beginners. A component is not something you turn on at run time from nothing. The settings file can only use components that were already included inside the program when it was built. If a component is not inside the program, no setting can summon it. This is the heart of why distributions exist, and the next chapter explains it fully.

flowchart LR
  A["Applications and infrastructure"] --> R["Receivers"]
  R --> P["Processors"]
  P --> E["Exporters"]
  E --> B["Backends and other Collectors"]
  X["Extensions: health, auth, storage"] -.-> R
  X -.-> E
    

The solid arrows show telemetry moving left to right. The dotted lines show extensions helping receivers and exporters without ever touching the data itself. Every distribution, official or custom, has this same shape. What differs is which concrete parts are installed in each box.

The word sounds fancy, but the idea is as simple as a packaged product.

A distribution is a particular, ready-to-run build of the Collector that bundles a chosen set of components, a version number, and a name. When you download “the Collector,” you are really downloading one specific distribution. It is a single program file (and usually a container image) that contains the Collector’s engine plus every component its maker decided to include.

The Collector is written in a programming language called Go. Go programs are compiled into one self-contained file. That is convenient, because you can copy one file to a server and run it. But it also means that the list of components is fixed at the moment of compiling. Unlike some software with plug-in folders where you drop in extra parts later, the Collector has no plug-in folder. What you compile in is what you get.

The three ingredients

• The engine. The core code that starts the program, reads settings, wires components together, moves data between them, and shuts down cleanly. Every distribution shares this common engine.
• The component set. The specific receivers, processors, exporters, extensions, and connectors included. This is the part that varies.
• The identity. A name, description, and version, so that people and tools can tell which build is running. Your custom build can carry your own name instead of the community’s.

Distribution versus configuration

Beginners often confuse two different things. The distribution decides what the program can do. The configuration, a settings file read at start-up, decides what the program actually does right now. You can have one distribution with many different configurations, such as an agent setup on every node and a gateway setup in a central pool, all running the same program file with different settings. You change behavior by editing configuration. You change abilities by building a new distribution.

Before building your own, know what is already on the shelf. Many teams find that a ready-made one is enough.

The OpenTelemetry community publishes several official distributions, and several companies publish their own. Names and exact contents change over time, so always check the current documentation, but the families below have been stable for a long time.

Why not just use the buffet?

The contrib distribution is wonderful for exploration because almost any component you can name is in it. For production, though, it has real downsides. It is large, so container images are bigger and slower to pull. It contains many components you will never run, and each unused component is still code that exists in your environment and can contain bugs or security weaknesses. It also changes often, and an upgrade can bring behavior changes in parts you do not even use. These downsides are modest for a small team but matter a lot for a large company running the Collector on thousands of machines.

Questions to ask about any ready-made distribution

Whether you are considering the official builds or a vendor’s, a handful of questions reveal how well it fits. Which exact components does it contain, and is the list published for every release? How quickly does it follow upstream releases, especially security fixes? Are the components it includes stable for the signals you plan to send? What support is promised, and for how long does each version receive fixes? Can you add one extra component if you need it, or are you locked out? Does it include proprietary parts that tie you to one provider? Honest answers tell you whether the shelf already holds the right product, or whether the time has come to build your own.

Teams often discover that the answers are good enough for years, and that is a perfectly fine outcome. The purpose of asking is not to justify building, but to make a clear, informed choice that you can explain to your colleagues and revisit when your needs change.

Teams reach for a custom distribution for a handful of repeatable reasons.

Reason one: a smaller footprint

A build with only the components you use produces a smaller file and a smaller container image. Smaller files download faster when thousands of machines pull them at once, start faster when a machine restarts, and use less memory because less code is loaded. In a design where an agent runs on every node of a very large cluster, even small savings per copy add up to real money across the fleet.

Reason two: a smaller attack surface

Every piece of code in your program is a potential place for a security flaw. If a serious weakness is announced in a component you never included, your build is simply not affected, and your security team does not need to rush a patch. Fewer components also means fewer third-party code libraries pulled in behind them, which shrinks the list of things you must monitor for vulnerabilities.

Reason three: your own components

Sometimes no existing component does what you need. Perhaps your company has an in-house storage system with its own data format, a special login mechanism, or a private rule for scrubbing data. You can write a new component and include it in your build. The only way to ship it is inside a distribution.

Reason four: predictability and control

With a custom build, you decide exactly when each component version changes. You can freeze a version that works, apply a fix to a component before the community releases it, or hold back an update that you have not tested. This predictability is valuable in regulated industries and in large organizations with strict change processes.

Reason five: consistency and identity

A single, company-wide distribution with a standard name gives every team the same tested base. Support staff know what it contains, security staff know what to scan, and everyone talks about the same thing. It can also carry your own version numbers, so an incident report can say “version 2.4 of our telemetry agent” rather than a vague community version.

Reasons not to build your own

Honesty matters here. Building your own means you now own a small software supply chain. You must keep it updated, test it, scan it, and publish it. For a team with a handful of services and no special needs, that effort may cost more than it saves. If you run only a few Collectors, a ready-made distribution that includes your components is often the wiser choice. Chapter 17 compares both paths in detail.

Let us open the program and see how the engine finds and starts its components.

Every component in the Collector comes with a small helper called a factory. A factory is like a recipe card plus a cook: it knows the component’s name, its default settings, and how to create a working copy of it when asked. The engine does not know in advance what components exist. Instead, at start-up, it is handed a registry, a directory of all factories that were compiled in. When your settings file says “use the batch processor,” the engine looks in the registry for a factory called batch, asks it to create one, and plugs it into the pipeline.

flowchart TB
  CFG["Settings file: which components to use"] --> ENG["Collector engine"]
  REG["Registry of compiled-in factories"] --> ENG
  ENG --> C1["Create receivers"]
  ENG --> C2["Create processors"]
  ENG --> C3["Create exporters"]
  C1 --> PL["Running pipelines"]
  C2 --> PL
  C3 --> PL
    

The diagram makes the key relationship clear. The settings file is only half of the input. The registry is the other half, and it is decided at build time. A custom distribution is, at its core, a custom registry. Building your own means deciding which factories go into that directory.

What gets generated

When you build a distribution with the official builder tool, it generates a small program whose main job is to assemble this registry. It writes the code that lists each chosen factory, sets the name and version, and wires everything to the engine. It then compiles that program into the final binary. You rarely need to read this generated code, but knowing it exists explains why builds need a compiler and why a build is a real software build, not just a settings change.

What else is in the box

• Dependencies. Each component pulls in other code libraries it needs, such as client libraries for a database or messaging system. These come along, and their sizes and security histories become part of your build.
• Built-in commands. The binary can validate a settings file without starting, and can list the components it contains. These are handy for automated checks and for confirming what a given binary can do.
• Internal telemetry. The Collector reports about itself, including its own version and name, which is how you identify your custom build in dashboards.
• Version metadata. The build records which version of the engine and each component it contains, so you can answer “what exactly is running?” later.

Not all components are equal. Knowing where one comes from tells you how much to trust it.

Components come from three kinds of places, and each has a different level of care behind it.

Stability levels

Each component declares a stability level for each signal it supports. This label tells you how safely you can depend on it. Terms used by the project include development (very early and likely to change), alpha (usable but may change in breaking ways), beta (mostly settled, small changes still possible), and stable (changes will be careful and well-announced). Components can also be marked deprecated or unmaintained, which means you should plan to move away from them.

Making choices from stability labels

When you select components for a production distribution, read the stability level of each one and the signals it applies to. A processor can be stable for traces but only alpha for logs. If your pipeline sends logs through it, the alpha label is the one that matters. Write down your reason whenever you include a component below beta, so that future maintainers understand the risk you accepted.

Why this matters for your build

In a custom distribution you are the editor of this list. The official contrib build includes everything regardless of maturity, so you inherit risk you may not have looked at. A custom build forces you to choose each component deliberately, and that act of choosing is itself a quality improvement. Teams often discover that their daily needs are served by fewer than twenty components, most of them stable or beta.

A single official tool turns a shopping list of components into a finished program.

The project provides a command-line tool called the OpenTelemetry Collector Builder, often shortened to ocb. You give it a manifest, a short file describing what you want, and it produces a custom Collector. The manifest is written in a simple settings format (YAML), and it is different from the Collector’s own settings file. Do not mix them up: the manifest says what is built into the program, and the Collector settings say how the built program behaves.

What the builder needs

• The Go toolchain. Because the output is compiled from source, the machine doing the build needs a Go compiler of a suitable version. Many teams run builds inside a container so that the compiler version is identical every time.
• Network access to code sources. The builder downloads the component code from public module sources, unless you have a private mirror. In locked-down environments, a company-run mirror of these sources is common.
• The manifest. The one file that describes your choice.

What the builder produces

The builder produces two things. First, a folder of generated source code containing the registry of your chosen components. Second, unless you tell it to skip compiling, the finished binary program. Teams commonly keep the generated source folder in version control for review, or regenerate it freshly in an automated pipeline. Both approaches work. Regenerating each time avoids stale files, while storing the generated code makes changes easy to inspect.

Can I build by hand instead?

Yes, in principle. A developer could write the registry code and the program entry point manually. Some very advanced teams do, particularly when they embed the Collector inside a larger product. But for most, the builder is the right tool: it is maintained by the project, it follows the conventions of official distributions, and it keeps up with changes to the engine’s internal structure so you do not have to.

Where the builder runs

Most teams run the builder inside their automated pipeline, in a fresh container that already holds the right compiler version. That way, nobody depends on whatever happens to be installed on a personal laptop, and every build starts from the same clean state. Developers can still run the same container locally to try a manifest change quickly before proposing it for review. Keeping the local and pipeline environments identical removes the familiar excuse that a build works on one machine and fails on another, and it shortens the time from an idea to a tested release candidate considerably.

Follow a custom Collector from an idea to a running program.

flowchart LR
  M["Manifest: your chosen components"] --> B["Builder tool"]
  B --> G["Generated source code"]
  G --> C["Go compiler"]
  D["Component code and libraries"] --> C
  C --> BIN["Custom Collector binary"]
  BIN --> IMG["Container image or package"]
  IMG --> FLEET["Running fleet"]
    

Reproducibility: the same input, the same output

A good build is reproducible, which means building from the same manifest and the same component versions today and next month yields the same program. To get this, pin exact versions for every component, pin the compiler version, and keep the dependency list file that Go generates. Without pinning, a build that worked on Monday may break or change behavior on Tuesday because some dependency published a new version overnight.

A manifest has only a few sections. Learn their purpose and you can read any of them.

Although we avoid code listings in this guide, it helps to understand the manifest’s sections in words. Think of it as a form with clearly labeled boxes.

Two powerful extras

The manifest can also carry replacement rules, which tell the build to use a different copy of some code than the default. For example, you can point to a patched copy of a component that fixes a bug the community has not yet released, or to a local folder while developing a new component. It can also carry exclusion rules, which forbid a specific version of a library, useful when you know that version has a security problem.

Matching versions

The engine, the core components, and the contrib components are released together in numbered steps, and their version numbers move in lockstep. If you mix a component from one release with an engine from a very different release, the build may fail because the internal interfaces no longer match. The simplest rule is to keep every component in your manifest on the same release line as the engine version, and to move them together.

The quality of a custom distribution is the quality of the choices you make here.

Start from your real pipelines, not from a catalog. Look at the settings files you run today, or the ones you plan to run. Every receiver, processor, exporter, extension, and connector named in them must be in the distribution. Then add any you know you will need soon, but resist the temptation to add “just in case” extras.

flowchart TB
  S["Start: list the pipelines you run"] --> Q1{"Is every needed component in an official build?"}
  Q1 -->|"Yes, and size is fine"| OFF["Use the official distribution"]
  Q1 -->|"Yes, but too big or too broad"| CUS["Build a custom distribution from existing components"]
  Q1 -->|"No, something is missing"| OWN["Build custom and write your own component"]
    

A method for picking

1. Inventory. Write each pipeline in plain words: “Receive from applications, protect memory, add labels, batch, send to backend and to Kafka.”
2. Map to components. For each step, find the matching component and note its stability level for your signal.
3. Prefer core and stable. If two components can do the same job, prefer the one with the higher stability level and the more active maintenance.
4. Add the safety trio. Nearly every pipeline benefits from a memory limiter, a batch processor, and a health check extension. Do not forget them.
5. Add debugging helpers. A simple exporter that prints data to the log is invaluable when something goes wrong, and costs almost nothing in size.
6. Review with others. Have a teammate challenge each inclusion: who uses this, and what happens if we remove it?

Example shapes for different roles

One distribution or several?

You might think the role table suggests building a different distribution for each role. Sometimes that is right, and sometimes a single company-wide build covering all roles is simpler. Consider the trade-off. A single build is easier to maintain, scan, and explain, but each role carries unused components. Several role-specific builds are smaller and tighter, but you must build, test, and release each one. A common compromise is two or three builds: a lean agent build, a fuller gateway build, and perhaps a special build for stateful tiers.

A closer look at the common component families

It helps to know the typical families of components so that you recognize what to look for in each box of the pipeline. The names below describe jobs, not exact component names, since the catalog changes with every release.

Questions that prevent bloat

Every proposed component should survive a few honest questions. Which pipeline uses it today, and who owns that pipeline? If we removed it, what exactly would stop working? Does another included component already do the same job? How large is it, and which libraries does it bring along? Is its stability level acceptable for the signal we send through it? These questions take minutes to ask and save months of carrying dead weight.

Think in terms of pipelines, not components

A final habit helps greatly. Do not think of components as an unordered pile. Think of complete pipelines: where data enters, which steps change it, and where it leaves. A receiver with no matching processors and no exporter is pointless, and an exporter that nothing feeds is dead weight. Drawing each pipeline on a simple diagram, then circling the components it needs, naturally produces a clean and complete list. It also reveals gaps, such as a pipeline with no memory limiter, before they turn into incidents.

When nothing on the shelf fits, you can craft a new part and snap it into your distribution.

Custom components are the most powerful reason to have a custom distribution. A team may need to receive data from an in-house system that speaks a private protocol, redact data using company-specific rules, or export to a storage system that no public component supports. Writing a component is a software development task done in Go, and this guide does not teach programming. Instead, it explains the concepts so you can plan and judge such work.

The pieces of a component

• A factory. The recipe-and-cook from Chapter 5. It announces the component’s type name, its default settings, and a way to create a live copy. Without a factory, the engine cannot find it.
• A configuration shape. A definition of which settings the component accepts, with default values and validation rules, so that a mistaken setting is caught at start-up with a clear message instead of failing mysteriously later.
• A lifecycle. The component must start and stop cleanly. Starting opens connections or begins listening. Stopping flushes what it holds and releases resources. Poor shutdown behavior is a leading cause of lost data.
• The data handling. A receiver turns outside data into the Collector’s internal format and passes it on. A processor accepts internal-format data, changes it, and passes it on. An exporter takes internal-format data and sends it out, handling retries and errors.
• Self-reporting. Good components report their own metrics, such as how much they accepted, refused, or failed, so operators can see their health.

Helpful project tooling

The project provides helper libraries and a code generator for component metadata. Using the standard helpers means your component inherits well-tested behavior for things like retry, queueing, and self-reporting, rather than reinventing them. It also keeps your component consistent with community components, so operators find familiar settings and metrics.

Where should the code live?

Keep your component in your own code repository with its own version numbers, tests, and documentation. Reference it from the manifest like any other component. If it is generally useful and not company-secret, consider contributing it to the community collection later, since shared components get shared maintenance. Avoid copying a contrib component into your own repository and editing it, because you will lose the ability to take upstream fixes.

Questions to ask before writing one

1. Can an existing component, perhaps combined with a general-purpose transformation processor, do the job? Often the answer is yes.
2. Who will own and maintain this component for years?
3. How will it be tested, including failure cases such as the destination being down?
4. How will it keep up when the engine’s internal interfaces change in later releases?
5. What happens to data if it fails: retry, drop, or hold?

A distribution also decides how the program finds and prepares its settings.

When the Collector starts, it must load its settings from somewhere. The places it can look are called providers, and which providers exist is again fixed when you build. Typical providers read from a file on disk, from an environment variable, from a web address, or from the contents of a string passed directly. A custom distribution can include only the providers you trust. For example, a locked-down build might allow files and environment variables but refuse to fetch settings from the web, closing a path an attacker could abuse.

There are also converters, small helpers that adjust loaded settings, such as expanding values or applying small changes before the settings are checked. Together, providers and converters make your distribution’s start-up behavior predictable.

Why this matters for fleets

In a large fleet, settings are rarely hand-written on each machine. They are produced by templates, stored in version control, and delivered by automation. The providers you build in determine which delivery methods work. Choose them to match your delivery system, and keep the list short. A smaller list of ways to load settings is a smaller list of ways to attack the program.

Secrets and sensitive settings

Passwords and tokens should never be pasted directly into settings files. Instead, the settings file refers to an environment variable or a secret store, and the platform fills it in securely at run time. Make sure your distribution includes the provider that reads from the place where your platform delivers secrets, and test that a missing secret leads to a clear failure message rather than silent misbehavior.

A binary on a developer’s laptop helps nobody. It must become something your fleet can pull, verify, and run.

Once the compiler produces the program file, you wrap it for delivery. The two most common wrappers are container images and operating system packages. A container image is a sealed bundle holding the program and its minimal surroundings, runnable by any container platform. An operating system package is the traditional installable file for a Linux distribution, which a package manager can install and update. Teams running Kubernetes use images. Teams running plain servers often use packages, and many produce both from the same build.

flowchart LR
  SRC["Manifest in version control"] --> CI["Automated build pipeline"]
  CI --> T["Tests and scans"]
  T --> ART["Binary, image, packages"]
  ART --> SIGN["Sign and record contents"]
  SIGN --> REG["Private registry"]
  REG --> STG["Staging fleet"]
  STG --> PROD["Production fleet"]
    

Every step in the diagram has a purpose. The manifest in version control is the single source of truth. The automated pipeline builds the same way every time, with no human typing commands on a laptop. Tests and scans catch problems before release. Signing proves the artifact came from your pipeline. The registry stores approved builds. Staging gives you a rehearsal before production.

Making a lean container image

Because the Collector compiles into one self-contained file, the container around it can be extremely small. Many teams use an almost empty base image, containing little more than the program, security certificates for encrypted connections, and time zone data. A small base has few packages, which means few things to scan, few things to patch, and a faster pull. Run the program as a non-privileged user, and mark the file system read-only where the platform allows.

Many chips, one release

Modern fleets often mix processor types, such as common server chips and ARM-based chips that cost less in some clouds. Build for each target and publish a combined image that lets each machine pull the right one automatically. Test on every target, since subtle differences, though rare, do exist.

Labels and identity

• Give the distribution a clear name that is not confusable with the official one, so nobody reports a bug in the community project that actually belongs in your build.
• Record the engine version and your own release number inside the image labels and in the program’s reported version.
• Attach a bill of materials, a list of every component and library inside the build, so security tools and auditors can inspect it.
• Keep release notes that say which components changed, and why.

A build that compiles is not a build that works. Tests turn hope into evidence.

Because the Collector sits on the path of all your telemetry, a faulty build can blind your whole organization. Testing a distribution has several layers, each catching a different kind of problem. Automate them so that every new build passes through the same gates.

Test the components you wrote more deeply

Community components are tested by their maintainers and many users. Your own components are tested only by you. Give them thorough tests on their own, including strange inputs, slow or broken destinations, large payloads, and shutdown during activity. Include tests that run against the engine version you target, and rerun them whenever you upgrade.

Building a distribution once is easy. Keeping it healthy for years is the real job.

The Collector project releases new versions frequently, and each release may add components, improve performance, fix security problems, change behavior, or remove old features. Your distribution must follow along at a sensible pace. Falling far behind makes each later upgrade bigger and riskier, and leaves known security flaws unfixed.

A calm upgrade routine

1. Watch for releases. Use automated dependency tools that open a change proposal whenever a newer version of a component or the engine appears.
2. Read the release notes. Focus on breaking changes, deprecations, and changes to the specific components you include. Because your build contains only some components, most release notes will not apply to you, and that is one of the benefits.
3. Upgrade together. Move the engine, the builder, and all components to the same release line in one change.
4. Run the gates. Run the full test suite from Chapter 14, including your configuration validation against real settings.
5. Canary and roll out. Release gradually, compare health metrics, and keep the previous build ready for fast rollback.
6. Record the result. Update release notes and the bill of materials so everyone knows what changed.

Handling breaking changes

Occasionally a component renames a setting, changes a default, or retires a feature. The project usually announces this ahead of time through deprecation notices, and some changes can be controlled by feature gates, switches that let you opt into or out of new behavior temporarily. Use feature gates to test new behavior early and to delay painful changes while you adapt, but do not rely on them forever, since gates are meant to be temporary.

Your own numbering

Give your distribution its own version numbers that tell your users what matters to them. A common pattern is to include the engine version it is based on plus a counter for your own revisions, so that “based on engine version X, revision 3” tells a story at a glance. Whatever scheme you choose, keep it consistent and documented.

Support windows

Decide in advance how long you support each release. Without a clear rule, old versions linger across the fleet, nobody knows which ones are safe, and emergencies force chaotic upgrades. A policy such as “we support the newest two release lines and retire older ones after a set period” keeps the fleet healthy.

When you build your own program, you become responsible for what goes into it and for proving what came out.

A software supply chain is the path code travels from its authors to your running systems: component code, the libraries they use, the build tools, the build machine, and the delivery system. An attacker who tampers with any link can influence your program. Because the Collector handles all your telemetry, which may include sensitive information, and often runs with access to the machines it monitors, its supply chain deserves care.

Reducing the attack surface

The most direct security benefit of a custom build is exclusion. Every component you leave out is code that cannot be attacked. A build of fifteen components has far fewer libraries underneath than a build of hundreds, so a vulnerability announcement is much more likely to be irrelevant to you. Also leave out powerful but risky features you do not need, such as the ability to fetch settings from arbitrary web addresses or to run debugging pages on open ports.

Protecting the build itself

• Pin versions and checksums. Go records the exact version and a fingerprint of every dependency. Keep those records in version control, so a changed or substituted library is detected.
• Use a private mirror. A company-run copy of the public code sources gives you a cache that keeps builds working if a public source disappears and lets you scan or block specific libraries.
• Build in clean, controlled environments. Automated pipelines running in fresh containers are harder to tamper with than a developer laptop with unknown tools installed.
• Limit who can change the manifest. Treat the manifest like production code. Require review and approval for any change, since adding a component changes what code runs on every machine.

Proving what you shipped

Produce a bill of materials, a complete list of components and libraries, for every release. Digitally sign images and binaries so that machines in your fleet can check that what they are about to run came from your pipeline and was not changed on the way. Some platforms can be set to refuse any image that is not signed by your key. Keep the signing keys in a protected store with very limited access.

Watching for vulnerabilities

New weaknesses are announced constantly. Scan every release, and also re-scan old releases still in use, since a build that was clean last month may be affected by something announced today. Because your bill of materials is precise, you can quickly answer the question every security team asks during an incident: “Are we affected?”

Runtime hardening

Combine the secure build with a secure setup. Run as a non-privileged user, use a read-only file system, limit network access to the minimum, encrypt links between components, and restrict who can send data to receivers. A hardened build running with broad permissions still has a large exposure.

Neither path is always right. Compare them honestly against your own situation.

A good test is to ask who would be paged if a build broke at night. If nobody would, the organization is not ready to own a distribution. If you have a platform team, a pipeline, and a reason, the investment usually pays back through fewer surprises, smaller costs, and stronger security.

The payoff of a custom build shows up in numbers, and the discipline of running it shows up in operations.

Where the savings come from

A focused build is smaller because less code is compiled in. It starts faster because less needs to be set up at launch. It may use somewhat less memory because fewer components register themselves and fewer libraries are loaded. The savings are not guaranteed to be dramatic for every setup, so measure them. Build both a custom and an official version, run them with the same settings and the same data, and compare image size, start-up time, memory at idle, and memory under load. Use your own measurements to justify the work, rather than assuming.

Why the fleet size matters

If you run ten Collectors, saving a small amount of memory each hardly matters. If you run an agent on twenty thousand nodes, saving the same small amount multiplies into significant computing cost and noticeably faster fleet-wide rollouts. This is the reason large organizations are the most common builders of custom distributions.

Operating the fleet

• Manage settings as code. Store settings in version control and deliver them through automation. Validate them with the built-in command against the exact distribution version they will run on.
• Report what is running. The program reports its own name and version in its internal metrics. Use those to build a dashboard showing how many copies run each version in each environment.
• Roll out in rings. Release first to internal test systems, then to a small production slice, then wider. Pause automatically if health metrics worsen.
• Consider remote management. A newer open standard called OpAMP lets a management server check agent status and push settings and updates to a fleet. It is useful when you must manage very many Collectors, though it adds its own components and security needs.
• Watch resource use after each release. Compare CPU, memory, refused data, and failed exports between old and new versions during rollout.

Connecting to scaling patterns

Custom distributions fit naturally with scaling architectures. A lean agent build runs on every node. A gateway build adds the redaction and authentication components your policy needs. A sampling-tier build carries the memory-hungry tail sampling processor and little else. Each tier gets just what it needs, which makes each tier cheaper to run and easier to reason about.

The same few errors appear again and again. Learn them now and skip the painful lessons.

More mistakes worth avoiding

• Forgetting the essentials. Leaving the memory limiter, batch processor, or health check out of the build, then finding that settings cannot use them.
• Naming confusion. Calling your build by the community’s name, which misleads support staff and users about what it contains.
• No owner. A distribution with no named owning team decays quickly. Assign responsibility from day one.
• Overloading one build. Putting stateful, heavy components into a build used by every node agent, so every node carries weight it never uses.
• Untested custom components. Shipping a homemade component without failure tests, then discovering its shutdown loses data during the first deployment.
• Ignoring stability labels. Including development-stage components in critical pipelines without recording the risk.
• Mismatched release lines. Mixing components from very different releases, causing build failures and odd run-time behavior.
• Floating image tags. Deploying a moving “latest” label and losing track of which version runs where.

Best practices in one place

Four realistic illustrations show how teams use custom distributions. They are composites, not accounts of specific companies.

A back-of-the-envelope savings estimate

Suppose each node’s agent in the official build uses about a hundred and fifty megabytes of memory at idle, and a lean custom build uses about a hundred and ten. That is forty megabytes saved per node. Across ten thousand nodes, the saving is about four hundred gigabytes of memory across the fleet, which may be the equivalent of several large servers’ worth of capacity. Add smaller images and faster rollouts, and the cumulative effect becomes meaningful. The figures here are purely illustrative. The method is real: measure both builds under the same settings, multiply the difference by fleet size, and compare against the cost of owning the pipeline.

A distribution is a product. Products need owners, rules, and a way to say yes or no to change.

Technical steps are only half of success. The other half is people and process. Without clear ownership and rules, a custom distribution turns into a dumping ground, where each team adds its favorite components until the build is as bloated as the one you started with.

Define roles

• Owner. A named team responsible for the pipeline, the release schedule, and the support promise.
• Reviewers. People with security and operations knowledge who must approve any change to the component list.
• Component sponsors. For each included component, a person or team that depends on it and answers questions about it.
• Users. Application and platform teams who consume the distribution and give feedback.

A simple request process for new components

1. The requester explains the need and why existing components do not meet it.
2. The owner checks stability level, maintenance activity, security record, and dependencies the component would bring in.
3. A security reviewer approves or asks for changes.
4. The component is added to a test build, tried in staging, and measured for size and resource effect.
5. It is released in a normal release, announced in the release notes, and its sponsor is recorded.

A regular removal process

Just as important as adding is removing. Once or twice a year, review the list. For each component, check whether any settings file still uses it. Anything unused for a long time becomes a candidate for removal, announced ahead of time so nobody is surprised. This keeps the distribution lean as needs change.

Documentation people actually read

Publish a short guide listing what the distribution contains, how to request changes, how versions are numbered, how long each version is supported, how to validate settings, and where to ask for help. Keep a changelog in plain language: what was added, removed, upgraded, and why. When something goes wrong, good documentation turns a night of confusion into a few minutes of checking.

Measuring success

Decide what a good distribution looks like and measure it. Examples include image size, start-up time, time from upstream release to your release, percentage of the fleet on supported versions, number of vulnerability alerts that applied to your build, and number of incidents traced to the Collector itself. These numbers show whether the effort is paying off and where to improve next.

Let us walk through the whole decision process for an imaginary company, using plain words and no code.

Meet a made-up company called Northwind Parcels. It runs a delivery platform with about four hundred services on Kubernetes across two regions. Telemetry volume is high and growing. The platform team has decided to move from the official contrib build to its own distribution. Here is how they think, step by step.

Lessons from the example

• The process starts with jobs, not with a catalog of components.
• Different tiers deserve different builds, but too many builds creates upkeep. Three was a sensible balance here.
• Measuring revealed that benefits differ per tier. Honest numbers keep expectations realistic.
• The pipeline and rollout rules matter as much as the manifest.

A custom build rarely lives alone. It plugs into the tools that deploy, manage, and observe it.

Kubernetes tooling

The OpenTelemetry Operator, the helper for running Collectors on Kubernetes introduced in the companion guide on scaling patterns, lets you name the container image a Collector should use. That is how a custom distribution enters the cluster: you publish your image to your registry and point the Operator at it. Packaging tools for Kubernetes, such as chart-based installers, also let you override the image name and tag. Because the engine and its settings format stay the same, the Operator’s modes, such as daemon set, deployment, stateful set, and sidecar, work the same with a custom image as with an official one, provided the build contains the components that the Operator’s own features expect. For example, if you use the Target Allocator to divide scraping among replicas, your build must include the scraping receiver that works with it.

Fleet management

For very large fleets, a management protocol called OpAMP allows a central server to see which Collectors are running, what version and settings each has, and to send them new settings, and in some designs new binaries. A small helper program can supervise the Collector process on a machine and carry out these instructions. If you plan to use this, your distribution identity and version reporting become even more important, since the management server relies on them to know what it is talking to.

Observability of the Collector itself

Your build reports its own health in the same standard way as any Collector. Make sure your dashboards group by distribution name and version, so you can compare health of release 2.4 against release 2.5 during a rollout. If you wrote custom components, make them report their own counters for accepted, refused, and failed items using the same conventions, so that operators do not need special knowledge to read them.

Application instrumentation

Notice that a Collector distribution is separate from the libraries and agents used inside applications to produce telemetry. Those application-side tools have their own versions and their own distribution choices. A custom Collector does not require changing any application, which is one of the reasons it is a comfortable place for a platform team to take control.

Policy engines and admission rules

Large organizations often enforce rules on what may run in their clusters, such as “only signed images from our registry.” A custom distribution signed by your own pipeline fits naturally into these rules, while an unmodified image pulled from a public registry may be blocked. This is another practical reason regulated companies publish their own builds, even when the contents are nearly identical to an official one.

When something goes wrong, a calm checklist beats guesswork. And a few definitions keep conversations clear.

Common problems and likely causes

A simple debugging order

1. Confirm which distribution and version is running, using its reported identity.
2. List its components and compare against the settings you expect.
3. Validate the settings against that exact build.
4. Turn on a simple debug exporter in a test copy and watch whether data flows.
5. Read the Collector’s own metrics for refused, dropped, and failed items.
6. Compare with the previous known-good build to isolate what changed.

Glossary in plain words

A custom Collector distribution is your own build of the OpenTelemetry Collector, assembled with the official builder tool from a manifest that lists exactly the components you choose, including any you wrote yourself. Because the component set is fixed when the program is compiled, a distribution decides what the Collector can do, while configuration decides what it actually does. Building your own gives you a smaller footprint, a smaller attack surface, control over change, room for in-house components, and a clear identity. It also gives you work: a pipeline to run, tests to maintain, upgrades to follow, and a supply chain to protect.