Before writing anything, let us be clear about what we are making and what problem it solves.

The simple definition

A processor in the OpenTelemetry Collector is a small piece of software that receives a batch of telemetry, does something to it, and passes it on. Telemetry means the three kinds of data that applications produce: traces (the journey of a request), metrics (numbers measured over time), and logs (text records of events). A processor can add labels, remove labels, rename things, drop whole items, combine items, or hold items for a while before sending them on.

A custom processor is one that you write yourself, because no existing one does what you need. It becomes part of your own build of the Collector, and it behaves exactly like the built-in ones: you list it in a configuration file, and the Collector runs it.

Why not just change the applications?

You could add the same labels inside every application. But a company may have hundreds of services written in many languages, and changing all of them for each new rule is slow and easy to get wrong. A processor works in one central place. Change it once, and every service benefits at once, with no new release of any application.

Common reasons for a custom processor

The same word is used in two different places, and mixing them up causes a lot of confusion for newcomers.

Meaning one: the Collector processor

The Collector is a separate program that receives telemetry from many applications, processes it, and sends it to storage. A processor in this sense is a pipeline stage inside that program. This is what most of this guide is about. Collector components are written in the Go programming language.

Meaning two: the SDK span processor

Inside each application, the OpenTelemetry SDK (the library that creates telemetry) also has things called span processors and log record processors. These live inside the application. They are called when a span starts and when it ends. The built-in ones batch spans before export or send each span immediately. You can write your own in whatever language the application uses, such as Java, Python, JavaScript, or Go. Metrics do not have processors in the SDK in the same way; they use views and readers instead.

Which one should you pick?

Use an SDK processor when you need live information that disappears after the request, such as who the logged-in user is, or when you must stop sensitive data before it ever leaves the application. Use a Collector processor when the change is the same for many services, depends on data that lives outside the application, or should be changeable without redeploying anything. This guide focuses on the Collector, because that is where custom processors are most useful and where the engineering is deepest. Chapter 22 mentions an SDK-side example for comparison.

To build a part, you need to know the machine it fits into. Here is a gentle tour.

The five kinds of Collector component

Pipelines

A pipeline connects these components for one kind of data. A traces pipeline has receivers, then processors in a fixed order, then exporters. The same Collector can run several pipelines at once, such as one for traces and one for logs. Processors run in exactly the order you list them, and that order matters a great deal.

flowchart LR
  R1["OTLP receiver"] --> P1["Memory limiter"]
  R2["Other receiver"] --> P1
  P1 --> P2["Your custom processor"]
  P2 --> P3["Attributes processor"]
  P3 --> P4["Batch processor"]
  P4 --> E1["Exporter to backend"]
  P4 --> E2["Exporter to archive"]
    

In Figure 1, data enters on the left from two receivers. The memory limiter comes first, to protect the Collector. Your custom processor comes next, then another standard processor. The batch processor is last, grouping data for efficient sending. Both exporters receive a copy.

The shape of the data

Inside the Collector, telemetry is held in a structure called pdata (short for pipeline data). Understanding its shape is the key to writing processors, because every processor walks through it. For traces, the shape is a set of nested boxes.

flowchart TD
  T["Traces batch"] --> RS["Resource spans, one per source such as a service instance"]
  RS --> RA["Resource attributes such as service name"]
  RS --> SS["Scope spans, one per instrumentation library"]
  SS --> SP["Span, one step of a request"]
  SP --> SA["Span attributes, events, links"]
    

Look at Figure 2. A batch contains many resource boxes. A resource describes who sent the data: the service name, the version, the machine, the cluster. Inside each resource are scope boxes that describe which library produced the items. Inside each scope are the spans themselves, each with its own attributes. Logs follow the same layout with log records at the bottom, and metrics have metrics with data points at the bottom.

This layout matters for performance. A single resource, such as “checkout service, instance 7”, may hold thousands of spans that all share the same resource attributes. A smart processor does its expensive work once per resource, not once per span.

The best custom code is the code you never had to write. Let us check the ready-made options first.

The toolbox that already exists

What the transformation language can already do

OTTL is powerful. With the transform and filter processors, you can write conditions such as “if the service name matches this pattern and the status is an error, then set this label to this value”. You can combine labels, split text, use patterns, and work on traces, metrics, and logs. Many teams are surprised how far this goes. It is a good rule to try OTTL first and write custom code only when it clearly falls short.

Where OTTL and the standard processors fall short

• Outside lookups. They cannot call your internal catalog service or read a file that changes every hour, and then keep the result fresh.
• State across batches. Rules that need memory of earlier data, such as “keep the first ten errors of each kind per minute”, need a place to keep counts.
• Complex algorithms. Custom hashing, special rate limiting, or business calculations may not fit in expressions.
• Special performance needs. A very hot path may need tuned code instead of general expressions.
• New behavior shared across teams. A rule that many teams will use may deserve a proper component with its own settings and documentation.

flowchart TD
  Q1{"Can the change be done in the application with a small code change"} -->|"yes and only one or two services"| A1["Change the application or its SDK"]
  Q1 -->|"no or many services"| Q2{"Can standard processors or OTTL express the rule"}
  Q2 -->|"yes"| A2["Use standard processors and configuration only"]
  Q2 -->|"no"| Q3{"Does it need outside data, state across batches, or special algorithms"}
  Q3 -->|"yes"| A3["Build a custom processor"]
  Q3 -->|"no"| A4["Re-check the rule, it may be simpler than it looks"]
    

A Collector processor is a small Go module with a handful of files, each with one clear job.

How the pieces talk to each other

When the Collector starts, it reads the configuration, finds the processor name, and asks that processor’s factory to create an instance. The factory hands the Collector a processor object. For every batch of telemetry that arrives, the Collector calls the processor’s function, which does the work and passes the result to the next stage. When the Collector shuts down, it tells the processor to stop. Chapter 8 follows this path in detail.

Naming and identity

Every component has a type name, a short lowercase word such as batch or attributes. This is what appears in the configuration file. Choose a clear, specific name that describes what the processor does, not who wrote it. A name like ownershipenricher tells a reader what to expect, while mycompanyprocessor tells them nothing. Names should not clash with the ones in the official repositories, because your build may someday include both.

Stability levels

Collector components declare how mature they are: development, alpha, beta, or stable. Even for internal code, being honest here helps your users. A new processor should start at a low level and move up only after it has run in production, has full tests, and has settings that you promise not to break without notice.

These two parts decide how pleasant your processor is to use, long before any telemetry flows.

The factory: a builder that knows your recipe

The factory has four small duties. It reports the processor’s type name. It can create the default configuration, so that users only need to write the settings they care about. It states which kinds of telemetry the processor supports, for example traces only, or traces and logs. And for each supported kind it has a function that creates the processor from the settings and a handle to the next stage. Each supported kind also states a stability level.

The Collector provides helper functions to build factories and processors, so you rarely need to write the plumbing by hand. Exact names and details change between Collector releases, so always read the guide for the version you build against rather than copying old examples blindly.

Supporting more than one signal

If your rule makes sense for traces, metrics, and logs, support all three, but think carefully. The three kinds of data have different shapes and different risks. A logic that is easy on spans may be tricky on metric data points, whose meaning depends on their type, such as gauges, sums, and histograms. Many good processors start with one signal and add the others later, each with its own tests.

The configuration: the order form

The configuration describes every setting a user can choose. In Go it is a plain structure with a label for each field so that the Collector can fill it from the YAML file. Good configuration follows a few simple habits.

• Sensible defaults. The processor should work with almost no settings. Every required setting is a chance for someone to make a mistake.
• Validation at startup. A configuration object can check itself. If a time setting is negative, or a file path is empty, the Collector should refuse to start and say exactly why. Failing loudly on the first minute is far better than failing quietly on the first night.
• Explicit units. Use time durations and say what they mean. A setting called “interval” is vague. A setting called “reload interval” with a duration is clear.
• Strictness. The Collector rejects settings it does not recognize, which catches spelling mistakes. Keep it that way.
• Secrets by reference. Never ask users to paste passwords into the file. Allow references to environment variables or secret stores, which the Collector’s configuration system already supports.
• Few knobs. Every setting must be documented, tested, and supported forever. Offer fewer choices with better defaults.

Write the settings guide as you go

For every setting, write three lines in the README as soon as you add it: what it does, what the default is, and an example of when you would change it. Add a short complete example configuration that someone can copy and run. Documentation written a month later is always thinner and often wrong, because the reasons behind each choice have been forgotten. A good test of your documentation is to hand it to a teammate who has never seen the processor and see whether they can set it up without asking you a single question.

This is the heart of the component: the function that is called again and again with batches of telemetry.

The shape of the work

For each kind of data, the processor has one function. It receives the context of the call and a batch of data. It returns the batch, possibly changed, and an error if something went wrong. The Collector provides a helper that wraps your function into a full processor object, adds the standard behavior around it, and wires it to the next stage. Your job is only the middle: look at the batch, change what needs changing, and hand it back.

Walking the nested boxes

Remember the layout from Chapter 3: resources contain scopes, and scopes contain items. Almost every processor follows the same walk. It loops over the resources. For each resource it may do resource-level work, such as reading the service name. Then it loops over the scopes, and over the items inside them, doing item-level work when the rule needs it. To remove items, the data structures offer a helper that removes every item matching a condition in one pass, which is safer and faster than deleting while looping. If a resource or scope ends up empty after removal, a tidy processor removes the empty box too.

Declare whether you change the data

Every processor tells the Collector whether it mutates data. This one flag is easy to forget and causes strange bugs. Here is why it matters. When one receiver feeds two pipelines, the Collector would like to share a single copy of the data to save memory. But if one pipeline changes the data while another is still reading it, the second pipeline sees corrupted results. So the Collector makes a private copy for any consumer that says “I mutate”. If your processor changes data but forgets to say so, it may silently alter data that other pipelines are using.

Ownership: the “do not touch” rule

Once you pass the batch to the next stage, it belongs to that stage. Do not read it, change it, or keep a reference to it afterward, and never hand it to a background task that outlives the call. If you need to remember something from a batch, copy the values you need into your own structures before you pass the batch on.

Concurrency: many calls at once

The Collector may call your function from several threads at the same time. Any shared state, such as a lookup table, a counter, or a cache, must be safe for parallel use. There are two common styles. One is a lock that protects the state, with many readers allowed together. The other, usually better for read-heavy cases, is an immutable snapshot: build a fresh, complete lookup table in the background, then swap a single pointer to it in one atomic step. Readers always see a whole, consistent table, never a half-built one, and they never wait.

The three signals are not the same

The same processor idea needs different care for each kind of data. Traces are made of spans. A span has a name, a kind, a status, start and end times, attributes, small notes called events, and links to other spans. If your processor renames spans, remember that backends group spans by name. A name that contains an ID, such as a user number, creates millions of tiny groups and makes charts useless.

Logs are made of log records. Each has a time, a severity such as “info” or “error”, a body that holds the message, attributes, and often the IDs of the trace and span it belongs to. If your processor changes or masks the body, keep the trace and span IDs intact, because they are the thread that connects logs to traces. The body may be plain text or a structured value, so check what you received before assuming it is text.

Metrics are the trickiest. A metric has a name, a unit, and a type, such as a gauge (a value at one moment), a sum (a running or per-interval total), or a histogram (counts in buckets). Each metric holds data points, and each data point has its own attributes. Sums also carry a temporality, which says whether each point is a running total since the start or only the change since the last point, and a flag that says whether the value can only go up. If your processor removes or changes an attribute on data points, two separate series may become identical and collide, and totals can become wrong or double counted. Any processor that touches metric attributes must think about how series merge.

Errors: what to do when something goes wrong

Errors travel back to whoever sent the data, which may be an application or another Collector. Two kinds exist. A retryable error says “try again later”, and the sender may resend the same batch. A permanent error says “this will never work”, and the sender should give up. Think carefully about which one you mean, because a wrong choice can create a storm of useless retries or lose good data.

For enrichment-style processors, a strong habit is to fail open. If a lookup fails, pass the data on unchanged, count the failure in a metric, and keep going. Blocking the whole pipeline because a side lookup is down would punish every service for a minor problem. For rules where passing data unchanged would be unsafe, such as removing secrets, do the opposite and fail closed: drop or reject rather than leak. Decide per rule, write the decision down, and make it visible to users in the documentation.

If your processor removes every item in a batch, there is nothing left to send. The helper library offers a special “skip” signal for this case, which tells the framework to stop the batch quietly without passing an empty batch downstream and without reporting an error.

A processor is born, works, and dies. Doing each stage well is what separates a hobby project from production code.

flowchart LR
  C["Read configuration"] --> V["Validate settings"]
  V --> F["Factory creates the processor"]
  F --> S["Start: open files, begin background tasks"]
  S --> RUN["Run: handle batches, often in parallel"]
  RUN --> SH["Shutdown: stop tasks, flush state, release resources"]
    

Creation: keep it light

When the factory creates the processor, it should only build the object and store the settings. Do not open files, call services, or start background work here. Creation may happen in situations where the Collector is only checking that a configuration is valid, and heavy work at that moment slows down checks and can cause surprising side effects.

Start: get ready, but never block

The Collector calls Start once all components are created. Here you do the real preparation: load the first copy of your lookup data, open connections, and begin background tasks. Start receives a handle to the “host”, through which you can find extensions, such as a shared credential provider. Two rules apply. First, Start must return promptly. If you must wait for something slow, do it in the background. Second, if the processor cannot work at all, such as a missing required file, return an error and let the Collector refuse to start. Failing at startup is good, because someone is watching at that moment.

Run: the steady state

During the run, batches arrive and background tasks do their periodic work, such as refreshing data. Everything shared between these two worlds, the batch handlers and the background tasks, needs the safe-sharing methods of Chapter 7.

Shutdown: leave no mess behind

The Collector calls Shutdown when it is stopping or rebuilding its pipelines. Your processor must stop its background tasks, close files and connections, and, if it holds data in memory that should be sent, flush it within the allowed time. Shutdown should be safe to call more than once and should respect the deadline carried in its context. A common bug is the leaked background task: a loop that never stops. Every restart then leaves one more loop running, and memory slowly climbs for days.

What about configuration changes?

Collectors are usually restarted or have their pipelines rebuilt when the configuration changes. In that case, old processor instances are shut down and new ones are created. Design for that: keep no important state only in memory unless you can afford to lose it, and make startup and shutdown quick and clean.

Some processors remember things between batches. This power is useful, and it brings extra risks.

A stateless processor treats every batch on its own, like a clerk with no memory. A stateful processor keeps information across batches, for example counting how many errors it saw this minute, remembering which traces it has already seen, or holding items until a rule is satisfied. Stateful designs allow smart behavior such as aggregation, de-duplication, rate limiting, and grouping, but they raise hard questions.

Always bound your memory

Any structure that grows with the data must have two limits: a maximum size and a maximum age. Without them, one unusual day, such as a flood of new label values, will push your Collector out of memory and take down telemetry for everyone. When the limit is reached, decide in advance what gives: drop the oldest, drop the newest, or pass data through unprocessed. Then count each time it happens.

Timers and flushing

Processors that hold items need a timer to release them. Choose the time basis carefully. Using the time the data arrived is simple. Using the time written inside the data is more accurate but must cope with late and out-of-order items. Always flush what you hold when shutting down, within the time allowed.

A small design: a per-key rate limiter

Suppose you want to stop any single service from sending more than a set number of spans per second, so that one noisy service cannot flood the backend. A simple design keeps a small counter for each service name. Each time a batch arrives, the processor looks at how many spans each service sent in the current second, compares that with the limit, and drops the extra spans. The structure behind it is a map from service name to a token bucket, a counter that refills steadily over time. The risks are the ones named above. The map must have a maximum number of entries and an expiry for idle services, or a flood of invented service names will grow it without end. The limit applies per Collector copy, so with five copies the real ceiling is five times higher unless traffic is routed by service name. And every drop must be counted per service, so that the owners can see they are being limited and fix the cause.

Many instances, one truth

In real deployments you run many Collector copies for capacity and safety. Each copy has its own memory, so state is per copy. A counter that says “no more than ten errors per minute” is actually “ten per minute per copy”. If a rule depends on seeing all related data in one place, such as all spans of one trace, you need routing in front so that related data reaches the same copy. The load balancing exporter does this for traces, as used in tail sampling. Without such routing, stateful rules give approximate results at best.

One instance per pipeline

Here is a detail that surprises many builders. If the same processor name appears in two pipelines, the Collector creates two separate instances, one per pipeline. They do not share memory. If both need the same lookup table, each loads its own copy, doubling the work and the calls to the catalog. A common cure is to put the shared part, such as the catalog client and its cache, in an extension, which exists once per Collector. The processors then ask the extension for answers. This design also makes the shared piece easier to reuse across different processors.

Let us design a complete example from need to behavior, in plain words, so that every earlier idea meets a real case.

The need

An imaginary company, which we will call Northwind (a made-up name), has 2,000 services. A central file lists, for each service, the owning team, the service tier (such as “critical” or “internal”), and a cost center. The file changes daily as teams move and services are born or retired. The company wants every trace, log, and metric to carry team.name, service.tier, and cost.center, so that dashboards, alerts, and cost reports can be grouped by owner. Applications must not need changing.

Can the standard tools do this? Static rules in the transform processor could hold a fixed mapping. But 2,000 entries that change daily, loaded from an external source, are a poor fit for hand-edited configuration. This is a real reason for custom code, under the test from Chapter 4: outside data that changes.

The data flow

flowchart LR
  CAT["Ownership catalog file or service"] -->|"reload every few minutes"| LOAD["Background loader"]
  LOAD -->|"swap in a new snapshot"| SNAP["In memory lookup table"]
  IN["Incoming batch"] --> PROC["Enricher: one lookup per resource"]
  SNAP --> PROC
  PROC --> OUT["Batch with team and tier labels"]
  PROC --> MET["Internal metrics: hits, misses, reload status"]
    

The settings

The behavior, step by step

The numbers

The catalog has 2,000 services at about 100 bytes each, so the table takes roughly 200 kilobytes of memory, which is tiny. Suppose the pipeline handles 400,000 spans per second and a typical batch has about 100 spans per resource. That means about 4,000 lookups per second. A lookup in an in-memory table takes well under a microsecond, so the total cost is a small fraction of one percent of a single processor core. Looking up once per span instead would be 100 times more calls and many more allocations. The “one lookup per resource” design is what keeps this processor almost free.

A processor that cannot explain itself is a black box in the middle of your most important pipeline.

The metadata file and generated code

Collector components describe themselves in a file called metadata.yaml. It states the component’s type name, which signals it supports and at what stability, who maintains it, and which internal metrics it produces. A tool named mdatagen reads this file and writes helper code: constants for the type and stability, builders for the internal metrics, and standard tests such as the lifecycle test. Because the code is generated, you keep the description and the code in agreement by changing only the file and running the tool again.

Internal telemetry: telling the truth about your own work

The Collector gives each component a set of tools for its own telemetry: a logger, a meter for metrics, and a tracer. Use them. For our enricher, a good set of internal metrics looks like this. The exact names get a standard prefix from the framework and may vary by version.

Logging habits

• Log events, not items. Say “catalog reloaded, 2,014 entries” once per reload. Never log a line for each span, or the logging will cost more than the processing.
• Choose levels carefully. Startup and reload results are informational, repeated failures are warnings, and fatal configuration problems are errors.
• Keep secrets and personal data out of logs. Do not print label values that might be sensitive.
• Limit repeats. If the catalog is down for an hour, one warning per attempt is fine. A warning for every batch is not.

A processor sits in the path of all your telemetry. Testing it well is not optional.

Properties worth asserting

• Idempotence. Running the processor twice gives the same result as running it once.
• Order independence. The result does not depend on the order of items within a batch.
• No loss. Unless a rule says to drop, the count of items out equals the count in.
• No input surprises. Empty batches, missing labels, labels of the wrong type, and extremely long values never crash the processor.
• Bounded resources. Memory stays flat under a long run with changing label values.

A golden-file test in plain words

A golden-file test works like checking homework against an answer key. You keep two files in the test folder: an input file that holds a realistic batch, and an expected-output file that holds exactly what the processor should produce. The test feeds the input through the processor, reads the result, and compares it with the expected file. If they differ, the test fails and prints the difference. When you change behavior on purpose, you review the new output with your own eyes, and only then replace the expected file. This habit catches small, accidental changes that no one would think to write a separate check for.

A processor is only a part. To run it, you must assemble a Collector that includes it.

Why you need your own build

The standard Collector programs that you download contain a fixed list of components. Your processor is not in that list, and there is no plug-in folder where you can drop it in at runtime. Instead, you create a custom distribution: a Collector program built from a chosen list of components, including yours. A tool called the OpenTelemetry Collector Builder, often shortened to ocb, does this assembly for you.

The manifest: your shopping list

You give the builder a small YAML file called a manifest. It has a section describing the output and a list for each kind of component.

flowchart LR
  SRC["Your processor module"] --> MAN["Builder manifest listing all components"]
  CORE["Official components at pinned versions"] --> MAN
  MAN --> OCB["Collector builder tool"]
  OCB --> GEN["Generated main module"]
  GEN --> BIN["Custom Collector binary"]
  BIN --> IMG["Container image"]
  IMG --> DEP["Deploy to clusters"]
    

Build problems you are likely to meet

• Version clash errors. The builder complains that two components need different versions of the same core module. Align every component to one release.
• Missing or stale modules. A local path points to a folder that moved, or an old downloaded copy is being used. Clear the module cache and check the paths in the manifest.
• A component that no longer exists under its old name. Names and locations change between releases. Read the release notes and update the entry.
• A binary that runs on your laptop but not in the cluster. The platform differs, such as a different processor architecture. Build explicitly for the target platform.
• A very large binary and slow starts. Too many components were included. Remove the ones you do not use.

Keep the versions in step

Collector components are released together and are built against the same core version. If your processor is compiled against one version and the rest of the Collector is another, the build may fail, or worse, compile but behave strangely. The safe habit is to pin every component to the same release and update them all together. The builder’s own version should match the version of the Collector that you aim for.

Make the build repeatable

• Run it in continuous integration, not on a laptop, so every build uses the same tools.
• Pin versions exactly. A build today and a build next month from the same manifest should produce the same program.
• Include only what you use. Every extra component adds size, startup time, and attack surface. A lean Collector with twelve components beats a heavy one with two hundred.
• Produce a software bill of materials and sign the result, so that you can later answer “what exactly is running?”.
• Build for each platform you run, such as Linux on both x86 and ARM processors.

The last step of making a processor useful is placing it in the right spot, with the right neighbors.

Two places in the configuration

A processor appears twice in a Collector configuration file. First, it is declared in a section for processors, with its settings. Then it is listed by name inside one or more pipelines in the service section. Declaring without listing does nothing. Listing without declaring is an error. If you need the same processor with different settings in different places, you give each a short suffix after a slash, which makes them separate named instances.

Order is behavior

Processors run in the order they are listed, so the order is part of your design. A common, sensible order looks like this.

Think about why each step sits where it does. If you filter by team before the team label exists, the filter finds nothing. If you enrich after sampling, you waste effort on data that was already thrown away, but you also lose the chance to sample by team. Neither choice is always right. What matters is that you choose on purpose and write the reason beside the configuration.

A wiring story

A platform team added the ownership enricher to the gateway’s traces pipeline and tested it. The labels appeared. Then the team tried to filter out data from a retired team, and nothing was filtered. The cause was the order: the filter was listed before the enricher, so at that point the team label did not exist yet, and the rule never matched. Moving the enricher earlier fixed it in a minute. The lesson is that a processor can work perfectly and still look broken, because of its neighbors. When something seems wrong, print the data after each stage and look at where the label first appears.

Checking your work

• Validate before you deploy. The Collector can check a configuration file without running it and report mistakes.
• Look at the output. A debug exporter prints what passes through, so that you can see with your own eyes that the new labels are present.
• Use the Collector’s own metrics. Compare items received and items sent around your processor to confirm that nothing disappears by accident.
• Try it with a copy first. Run a second pipeline with the new processor on a small share of traffic, or on a test environment, before touching production.

A processor runs on every batch, all day. A tiny cost per item becomes a large bill when multiplied by millions.

Do the multiplication

Imagine a busy pipeline handling 400,000 spans per second at its peak. If your processor adds a fixed amount of work for each span, here is what it costs in processor cores.

A slow rule, such as a poorly written pattern match that takes fifty microseconds, quietly eats twenty cores. This is why small habits matter.

Habits that keep a processor fast

Your processor runs in the sender’s waiting time

In a Collector, the path from receiver through the processors is normally followed in one go, in the call that delivered the data. Until your processor returns, the receiver cannot answer the sender. A slow processor therefore slows the acknowledgement that applications and other Collectors wait for, and when many are waiting, queues build up in front of it. Exporters have their own queues that decouple sending from processing, but the processors in front of them are in the direct path. Keep the work in the hot path short, and move anything slow to the background.

Memory and garbage

The Collector is written in a language that cleans up unused memory automatically, and the cleaning has a cost that grows with the number of objects created. A processor that creates many short-lived objects for every span makes the cleaner work harder and adds pauses. Prefer reusing structures, reading values in place, and building new lists only when needed. Size the memory of each Collector copy with room for peaks, and use the memory limiter together with a soft memory limit for the runtime, as the Collector’s own documentation recommends for your version. Do not invent your own pools and tricks before a measurement proves that they help, because clever memory handling is a rich source of bugs.

Where the processor sits relative to batching

The batch processor groups many small deliveries into big ones. If your processor runs before it, it receives many smaller batches, so any per-call overhead is paid more often. If it runs after it, it receives fewer, larger batches, which spreads fixed costs but increases the peak memory touched at once. For processors with real per-call costs, placing them after batching can help. For processors whose output the batch should see, such as those that drop or merge items, placing them before is better. Measure both with real data.

Scaling out

Stateless processors scale easily: add more Collector copies behind a load balancer. Stateful ones need more thought, as Chapter 9 explained. Whatever the type, keep the CPU and memory limits of each copy realistic, test with peak traffic rather than average traffic, and use the Collector’s memory limiter so that overload turns into polite refusals instead of crashes.

Your processor sees all the telemetry that passes through it. That makes it a trusted insider, and trusted insiders must be careful.

Treat processor code like production code

Telemetry often contains customer identifiers, internal addresses, error messages with fragments of private data, and sometimes secrets that applications leaked by mistake. A bug in your processor, or malicious code hiding in one of its dependencies, could read or send out all of it. So apply the same standards you apply to your most sensitive services: peer review for every change, automated scans of dependencies, and a clear owner on call.

Supply chain care

• Pin and review dependencies. Keep the list small and update deliberately.
• Scan for known vulnerabilities in continuous integration and on a schedule.
• Build from trusted sources and sign the result, so that deployments can verify what they run.
• Avoid surprise network calls. A processor that quietly contacts unknown hosts is a red flag. List and approve every outside address it may reach.

Never trust the incoming labels

Labels on telemetry come from applications, and applications can be wrong or even hostile. Your processor must cope with extremely long values, strange characters, wrong types, and missing fields without crashing or using unlimited memory. There is also a subtle spoofing risk. Suppose cost reports are grouped by the team label. If your enricher keeps labels that applications already set, a careless or dishonest service could claim to belong to another team and shift its costs. For labels that must be authoritative, such as team and cost center, overwrite whatever the application sent, and record the original under a different name if it is useful for debugging.

Protect the data the processor reads

If your processor loads a catalog from a file or a storage bucket, give it read-only access to exactly that one place. Keep credentials out of configuration files and out of logs. Check the catalog’s size and shape before using it, so that a corrupted or oversized file cannot exhaust memory or put nonsense into every span.

Do not leak through your own telemetry

Internal logs and metrics are data too. Do not write label values into logs, and avoid metric labels that hold user-level detail. Sensitive data in the Collector’s own logs is a surprisingly common leak.

Run it safely

Run the Collector as a normal, non-administrator user, with a read-only file system where possible, a minimal base image, and only the network access it needs. These steps limit the damage if something does go wrong.

A tested binary is not yet a running service. Here is how a custom Collector reaches production without surprises.

Where it runs

Collectors usually run in one or two layers. An agent layer sits close to the applications, as a sidecar next to each workload or as one copy per machine. A gateway layer is a central, scalable group that receives from the agents and sends to the backends. Enrichment from a central catalog is usually best placed in the gateway, because only the gateway needs the catalog, and the agents stay small and simple. Rules that must happen before data leaves a machine, such as removing secrets, belong in the agent.

Configuration, secrets, and change control

Treat the Collector configuration like code. Keep it in version control, review every change, and test it with the validation command before it is shipped. Use environment variable references and secret stores for passwords and tokens, so that no secret ever appears in the repository. Keep one base configuration for all environments and small overrides for differences such as addresses and limits. This way, what you tested in the staging environment is almost exactly what runs in production.

Roll out in small, reversible steps

Custom code buys freedom and charges for it. Let us look at both sides honestly.

Comparing the ways to get the job done

Trade-offs to discuss with your team

• Speed to ship versus long-term cost. Custom code is quick to write and slow to retire. Ask who will own it in two years.
• Central power versus central risk. One processor can fix every service at once, and it can break every service at once too.
• Flexibility versus simplicity. A configurable processor is helpful, and every option is something to test and support.
• Own it versus contribute it. A processor useful to others may be better shared with the community, which brings reviews and maintenance help, at the price of public standards and slower changes.

Patterns are habits that worked for many builders. Anti-patterns are the traps they fell into.

Patterns that work

Anti-patterns to avoid

A short list of habits to adopt, and the slips that appear again and again.

Best practices

• Write the rule in one sentence first. If you cannot, the design is not ready. “Add team, tier, and cost center to every resource from the ownership catalog” is a good sentence.
• Try the standard tools first and keep the evidence of why they were not enough.
• Start with one signal. Make traces perfect, then add logs and metrics with their own tests.
• Decide the failure behavior for each rule and write it in the README.
• Bound everything. Sizes, ages, retries, and time spent.
• Make it observable. Metrics for each decision and an age gauge for every cached thing.
• Document for users, not just for yourself. Settings, defaults, examples, order advice, and known limits.
• Own it on purpose. A named team, a place for questions, and a runbook for the night something breaks.

Common mistakes and quick fixes

Writing the processor is the first month of its life. Maintaining it is the next several years.

The Collector keeps moving

The Collector is released often, and many of its internal programming interfaces are still allowed to change between releases, especially those not yet marked stable. Your processor depends on those interfaces. A release that renames a helper, changes a function’s inputs, or retires an old way of doing things can stop your build until you adapt. This is normal for a fast-growing project, and it means that upgrades must be a routine habit, not a rare emergency.

An upgrade routine that works

Version your own work

Give your processor its own version numbers with a clear meaning: a small number change for fixes, a middle change for new features that keep old configuration working, and a big change when settings or behavior change in ways that can break users. Keep a short change log. Users will thank you.

Retiring a custom processor

Good news sometimes arrives: the standard tools grow and one day cover your need. When that happens, switching back to a standard component removes your maintenance cost entirely. Keep an eye out for such changes, and plan the move with the same gradual steps as any rollout. A custom processor that you can delete is a success, not a failure.

Contributing upstream

If your processor solves a problem that many companies share, consider offering it to the community repository. The usual path is to describe the idea in a public issue first, find a maintainer willing to sponsor it, follow the project’s component template and documentation rules, provide tests, name the people who will maintain it, and start at a low stability level that rises as real use grows. Contributing brings reviews and shared maintenance. It also means your design will be shaped by others, and the pace follows the community’s schedule. Processes change from time to time, so read the contribution guide of the repository before you start.

The patterns in this guide appear in components that you can read today, plus one clearly labeled made-up story.

Some processors do more than add labels. They shrink, reshape, or combine data, and each of these powers brings its own care.

Dropping

Dropping is the simplest way to reduce cost, and the easiest way to hurt yourself. A processor that drops must be able to explain every drop. Count dropped items by reason, never drop silently, and keep a switch that turns the dropping off during an investigation. If a rule can drop an entire batch, remember the special “skip” signal from Chapter 7, so that nothing empty travels onward. When unsure whether data is safe to drop, pass it on. It is cheaper to keep a little too much than to lose the one trace you needed.

Grouping

Grouping moves items into new boxes by a shared label. For example, spans that arrive in boxes by sending instance may need to be regrouped under a box for their service. The standard library has a processor that does this by attributes. When you write your own, remember the nested layout from Chapter 3: items live inside scopes inside resources, and moving an item to a new resource means building that resource’s box first. Grouping increases the number of boxes, which can slow downstream processors that work once per box, so measure the effect.

Aggregating

Aggregating combines many items into fewer summary items. A common case is turning many metric data points into one point per minute. Others include counting spans by name and status over a window. Aggregation saves money, but it asks hard questions. What is the window, and which clock decides it? What happens to late items that arrive after the window closed? What happens to a half-filled window when the Collector shuts down? Which attributes are kept, and which are removed, and will series collide when they are merged? How are the types of metrics combined? A sum can be added, but a gauge cannot be added meaningfully, and histograms must have matching buckets. Each question needs an answer in the documentation and a test.

Check the standard library first

Many common reshaping jobs already have standard processors, including regrouping by attributes, converting running totals to per-interval changes, and transforming metrics by rule. Aggregation and conversion logic is subtle, and well-used components have already met the strange cases. Write your own only for the parts that are truly specific to your company.

Short answers to the questions people ask most when they start.