Every program that receives data from outside has a danger hiding inside it: what if too much arrives at once?

Telemetry traffic is never smooth

Applications send traces, metrics, and logs to a central program called the OpenTelemetry Collector. In calm times the amount of data is predictable. But real systems have spikes. A new release can double the number of log lines. A retry storm can multiply the number of spans. A single misbehaving service can start sending metrics with thousands of new label combinations.

The Collector must hold data in memory while it processes it. It may batch the data, enrich it, or wait to send it to a backend that is temporarily slow. When data arrives faster than it leaves, the extra data piles up in memory, and the pile keeps growing.

What happens when memory runs out

Every computer has a limited amount of memory. In a container platform such as Kubernetes, each container is given a strict memory allowance. If the program inside uses more than its allowance, the platform kills it immediately. This is called an out-of-memory kill, or OOM kill. The Collector disappears without warning, and every piece of data it was holding in memory is lost.

Worse, when the Collector restarts, the senders often retry all their pending data at once. That creates another spike, which may crash the Collector again. This repeating pattern is called a crash loop, and it can leave you blind exactly when you need observability the most.

To place the memory limiter correctly, you first need a clear picture of the Collector’s structure.

The Collector in one paragraph

The OpenTelemetry Collector is a standalone program that receives telemetry, processes it, and exports it onward. It is vendor neutral, which means it can accept data in many formats and send it to many different backends. Teams use it to avoid sending data directly from every application to every backend, and to centralize tasks such as filtering, batching, and retrying.

The four building blocks

Order matters

Inside a pipeline, processors run one after another in the order you list them. The output of the first becomes the input of the second. This ordering is not a minor detail. It decides what each processor sees and what protection each one can offer. For the memory limiter, the order is especially important, as later chapters will show.

flowchart LR
    R["Receivers"] --> P1["Processor 1"]
    P1 --> P2["Processor 2"]
    P2 --> P3["Processor 3"]
    P3 --> E["Exporters"]
        

A plain definition first, then the behavior that makes it useful.

The definition

The memory limiter processor is a component of the OpenTelemetry Collector that periodically measures how much memory the Collector is using and, when that memory gets too high, starts refusing new data so that the Collector can recover before it crashes. It is designed to prevent out-of-memory situations caused by sudden surges of incoming telemetry.

The three things it does

What it is not

It is not a data filter that chooses which telemetry is important. It does not look at the content of spans, metrics, or logs. It makes no decisions based on what the data says, only on how full memory is. It is also not a replacement for giving the Collector enough memory in the first place, and not a replacement for scaling out when your normal load grows.

You do not need to be a systems expert, but a few simple ideas make the settings much easier to understand.

The Collector is written in Go

The Collector is a program written in the Go language. Go manages memory automatically. It hands out memory to your data as needed and has a background cleaner, called the garbage collector, that finds memory no longer in use and frees it. This is convenient, but it means memory use is not always a precise, instant number. The cleaner runs from time to time, not constantly.

The heap

The part of memory where the program stores most of its changing data, such as batches of spans waiting to be sent, is called the heap. The memory limiter looks at the heap size when it measures. Besides the heap, the process also needs some extra memory for itself, such as program code and internal bookkeeping, so the total used by the container is somewhat higher than the heap alone.

Where the memory goes inside a Collector

The processor works with two thresholds and a safety margin. Understanding them is the heart of the topic.

The hard limit

The hard limit is the maximum amount of memory, in mebibytes, that you want the Collector process heap to use. A mebibyte is a standard unit of memory slightly larger than a megabyte. When measured memory goes above the hard limit, the processor not only refuses data but also triggers garbage collection to push memory back down.

The spike limit

The spike limit is a safety margin. It estimates how much memory could suddenly be added between two checks. If the Collector is receiving data very fast, memory could jump a lot in one second. The spike limit is the room you keep in reserve for that jump. If you do not set it, a default of about twenty percent of the hard limit is commonly used.

The soft limit

The soft limit is not set directly. It is calculated as the hard limit minus the spike limit. Once memory goes above the soft limit, the processor starts refusing new data. Because the soft limit sits below the hard limit, refusing starts early enough that the spike margin can absorb data still in flight.

stateDiagram-v2
    [*] --> Normal
    Normal --> Refusing: Memory above soft limit
    Refusing --> Normal: Memory below soft limit
    Refusing --> ForcedGC: Memory above hard limit
    ForcedGC --> Refusing: Garbage collection done, still above soft limit
    ForcedGC --> Normal: Memory drops below soft limit
        

A worked example in words

Suppose you set the hard limit to 1,000 mebibytes and the spike limit to 200 mebibytes. The soft limit is then 800 mebibytes. While memory stays under 800, everything flows normally. When memory climbs past 800, the processor begins refusing incoming data. If memory somehow still climbs past 1,000, it also forces garbage collection. When memory falls back under 800, data is accepted again.

There are only a few settings, which is part of what makes the processor easy to adopt.

Fixed numbers versus percentages

You choose between two styles. Fixed numbers are simple and predictable, and they are good when you control the container size. Percentages adapt automatically to the memory the process can see, which is helpful when the same configuration file is shared across environments with different resource sizes. You should pick one style for the hard limit and not mix the two for the same threshold.

Refusing data sounds simple, but what the sender does next decides whether you lose anything.

What refusing really means

When the memory limiter is in refusing mode, it does not quietly throw data away. It returns an error up the chain, back to the receiver that handed it the data. The receiver is then expected to tell the sender that the Collector is temporarily unable to accept the data. This signal going backwards against the flow of data is called back-pressure.

Why back-pressure is a good thing

Back-pressure moves the problem to the place that is best able to handle it. Instead of the Collector crashing and losing everything, the sender, which has the data, is told to wait. Many senders keep the data in a small buffer and retry after a short delay. By the time they retry, the Collector has recovered.

It depends on the receiver and the sender

How well this works depends on both ends. A push-style receiver, such as one accepting OTLP calls, can return a retryable error to the application’s exporter. A pull-style receiver, which scrapes data from targets, simply skips or fails that scrape, and the data for that moment may be lost, although the next scrape will usually succeed.

sequenceDiagram
    participant A as Application Exporter
    participant R as Receiver
    participant M as Memory Limiter
    participant X as Rest of Pipeline
    A->>R: Send batch of telemetry
    R->>M: Pass data to first processor
    M->>M: Check memory state
    alt Memory below soft limit
        M->>X: Accept and forward
        X-->>A: Success
    else Memory above soft limit
        M-->>R: Refuse with error
        R-->>A: Retryable error response
        Note over A: Wait and retry later
    end
        

If you remember one placement rule, remember this one.

The recommendation

The memory limiter should be the first processor in each pipeline, placed immediately after the receivers. This is the standard guidance in the Collector documentation, and the reasons are practical.

What goes wrong with bad placement

If the limiter is placed after a batching processor, the batcher may already have accepted and stored a large amount of data before the limiter gets a chance to say no. The data in the batcher still occupies memory, and the limiter can only protect against data that has not yet reached it. In effect, you have put the guard inside the shop instead of at the door.

flowchart LR
    R["Receiver"] --> M["Memory limiter first"]
    M --> A["Attribute and filter processors"]
    A --> B["Batch processor"]
    B --> E["Exporter"]
        

Good settings come from understanding your container, your traffic, and your safety margins.

Start from the container allowance

The first number to know is the total memory your Collector container or machine is allowed to use. The memory limiter’s hard limit must sit comfortably below that total. This is because the process uses extra memory beyond the heap, and because the limiter reacts with a small delay. If the hard limit equals the container allowance, an OOM kill can still happen before the limiter acts.

A common guideline

A widely used rule of thumb is to set the hard limit at roughly eighty percent of the container’s memory allowance, and the spike limit at roughly twenty to twenty-five percent of the hard limit. This leaves a buffer for non-heap memory and for the fast spikes between checks. Treat these as starting points, then adjust using real measurements from your environment.

Match the spike limit to your traffic pattern

If your traffic is steady, a smaller spike margin is fine. If your traffic arrives in sharp bursts, for example from batch jobs that send large payloads at once, a larger margin or a shorter check interval is wise. The goal is that the largest realistic jump in memory between two checks fits inside the spike margin.

The memory limiter is one safety tool among several, and it works best in combination.

The runtime memory limit setting

Go programs can be given a soft memory target through a standard environment variable named GOMEMLIMIT. When set, the Go runtime tries to garbage collect more aggressively as the process approaches that target. Many operators set this to a value a little below the container allowance, and some also align it with the memory limiter’s hard limit. It complements the limiter by making cleanup happen sooner, while the limiter controls the inflow of data. Confirm the current recommended practice in the Collector documentation for your version.

The old ballast approach

Older guides mention a memory ballast, which was a large block of unused memory allocated on purpose to change how the garbage collector behaved. That approach has been replaced by the runtime memory limit setting, and recent Collector versions have moved away from it. If you find tutorials describing ballast configuration, treat them as outdated and check current guidance.

Batch processor

The batch processor groups data before sending. Larger batches and longer wait times mean more data held in memory. It is wise to keep batch sizes reasonable and to place the batch processor after the memory limiter, so the limiter can control how much data arrives to be batched.

Exporter queues and retries

Exporters often have a sending queue that holds data while a backend is slow, and retry settings that decide how long to keep trying. A large queue can become the main consumer of memory during an outage. The queue size should be chosen with the memory budget in mind. Where durability matters, some deployments store the queue on disk through a storage extension, which reduces memory pressure and survives restarts.

Let us follow a realistic spike from the first signs to full recovery.

The scenario

A retail company runs a gateway Collector with a memory limiter configured as the first processor. A flash sale begins, and the volume of traces triples in a few minutes. At the same time, the tracing backend slows down because it is also under load.

flowchart TD
    S["Spike in incoming data"] --> Q["Queues fill, memory rises"]
    Q --> C{"Above soft limit?"}
    C -- "No" --> N["Keep accepting data"]
    C -- "Yes" --> R["Refuse new data with errors"]
    R --> B["Senders retry later"]
    B --> D["Queues drain, memory falls"]
    D --> C
        

What would have happened without the limiter

Without the memory limiter, the queue would have kept growing until the container exceeded its allowance. The platform would have killed the Collector, and all queued data would have vanished. When it restarted, every application would have retried at once, and the cycle might have repeated. The limiter converts a dangerous crash into a calm pause.

A safety valve that no one watches is a mystery. You should know when yours is doing its job.

Signals worth watching

The Collector exposes its own internal metrics, which describe its health. The memory limiter contributes counters that show how much data it has refused, separately for spans, metric points, and log records. A rising refusal count is a direct sign that the Collector is under memory pressure and sending back-pressure to your applications.

Alerting suggestions

Create alerts on a sustained non-zero refusal rate, not just a single blip, because brief refusals during a spike are normal and healthy. Alert separately on any container restart for the Collector, since that indicates the limiter did not do its job. Combine these with dashboards that show memory, queue sizes, and refusals side by side, so the story is easy to read.

Where you run your Collector changes how you think about memory protection.

Agent pattern

In the agent pattern, a small Collector runs next to each application, for example on every host or as a sidecar container. It receives data from nearby applications and forwards it onward. These Collectors usually have small memory allowances, so the limiter is especially valuable to keep them from competing with the application for resources.

Gateway pattern

In the gateway pattern, a central pool of Collectors receives data from many sources. These instances carry the most traffic and the most risk of sudden spikes. They benefit from the limiter, from careful queue sizing, and from the ability to scale horizontally behind a load balancer.

Why one limiter is not enough for scale

The limiter protects a single instance. It does not coordinate across instances. If one instance is refusing data, a load balancer might still send new traffic to it, unless it knows the instance is struggling. Good gateway designs combine the limiter with health checks, sensible load balancing, and automatic scaling based on resource use.

Like every tool, the memory limiter has strengths and costs.

A defense against overload and abuse

Because the limiter reacts to memory pressure regardless of the cause, it also gives some protection against abusive or accidental floods. A misconfigured service or a malicious sender that blasts huge amounts of data at the Collector cannot force it to consume unlimited memory. The limiter will start refusing before the process crashes. It is not a complete security control, but it is a helpful safeguard for availability.

A short field guide for when things do not behave as expected.

Troubleshooting guide

Habits that work

Traps to avoid

Quick answers to the questions people ask most often.

It is not forced by the software, but it is strongly recommended for any production Collector. Without it, a surge of data can push the process beyond its memory allowance and cause a crash.

It refuses to accept new data when memory is high, by returning an error to the receiver. Whether data is actually lost depends on the sender. If the sender retries, the data is usually delivered later. If the sender does not retry, the data may be lost.

The soft limit is the point at which the processor starts refusing new data. It equals the hard limit minus the spike limit. The hard limit is the higher ceiling at which the processor also forces garbage collection to reduce memory.

Placing it first means refused data does no further work and consumes no extra memory, and the refusal reaches the receiver quickly. Later processors, which may hold data, receive only what the Collector has room for.

Yes. The processor supports percentage-based settings relative to the memory available to the process. This is useful when the same configuration is used on machines or containers of different sizes. Confirm the exact behavior for your Collector version.

A check interval of around one second is the usual recommendation. If your traffic is very spiky, a shorter interval or a larger spike margin reduces the chance of memory overshooting between checks.

No. It protects one instance against short surges. If your normal load regularly approaches the limits, you need more memory per instance or more instances.

It is good practice on both. Agents keep local resource use in check, and gateways handle the largest and spikiest loads. Together they allow back-pressure to flow smoothly through each tier.

Watch the Collector’s own metrics for refused spans, metric points, and log records, along with memory use, exporter queue sizes, and container restarts. Occasional refusals during spikes show the limiter doing its job. Container kills show it is not enough.

Yes, the processor is designed to work with traces, metrics, and logs. You add it to each pipeline you run so that all signal types are protected.

The memory limiter processor is a small but vital part of a healthy OpenTelemetry Collector. It watches memory at regular intervals, refuses new data when memory passes a soft limit, and forces cleanup when memory passes a hard limit. By sending errors back to receivers, it creates back-pressure that gives senders a chance to wait and retry, turning a possible crash into a short pause. It works best as the first processor in every pipeline, with limits set comfortably below the container allowance, with sensible queue sizes, and with senders that retry. It is a guard against surges, not a cure for chronic under-provisioning, so monitor refusals and scale when the pressure is constant rather than occasional.