Before we meet any queue, we need to feel the problem it solves.

Telemetry travels in a chain

Telemetry is the information that your software produces about itself: traces that follow requests, metrics that count and measure, and logs that record events. In a typical setup, an application produces telemetry and hands it to a program called the OpenTelemetry Collector. The Collector receives the data, processes it, and sends it onward to a backend such as a tracing system, a metrics database, or a log platform. Every link in that chain can fail.

The application may restart. The Collector may restart. The network may blink. The backend may be down for maintenance, or overloaded, or simply slow. When a link fails, the data in transit must wait somewhere or be thrown away. Where it waits, and for how long, and whether it survives a crash, is the heart of this guide.

Why this matters to real teams

Where exactly can data be lost?

It is worth listing every hop, because each one has a different cure. Data can be lost inside the application before it is even handed over, on the network between the application and the Collector, inside the Collector while it waits in memory, on the network between the Collector and the backend, and finally inside the backend itself. A persistent queue addresses only one of those places: the waiting area inside the Collector. Knowing this keeps expectations realistic.

Two kinds of waiting

It helps to separate two situations. In the first, the Collector is healthy but the next hop is slow or down, so data waits in a queue while the Collector keeps running. In the second, the Collector itself stops, so anything that was only in memory is gone. A memory queue can help with the first situation but not with the second. A persistent queue is designed to help with both.

A friendly refresher on the Collector pieces that the queue lives inside.

The Collector in one sentence

The OpenTelemetry Collector is a standalone program that receives telemetry, processes it, and exports it. It is vendor neutral, so it can accept data in many formats and send it to many destinations. It is built from small pluggable components that you connect in a configuration file.

The five kinds of component

The pipeline picture

Components are arranged into pipelines, one per signal type. Data flows from receivers through processors to exporters. Extensions sit beside the pipelines and offer services that other components can request.

flowchart LR
  A["Applications"] --> R["Receiver"]
  R --> P1["Processor: memory limiter"]
  P1 --> P2["Processor: batch"]
  P2 --> E["Exporter"]
  E --> B["Backend"]
  X["Extension: file storage"] -.->|"offers disk storage"| E
        

What the exporter really does

An exporter looks simple from outside, but most exporters share a common helper layer that gives them useful behaviour for free. That layer provides three things that matter for us. The first is a timeout, which stops a single send attempt from hanging forever. The second is retry, which tries again after failures with growing pauses between attempts. The third is a sending queue, which holds data waiting to be sent and lets several worker routines send in parallel.

Receivers have memories too

Queues are not the only part of the Collector that benefits from storage. Some receivers need to remember where they stopped. A receiver that reads log files, for example, must know how far it has read in each file. Without storage, a restart would force it to either read everything again, which creates duplicates, or start from the end, which skips lines written while it was away. With a storage extension, it saves its position regularly and continues from the saved place.

This is why the storage extension is designed as a shared helper rather than as a feature of the queue. The same component can serve exporters that need a durable queue and receivers that need durable checkpoints, each with its own private space.

Not every exporter has a queue

The queue comes from the shared exporter helper, so it is available on the exporters that use that helper, which includes the commonly used network exporters. Some exporters or components do not use it. Always read the documentation of the exporter you choose to confirm that it offers a sending queue and a storage option.

The default behaviour is a fast queue that lives in memory. To value the persistent queue, you must first see where the memory queue stops.

How the memory queue works

When data reaches an exporter that has its sending queue enabled, the exporter does not send it immediately. It places the data in a queue and returns control to the pipeline. Worker routines, called consumers, take items from the queue and try to send them to the backend. If a send succeeds, the item is finished. If it fails, the retry logic tries again after a pause. While all this happens, new data keeps arriving and joining the queue.

This design is excellent for smoothing out bursts. If the backend is momentarily slow, the queue absorbs the extra data. It also lets the pipeline accept new data quickly instead of waiting for each slow network call.

Where the memory queue falls short

sequenceDiagram
  participant App as Application
  participant C as Collector
  participant Q as Memory queue
  participant B as Backend
  App->>C: Send telemetry
  C->>Q: Place in queue
  Q->>B: Try to send
  B-->>Q: Backend unavailable
  Q->>Q: Wait and retry
  Note over C,Q: Collector restarts here
  Note over Q: Everything in memory is lost
        

Intuition for memory queue sizing

Teams often pick a memory queue size by guessing, and then wonder why data was lost. A better habit is to translate the size into time. If a queue holds 1,000 batches and the Collector produces ten batches per second, the queue represents about 100 seconds of protection. If the backend is down for five minutes, the queue overflows long before the outage ends. If the Collector produces a hundred batches per second, the same queue covers only ten seconds.

Memory is also shared with everything else the Collector does. Raising the queue size without raising the memory available can push the process into the memory limiter’s refusals or even into being killed by the operating system. The persistent queue relaxes this tension because the backlog lives on disk and uses only a small amount of memory for the items currently being handled.

What happens when the queue is full

Every queue has a maximum size. When it fills, the exporter must decide what to do with new data. By default it refuses the new data, which means the data is dropped and an error is counted. Some versions offer an option to make the pipeline wait instead of dropping, which pushes pressure backwards towards the receivers and, eventually, the applications. Neither choice is free. Dropping loses data, and waiting can slow down the senders.

The idea is small and elegant: use the same queue, but keep its contents on disk instead of in memory.

A working definition

The persistent queue is the exporter’s sending queue configured to store its items in a storage extension instead of only in memory. Because the items are written to disk, they survive a Collector restart. When the Collector starts again, the exporter reads what was left in storage and carries on delivering it.

Notice what this is not. It is not a separate product or a new kind of exporter. It is the familiar sending queue with one extra setting that points to a storage extension. That single link turns a temporary shelf into a durable one.

The two pieces working together

The exporter does not know how to write to a disk by itself. It simply asks the storage extension for a place to keep data, and the extension gives it one. This separation is a good design: the same storage extension can serve several exporters and even other components that need to remember things between restarts.

What actually gets stored

Each item in the queue is a request: a package of telemetry waiting to be sent, along with the bookkeeping the queue needs to track it. When the item is placed in the queue, it is converted into a stored form and written to disk. When it has been sent successfully, it is removed. The queue also stores small records about its own state, such as where reading and writing currently stand, so that after a restart it can pick up exactly where it left off.

flowchart LR
  P["Pipeline"] --> SQ["Sending queue in exporter"]
  SQ -->|"write item"| FS["File storage extension"]
  FS --> D[("Disk files")]
  SQ --> W["Consumer workers"]
  W --> B["Backend"]
  D -.->|"read after restart"| SQ
        

Other storage extensions exist

The file storage extension is the most widely used choice because it needs nothing except a disk. The contributions repository also contains other storage extensions that keep data in a database or a key-value service. They follow the same idea: a component asks for a storage client and the extension decides where the data really goes. Availability, maturity, and the exact behaviour of these alternatives vary, so read their documentation before using one for queued telemetry. The rest of this guide concentrates on the file-based choice because it is the simplest and the most common.

What the persistent queue gives you

The extension is the part that actually touches the disk, so it deserves a careful look.

What the extension is

The file storage extension is an optional component, distributed in the community contributions repository of the Collector, that offers durable key-value storage backed by files on a local file system. Other components ask it for a storage client, and then read and write small pieces of data through that client. It is also used by receivers that need to remember progress, for example a log-reading receiver that must remember how far it has read in each file.

Underneath, the extension uses an embedded database engine that keeps data in a single file per client. You do not need to know the engine’s internals, but you should know three of its traits: it keeps data safely even if the process stops suddenly, it allows only one process to use a file at a time, and its files grow as data is written and do not automatically shrink when data is removed.

The settings that matter, in plain words

Why compaction matters

Imagine the queue holds a million items during a long outage, so the storage file grows to several gigabytes. When the backend returns and the queue drains, the items are deleted, but the file does not shrink on its own. The space inside is free for reuse, yet the file still occupies its large size on disk. Over many outages, this can quietly fill a volume.

Compaction solves this by copying the live data into a fresh, compact file and replacing the old one. It takes some time and some temporary extra disk space, so it is usually scheduled for startup or for moments when the file has clearly grown far beyond its needed size. A well-run deployment decides in advance how compaction will happen and leaves headroom on the disk for it.

One file, one owner

The storage engine allows only one process to open a file for writing. If two Collector processes point at the same directory, the second will wait for the lock until its timeout and then fail. This is a safety feature, not a bug: two writers on one file would corrupt it. It also means that when you run several Collector replicas, each must have its own directory or, better, its own volume.

A first-time setup checklist, in plain words

The steps below describe the order in which a careful engineer sets up a persistent queue. They are described in words so that you can adapt them to whichever configuration style your team uses.

Reading the startup messages

The Collector’s startup output tells you whether storage was set up correctly. Healthy output shows the extension starting, and shows the exporter opening its queue without complaint. Warning signs include messages about an unknown storage extension name, which usually means a spelling difference between the queue settings and the extension, messages about permission being denied, which point to ownership of the directory, and messages about timeouts opening a file, which suggest that another process holds it. Keep a copy of a healthy startup log as a reference for your team.

Stability and versions

The file storage extension and the queue options around it have evolved over several releases, and some settings have been added, renamed, or changed in meaning. Before you rely on an option described here, confirm it against the documentation for the exact Collector version you run. This is especially true for compaction behaviour and for how queue sizes are measured.

Now we follow a single piece of telemetry through the queue, in words and pictures rather than code.

The write path

When a batch of telemetry arrives at the exporter, it is first turned into a request object. If the queue is persistent, that request is converted to a stored form, typically the same compact format that OTLP uses on the wire, and handed to the storage client. The storage client writes it to the file, together with a small update to the queue’s own bookkeeping that records the new write position. Only after this write succeeds does the exporter tell the pipeline that the data was accepted.

This ordering is the secret of durability. The data is on disk before the sender is told “received”. If the Collector dies a millisecond later, the data is still there.

The read path

Consumer workers repeatedly ask the queue for the next item. The queue reads it from storage, converts it back into a request, and gives it to the worker. The worker tries to send it to the backend, using the timeout and retry rules. If the send succeeds, the worker tells the queue that the item is complete, and the queue removes it from storage and updates its read position. If the send fails for good, the item is dropped and an error is counted.

Items in flight

Between “taken by a worker” and “confirmed delivered”, an item is in flight. The queue keeps a small record of which items are currently in flight, also on disk. This record is what makes recovery after a crash possible, because it tells the restarted Collector which items might have been half-sent.

sequenceDiagram
  participant P as Pipeline
  participant Q as Persistent queue
  participant S as Storage on disk
  participant W as Consumer worker
  participant B as Backend
  P->>Q: Offer batch
  Q->>S: Write item and update positions
  S-->>Q: Write confirmed
  Q-->>P: Accepted
  W->>Q: Ask for next item
  Q->>S: Read item and mark in flight
  Q-->>W: Item
  W->>B: Send
  B-->>W: Success
  W->>Q: Mark complete
  Q->>S: Remove item
        

The cost of writing before accepting

Writing to disk before saying “accepted” is what makes the queue safe, and it is also what makes it slower. Each accepted batch requires at least one disk write. If the storage is configured to force every write all the way to the physical disk before continuing, the Collector waits for the disk each time, which is the safest and slowest choice. If it lets the operating system decide when to flush, writes are faster, but a sudden power cut can lose the most recent items.

This is a classic trade-off between safety and speed, and there is no universal answer. A crash of the Collector process alone, which is the most common event, does not lose data written to the operating system’s buffers, because the operating system still holds them. The risky case is power loss or a kernel crash. Choose the stronger setting when you worry about those events and the weaker one when throughput matters more.

What happens after a crash

When the Collector starts again, the exporter opens the same storage and reads its bookkeeping. It finds the items that were waiting and the items that were in flight. The waiting items simply stay in the queue. The in-flight items are placed back into the queue so that they will be sent again, because the Collector cannot know whether the backend received them before the crash.

flowchart TD
  A["Collector starts"] --> B["Open storage file"]
  B --> C["Read queue positions"]
  C --> D["Find waiting items"]
  C --> E["Find in-flight items"]
  E --> F["Put back at the front of the queue"]
  D --> G["Resume normal sending"]
  F --> G
        

The delivery guarantee: at least once

Because in-flight items are re-sent, the persistent queue gives you an at-least-once guarantee. Every item that was accepted will be delivered one or more times, but a few items may be delivered twice. This is the honest trade-off of simple durable queues. Guaranteeing exactly once would need cooperation from the backend, such as unique identifiers it can use to remove duplicates.

What about order?

Telemetry backends generally do not need strict ordering, because each record carries its own timestamp. The queue sends items roughly in the order they arrived, but with several workers sending in parallel and retries reordering things, strict order is not guaranteed. Items re-queued after a crash may arrive later than newer items. Treat order as best effort and rely on timestamps.

A queue that is too small drops data, and a disk that is too small fails at the worst moment. A little arithmetic prevents both.

The knobs of the queue

Think in time, not just in items

The most useful question is not “how many items can I store” but “how long an outage do I want to survive”. Decide the outage length first, such as thirty minutes or four hours. Multiply it by the rate of data arriving per second and by the average size of each item. The answer is the disk space you need, before safety margin.

A worked example

Suppose a gateway Collector receives about 5,000 spans per second. After batching, the average stored request is about 200 kilobytes and holds about 800 spans. You want to survive a two-hour backend outage.

Two hours is 7,200 seconds. At 5,000 spans per second that is 36 million spans. Dividing by 800 spans per request gives 45,000 requests. At 200 kilobytes each, that is about 9 gigabytes of raw data. Adding overhead for the storage format, temporary space for compaction, and a healthy safety margin, a team would reasonably provision several times that, perhaps 30 to 40 gigabytes, for this single queue.

A second worked example: how long to drain

Return to the earlier gateway. After the two-hour outage, the queue holds about 45,000 requests. New data still arrives at the normal rate of about 6 requests per second, since 5,000 spans per second divided by 800 spans per request is just over six. Suppose the backend, once recovered, can accept 20 requests per second from this gateway.

The queue drains at the difference between what the backend accepts and what arrives: 20 minus 6, which is 14 requests per second. Dividing 45,000 by 14 gives about 3,200 seconds, a little under an hour. If the backend can only take 8 requests per second, the difference is 2 per second, and draining takes more than six hours. The lesson is that backend headroom, not disk size, usually decides how long recovery takes.

Match the queue limit to the disk

The queue limit and the disk must agree. If the queue is allowed to hold more than the disk can store, the disk fills first, writes start failing, and the failure is messier than a clean “queue full”. Always calculate the maximum queue contents in bytes and make sure the volume can hold it with room to spare for compaction.

Speed of draining

After an outage, the queue must drain faster than data arrives, or it will never empty. Draining speed depends on the number of consumers, the speed of the backend, and the speed of the disk. If the backend can only accept 20 percent more than the normal arrival rate, a two-hour backlog takes about ten hours to clear. Plan the extra headroom, and consider limiting how aggressively you drain so that you do not overwhelm a backend that has just recovered.

The same queue can sit in several places. Each placement protects against a different failure.

Pattern A: Persistent queue on the edge agent

An agent Collector runs next to the applications on each host, or each site, and forwards to a central gateway. If the link to the gateway fails, the agent’s persistent queue keeps data until the link returns. This is the most common use, and it is especially valuable for remote sites with unreliable connections.

flowchart LR
  APP["Applications"] --> AG["Agent Collector with persistent queue"]
  AG -->|"unreliable link"| GW["Central gateway"]
  GW --> BE["Backend"]
  AG --> DK[("Local disk")]
        

Pattern B: Persistent queue on the gateway

A central gateway receives from many agents and exports to the backend. Here the persistent queue shields the whole fleet from backend outages. Because many sources depend on one gateway, this queue is often the most important and the largest. It should sit on reliable storage that outlives any single Collector pod or process.

flowchart LR
  A1["Agent 1"] --> GW["Gateway Collector"]
  A2["Agent 2"] --> GW
  A3["Agent 3"] --> GW
  GW --> Q["Persistent queue on durable volume"]
  Q --> BE["Backend"]
        

Pattern C: Queue at both layers

Large estates often use persistent queues on both agents and gateways. The agent queue absorbs gateway or link failures, and the gateway queue absorbs backend failures. The price is more disks to manage and a longer total time for data to reach the backend during recovery, since data may wait in two places.

Pattern D: A message system for stronger durability

If you need data to survive the loss of an entire machine, or you need to hold data for days, a persistent queue on one disk is not enough. In that case teams place a replicated message system, such as Kafka, between layers. Collectors export to the message system and other Collectors read from it. The local persistent queue still has a role: it buffers briefly while the Collector talks to the message system itself.

A quick decision guide

If you are unsure which pattern fits, answer these questions in order. Each answer narrows the choice.

The best way to trust a safety mechanism is to walk through exactly how it behaves when things break.

Story 1: The backend goes down for an hour

Story 2: The Collector crashes during the outage

The process is killed while the queue holds a large backlog. On restart, the exporter reopens the storage, finds the backlog, and puts in-flight items back at the front. Data that would have been lost with a memory queue is delivered later. Some items may arrive twice, and data that the applications tried to send while the Collector was down depends on the applications’ own buffering.

Story 3: The disk fills up

If the volume runs out of space, writes fail. The exporter then cannot accept new data into the queue, so new data is dropped or rejected and errors are counted. Existing stored data usually remains intact, but the pipeline is now losing new telemetry. This is why disk alerts must fire long before the volume is full.

Story 4: The storage file is damaged

Rare events such as abrupt power loss with unsafe disk settings, hardware faults, or manual tampering can damage a storage file. The Collector may fail to open it. Operators then face a choice between deleting the damaged file and losing its contents, or recovering it with outside tools. Keep a documented runbook for this, and decide in advance how much a lost queue costs your business.

Story 5: The whole node is lost

If the machine and its local disk are destroyed, the queue is gone with them. The persistent queue gives no protection here unless the volume itself is replicated or detached and reattached to another machine, which is possible with network block storage in the cloud.

Story 6: An upgrade changes the stored format

The way queue items are described on disk is an internal detail, and it can change between Collector versions. Most upgrades read old data happily, but some changes may make leftover data unreadable or may require a migration. Before upgrading a Collector that holds an important backlog, read the release notes, drain the queue where possible, and test the upgrade on a copy of the storage files in a safe environment.

flowchart TD
  S["Item accepted"] --> Q{"Is the backend reachable?"}
  Q -->|"Yes"| D["Delivered and removed"]
  Q -->|"No"| R["Retry with growing pauses"]
  R --> T{"Retry window over?"}
  T -->|"No"| Q
  T -->|"Yes"| X["Dropped and counted as failure"]
  Q -.->|"Collector restarts"| RE["Item re-read from disk"]
  RE --> Q
        

The queue does not work alone. Its behaviour depends on what comes before it and what it is connected to.

Placement in the pipeline

The sending queue is inside the exporter, which is the last stop of a pipeline. That means everything the processors do happens before data enters the queue. If a processor removes sensitive fields or filters unwanted data, the queue only ever stores the cleaned result. This is good news for privacy and for disk usage.

The memory limiter and the queue

These two guard against different dangers. The memory limiter protects against the Collector using too much memory. The queue protects against data being lost while the next hop is unavailable. When the memory limiter refuses data, the refusal travels back to the sender, which should retry. A pipeline with both is more robust than a pipeline with either one alone.

Batching and queue items

The batch processor, placed before the exporter, decides how many spans, data points, or log records go into each request. Larger batches mean fewer disk writes and fewer network calls, but each queue item is bigger and a failed send costs more to redo. Very small batches mean many tiny writes, which stresses the disk. Newer Collector versions also allow batching inside the exporter’s own queue settings, so check what your version offers.

flowchart LR
  R["Receiver"] --> ML["Memory limiter"]
  ML --> PR["Other processors"]
  PR --> BA["Batch"]
  BA --> SQ["Exporter sending queue"]
  SQ --> ST[("File storage")]
  SQ --> NW["Network send with retry"]
  NW --> BE["Backend"]
        

Where the milliseconds go

When you add a disk to the path, latency increases in a few specific places. Converting an item to its stored form takes a little processor time. Writing it takes disk time. Reading it back takes more disk time and processor time. For small, well-batched requests these costs are tiny compared with a network call to a remote backend. For very small, very frequent requests, the disk overhead grows relative to the useful work, which is another reason to batch sensibly.

Batching inside the exporter

Newer versions of the Collector can group data inside the exporter’s own queue settings rather than only in a separate batch processor. Doing the grouping at the exporter has a useful side effect: the size of each queue item is decided at the last moment, close to the disk and the network. This can produce steadier request sizes and more predictable disk use. Whether this option exists, and how it measures size, depends on your version, so treat it as something to evaluate in testing rather than assume.

Backpressure and the senders

When the queue is full and set to wait instead of drop, pressure travels back through the pipeline to the receivers and then to the senders. Well-behaved senders slow down or retry. Badly behaved ones may drop data themselves. Thinking of the whole chain, from application to backend, as one connected system helps you decide where data should wait and where it should be dropped.

A persistent queue is not free. A balanced view tells you when it is worth it.

The cost view

It helps to compare costs in the same units. The cost of a persistent queue is the price of the disk, the time to operate it, and a small loss in speed. The cost of not having one is the value of the data you lose during each restart or outage, multiplied by how often those events happen. If you lose ten minutes of telemetry a few times a year and nobody notices, the disk is not worth it. If you lose the hour that explains a major incident, one such event may be worth more than years of disk costs.

Be honest about which situation you are in. Teams in regulated industries, teams that bill customers from telemetry, and teams whose incident reviews depend on complete traces usually lean towards durability. Teams that use telemetry only for rough trends can reasonably accept a memory queue.

Memory queue versus persistent queue

Persistent queue versus a message system

When a persistent queue is not worth it

If your backend is reliable, your Collector rarely restarts, and losing a few minutes of telemetry is acceptable, a memory queue with sensible limits may be enough. Disk adds moving parts, and every moving part is something that can fail on a quiet Sunday night. Choose durability when you can name the specific loss you cannot afford.

Data on disk is data at rest. It needs protection, and the queue itself needs instruments so that you can trust it.

Security considerations

Reduce what you store

Because processors run before the queue, you can remove or mask sensitive fields before anything reaches disk. This is the best protection of all, since data that was never written cannot leak. Place redaction and filtering steps ahead of the exporter, and test that they really remove what you expect.

Encryption and compliance

The file storage extension itself does not encrypt the files. If your rules require encryption at rest, provide it at a lower level, for example by using an encrypted disk or volume. Remember that retention rules apply to queued data too. A backlog that sits on disk for hours is still personal data if it contains personal data, and your deletion and retention policies should account for it.

Monitoring the queue

The Collector reports its own metrics about queues and exporters. Exact names and labels depend on the version, but the ideas are stable. Watch these signals together.

A troubleshooting guide

When something goes wrong, symptoms usually point towards a small number of causes. This table is a starting point for your own runbook.

Teams keep rediscovering the same good habits and the same expensive mistakes. Here they are, named.

Patterns worth copying

Anti-patterns to avoid

A practical checklist, grouped by the stage of your journey.

Before you start

• Write down the loss you cannot accept. “We must not lose more than two hours of traces when the backend is down” is a clear target that drives every other number.
• Measure your real data rate and batch size. Sizing from guesses leads to full disks or empty promises.
• Check which exporters you use support queues and storage. Read their documentation for your exact version.

While you configure

• Use a dedicated local volume for each Collector instance and avoid network file systems.
• Set the queue size, the retry window, and the disk size together from your outage target.
• Put the memory limiter early and redaction before the exporter.
• Choose a compaction approach and leave disk space for it.
• Make permissions strict so only the Collector account can touch the files.
• Give shutdown enough time in your service manager or orchestrator.

After you launch

• Watch queue fullness, failures, and free disk on one dashboard.
• Test recovery regularly by blocking the backend and by deleting the pod or restarting the host.
• Review sizes after traffic growth. A queue sized for last year’s traffic may be too small today.
• Keep a runbook for full disks, damaged files, and orphaned volumes.
• Make duplicates harmless in your backend or dashboards.

A vocabulary cheat sheet

An outline for your runbook

A runbook turns panic into a checklist. A short one for the persistent queue should answer the following questions in plain language, with names and links filled in for your own environment.

• How do we know the queue is unhealthy? List the alerts, their meaning, and who receives them.
• What do we do when the queue is rising? Check the backend status first, then network paths, then credentials, before touching the Collector.
• What do we do when the disk is nearly full? Decide in advance whether to grow the volume, lower inflow, or accept dropping lower-value data.
• How do we recover a damaged storage file? Describe how to stop the Collector, preserve a copy, and start with a clean directory if recovery fails.
• How do we drain a replica before removing it? Write the exact order of steps and who approves them.
• How do we roll back? Describe how to return to the memory queue and what data that choice may lose.

Where the files live decides whether your queue is truly persistent. Different platforms need different care.

Plain servers and virtual machines

On a normal host, create a dedicated directory owned by the account that runs the Collector, and point the storage extension at it. If you run the Collector under a service manager, let the manager create and own a state directory for it, and give the process enough time to stop cleanly. Put the directory on a disk with enough free space and monitor that disk.

Containers

Inside a container, the writable layer disappears when the container is removed. To keep the queue across container replacement, mount a volume from the host or from a volume manager and point the extension at that mount. Make sure the user inside the container has permission to write there, and that read-only file system settings leave the storage path writable.

Kubernetes options

Why stateful sets fit gateways

A gateway Collector with a persistent queue is a stateful program, so a controller that gives each replica a stable identity and its own volume is a natural fit. When a replica restarts or moves, it reattaches to the same volume and finds its backlog waiting. A plain stateless deployment with a shared volume does not provide this, and sharing a volume would break the one-owner rule.

flowchart TD
  subgraph SS["Stateful gateway set"]
    G0["Gateway replica 0"] --- V0[("Volume 0")]
    G1["Gateway replica 1"] --- V1[("Volume 1")]
    G2["Gateway replica 2"] --- V2[("Volume 2")]
  end
  AG["Agents"] --> SVC["Service in front of gateways"]
  SVC --> G0
  SVC --> G1
  SVC --> G2
  G0 --> BE["Backend"]
  G1 --> BE
  G2 --> BE
        

The scale-down trap

When you reduce the number of replicas, the removed replicas stop running, but their volumes often remain, with any unsent data still inside. Nothing will send that data unless you run something that reads it. Plan a drain step before scaling down: stop new data from reaching the replica, wait for its queue to empty, and only then remove it. Otherwise your queue quietly becomes a graveyard of undelivered telemetry.

Cloud volume performance

Cloud block storage often has limits on operations per second and on throughput, and some types allow short bursts above a baseline before throttling. A persistent queue can generate steady writes and, during recovery, heavy reads. If the volume is throttled, the queue slows down at exactly the wrong moment. Choose a volume type whose sustained performance matches your peak recovery needs, and watch the volume’s own performance metrics rather than assuming that more space means more speed.

Permissions and security contexts

A very common failure on Kubernetes is a permission error: the volume is mounted, but the user running the Collector cannot write to it. Group settings on the pod or an initial step that fixes ownership usually solve it. Test with the exact security settings you will use in production, because strict policies often cause the surprises.

Shutdown time

When a pod is stopped, the platform waits a grace period before forcing it to end. A Collector with a large backlog needs a few seconds to close its storage cleanly. Set the grace period so that a normal stop is never cut off in the middle of a write.

How the ideas show up in practice. These situations are illustrative, built from patterns that teams commonly describe.

Typical situations

A night of backend trouble, step by step

A simple test plan

• Test 1, backend blocked: block the network path to the backend for ten minutes, confirm that the queue grows and storage files grow with it.
• Test 2, restart during the block: restart the Collector while the backend is blocked, and confirm that the backlog is still there afterwards.
• Test 3, unblock and drain: restore the path and confirm that the queue empties and that the backend receives all the data, noting any duplicates.
• Test 4, replace the volume holder: delete the pod or replace the host and confirm that the volume and its backlog follow.
• Test 5, fill the disk: in a safe environment, let the volume fill and observe that alerts fire and the Collector reports failures instead of crashing silently.

Questions to ask before you adopt it

• Where have we lost data before? Start protecting that link first.
• How long an outage must we survive? Let this number choose your disk and retry settings.
• Can our backend handle duplicates? If not, plan for deduplication.
• Who owns the disk? Someone must watch it, resize it, and clean it.
• Do we need to survive machine loss? If yes, a persistent queue alone is not enough.
• Can we test a real failure? If you cannot rehearse it, you cannot trust it.

You do not have to protect everything at once. Move from safe and simple to advanced, proving each step before the next.

Six stages

How to prove it worked

After rollout, gather evidence so that the work earns trust and funding. Compare the number of data points missing during past outages with those missing after the change. Record the longest outage survived without loss. Track the number of enqueue failures per month and aim for zero. Note how often engineers say “the data is missing from last night”, because a drop in that complaint is the clearest sign of success. A short report built from these facts helps the next team decide whether to follow the same path.

How mature is your setup?

Short, direct answers to the questions people ask most often.

No. It is the exporter’s sending queue with a link to a storage extension. You enable the storage extension, list it in the service section, and point the exporter’s queue at it.

Not by itself. The data lives on that machine’s disk. If the disk is lost or destroyed, so is the queue. Replicated volumes or a message system are needed for that level of protection.

Occasionally. Items that were in flight when the Collector stopped are re-sent after restart, because the Collector cannot know whether the backend already received them. Most observability backends tolerate this, but verify yours.

It adds disk work to each item, so throughput is lower and latency slightly higher. The size of the effect depends on your disk, batch sizes, and settings, so measure it in your own environment rather than trusting a general number.

The storage engine does not shrink its file automatically when data is removed. Free space inside is reused, but the file keeps its size on disk until compaction rewrites it. Configure compaction to reclaim the space.

No. A storage file can be used by only one process at a time. Give each Collector instance its own directory and, ideally, its own volume.

By default new data is rejected and counted as an enqueue failure. Some versions let you choose to make the pipeline wait instead, which pushes pressure back to the senders. Either way, a full queue is a sign that the backend has been unavailable for longer than planned.

Not by the extension. If you need encryption at rest, provide it with an encrypted volume or disk, and also restrict file permissions and reduce what is written by redacting data before the exporter.

Exporters that use the common exporter helper layer generally do, which covers many popular network exporters. Others may not. Check the documentation of the exporter and its version.

Look for storage files in the configured directory while the Collector runs, check the startup logs for storage messages, and run a restart test while the backend is blocked. If the backlog survives the restart, it is working.

It depends on where you fear loss. Agents benefit when links to the gateway are unreliable. Gateways benefit when the backend can be unavailable. Many large estates use both, accepting extra disks to manage.

No. It is a lighter tool for shock absorption on a single machine. A replicated message system survives machine loss, holds data for days, and allows multiple readers and replay. Choose by your durability needs and your appetite for operations.

Treat scale-down as a procedure. Stop new data reaching the replica, let its queue drain, and then remove it. If a volume was orphaned with data inside, run a temporary Collector against it to send what remains before deleting it.

Enable the file storage extension and the persistent queue on the one exporter that has lost data before, using a dedicated local volume, a size taken from your outage target, and alerts on queue fullness and free disk.

Often yes, when the same Collector version and exporter name are used on the new machine, because the queue finds its data by the component identity. Stop the old Collector first so that the files are closed cleanly, move the whole directory, and test with a non-critical queue before relying on it during an incident.

No. Items are subject to the retry window and the queue limit. After the retry window ends, an item is dropped and counted as a failure, even though it sits on disk. If the queue fills first, new data is rejected. The queue is a shock absorber with limits, not an archive.

They can share one directory, because each exporter gets its own storage file inside it, but separate volumes give better isolation when one signal is much heavier than the others. At minimum, size the directory for the sum of the queues that use it.

Restart the Collector, or delete its pod, while the backend is blocked and the queue holds data, then confirm that the data is still there afterwards. This one test proves the persistence, the volume, the permissions, and your recovery path at the same time.

The persistent queue is the OpenTelemetry Collector’s way of keeping waiting telemetry safe across restarts and backend outages. It is the familiar exporter sending queue connected to a storage extension, most often the file storage extension, which keeps items in files on a local disk. It gives you at-least-once delivery, far greater outage tolerance than memory alone, and a calmer Collector under pressure. It does not give you protection from machine loss, exactly-once delivery, or freedom from operations. Decide the outage you want to survive, size the queue, retry window, and disk together, place the volume where it really persists, protect and monitor it, and rehearse failure before you trust it.