Why Is Graceful Shutdown and Flushing Important in OpenTelemetry?
Discover why the final seconds of a program's life hold some of its most valuable telemetry, how that data is silently lost when a process exits too quickly, and how to shut down in the right order so that nothing important is left behind.
Think about the last post collection of the day at a village post office. Throughout the afternoon, letters pile up in a bag behind the counter. A van arrives every evening to carry the bag to the sorting centre. If the post office closes early one day and nobody calls the van, every letter in the bag stays behind. The senders believe their letters were posted. The recipients never receive them. Nobody sees an error, because nothing technically failed. The letters were simply never delivered.
Telemetry in a running application behaves in much the same way. Spans, metrics, and logs are not sent the instant they are created. They wait in memory, gathered into batches, until a timer or a size limit triggers delivery. If the program ends while data is still waiting, that data vanishes without a sound. This guide explains why that happens, why it matters far more than people expect, and how graceful shutdown and flushing make sure the final bag always reaches the van.
1The Problem: The Last Seconds Are the Most Valuable
Why losing the end of a program’s telemetry hurts more than losing any other part.
When engineers investigate a problem, they usually care most about what happened just before something went wrong. A service that crashes leaves its most important clues in the final moments: the request that triggered the failure, the error that was raised, the metrics that spiked, and the log lines that explained the state of the system. These are exactly the pieces of telemetry that are youngest, and therefore most likely to still be sitting in a memory buffer when the process ends.
This creates a cruel irony. The telemetry system is designed to help you understand failures, but it is most fragile at the very moment a failure happens. A process that is killed, crashes, or simply finishes quickly may never get the chance to send its newest data. The dashboards then show a calm, clean picture right up to the moment the system died, and the cause of death is missing from the record.
Lost telemetry produces no error message. The application works, the exporter is configured correctly, and data flows normally for hours. The only symptom is a mysterious gap at the end, which is easy to mistake for a quiet period or an unrelated glitch.
It is not only about crashes
It would be easy to assume that shutdown problems only matter during rare disasters. In reality, processes end constantly in healthy systems. Every deployment replaces running instances. Every scale-down event removes machines. Every scheduled job finishes and exits. Every serverless function is frozen after returning a response. In a busy organisation with automated deployments, instances are ending many times a day, and each ending is an opportunity to lose a little telemetry.
Imagine a news reporter who files stories by sending a courier at the end of each hour. If the reporter is called away in the middle of an hour, the last half hour of notes stays in the notebook. Over a month of such interruptions, the newspaper would be missing a surprisingly large share of events, and always the freshest ones.
Graceful shutdown and flushing are not advanced tuning features. They are basic hygiene, like closing a file before a program exits. Without them, telemetry is only best-effort for the final stretch of every process.
Who actually feels the loss
The people affected by missing end-of-life telemetry are rarely the ones who caused it. On-call engineers feel it first, because they open a trace during an incident and find that the request they need is not there. They lose precious minutes wondering whether the tracing system is broken, whether the request was sampled out, or whether the service really did nothing. Each of those doubts is a direct result of a small technical omission made months earlier by someone who was never on call that night.
Product and business teams feel it more quietly. They read dashboards that undercount activity around every release, and they may draw the wrong conclusions about traffic, conversion, or error rates. Capacity planners feel it when totals are slightly lower than reality. Finance and compliance teams feel it when usage-based metrics, such as the number of processed jobs, disagree with other records. In every case, the cost is a gradual erosion of trust in the data, and trust in observability data is much harder to rebuild than to protect.
2How Telemetry Leaves Your Application
An asynchronous pipeline with several waiting rooms.
To understand why shutdown matters, you first need a picture of what happens between the moment your code records something and the moment it reaches a backend. OpenTelemetry deliberately keeps this path off your critical path. Recording a span or a measurement must be fast, so the SDK does the cheap work immediately and defers the expensive work, such as network calls, to background threads or timers.
flowchart LR
A["Application code records telemetry"] --> B["SDK creates span, metric, or log record"]
B --> C["Processor or reader holds it in memory"]
C --> D["Timer or size limit triggers a batch"]
D --> E["Exporter sends the batch over the network"]
E --> F["Collector or backend receives it"]
Provider
The top-level SDK object for a signal, such as a tracer provider, meter provider, or logger provider. It owns the processors and exporters beneath it.
Processor or Reader
The stage that receives finished telemetry and decides when to hand it onward. For traces and logs it is a processor, and for metrics it is a metric reader.
Batch
A group of items collected together so that they can be sent in one network call, which is far cheaper than sending each item alone.
Exporter
The part that actually serialises the data and sends it, commonly over the OpenTelemetry Protocol to a Collector or a backend.
Queue
A bounded waiting area inside the processor. Items sit here until the next batch is sent, and items that arrive when it is full may be dropped.
Background worker
A thread, task, or timer that wakes up periodically to send batches without blocking your application code.
The waiting room is the risk
Step three in the picture, where data sits in memory waiting for a batch to be triggered, is where all the danger lives. Anything in that waiting room is invisible to the outside world. It cannot be seen by the backend, it cannot be recovered by anyone else, and it exists only in the memory of a process that may be about to disappear.
Think of a restaurant kitchen with a pass where finished dishes wait for a waiter. The cook can keep cooking quickly because the waiter handles delivery. But if the restaurant suddenly closes, every dish on the pass is wasted, however well it was cooked.
3Batching and Its Defaults
How long data waits, and why the numbers matter.
Batching is the technique that makes telemetry affordable. Sending one network request per span would overwhelm both the application and the network, so SDKs gather items and send them in groups. The standard batch processors for traces and logs send a group when either a time limit passes or a size limit is reached, whichever comes first. Metrics work differently. A periodic reader collects and sends the aggregated values at a fixed interval.
| Setting | Meaning | Common default |
|---|---|---|
| Schedule delay for spans and logs | How often the batch processor sends what it holds | About 5 seconds |
| Maximum queue size | How many items may wait before new ones are dropped | About 2,048 |
| Maximum export batch size | How many items go out in a single network call | About 512 |
| Export timeout | How long one export attempt may take before giving up | About 30 seconds |
| Metric export interval | How often aggregated metrics are collected and sent | About 60 seconds |
The numbers above are the commonly documented defaults in the specification, but individual SDKs, agents, and distributions can differ, and configuration may change them. Always confirm the real values for the version you run.
What those defaults mean in practice
With a five second schedule delay, a process that is killed at a random moment will, on average, have about two and a half seconds of spans and log records that were never sent, and up to five seconds in the worst case. With a sixty second metric interval, the same process will on average lose about thirty seconds of aggregated measurements, and up to a full minute in the worst case. These are not bugs. They are the intended trade-off that makes batching cheap.
A bus that leaves every five minutes saves fuel compared with a taxi for each passenger. But anyone who arrives just after the bus has gone must wait, and if the whole bus service is cancelled for the day, the waiting passengers never travel. Batching is the bus, and shutdown is the cancellation.
Why not simply send everything immediately
Some teams react to data loss by shortening the delay to almost nothing. That reduces the waiting room, but it multiplies network calls, increases CPU use for serialising tiny batches, and puts more load on Collectors. It can also hurt the application itself on busy systems. A better answer is to keep sensible batching and make sure the waiting room is emptied properly whenever the process ends.
Queue overflow: the other way to lose data
Abrupt exits are not the only way for items to disappear from the waiting room. The queue has a fixed size, and when it is full, new items are dropped rather than slowing the application down. This is a deliberate protection, because telemetry must never be allowed to stall the business logic. But it means that during a burst of activity, or while the exporter is struggling to reach a slow Collector, items can be lost long before any shutdown happens.
This matters for shutdown because a flush can only send what the queue still holds. If the queue overflowed a minute earlier, the final flush will deliver a clean, complete-looking batch that is missing a chunk of earlier data. Watching the drop counters of your processors, and sizing queues and batch sizes for your peak traffic rather than your average, is therefore part of the same discipline as shutting down properly.
4What Is Lost When a Process Exits Abruptly
A realistic estimate of the damage across a fleet.
Let us put numbers to the problem. Suppose a service runs on fifty instances, and each instance records about two hundred spans per second. The batch processor sends data every five seconds. At any moment, each instance holds roughly the last five seconds of spans in its queue, which is about a thousand spans. That is comfortably under the default queue size, so nothing has been dropped yet, but all thousand are waiting.
During a rolling deployment, all fifty instances are stopped and replaced, one after another. If each one is stopped abruptly, up to a thousand spans disappear from each, giving a worst-case loss of fifty thousand spans in a single deployment. A team that deploys ten times a day could lose half a million spans daily, always from the end of each instance’s life, and nobody would see an alert.
The metric side of the story
Metrics lose data differently. A metric data point summarises everything that happened in an interval. If an instance exits thirty seconds into a sixty second interval, the half interval of increments is lost. With cumulative reporting, the next instance starts a new series from zero, and the lost half minute simply never appears in any total. With delta reporting, the lost interval is a permanent hole in the sum, because delta data points are never repaired by later reports.
| Signal | What waits in memory | Typical loss on abrupt exit |
|---|---|---|
| Traces | Finished spans in the queue, plus any spans not yet ended | Up to about one schedule delay of spans, and all open spans |
| Metrics | Aggregated values since the last export | Up to about one export interval of activity |
| Logs | Log records in the queue | Up to about one schedule delay of log lines |
The cost is invisible but real
These losses distort the picture in subtle ways. Request counts around deployment times dip slightly, and graphs show small valleys exactly when new versions roll out, which can be mistaken for real changes in traffic. Error spans from instances that were struggling just before they were replaced never appear, so a problem that caused the replacement is hidden. Traces of long requests that were in flight when the instance stopped look incomplete or absent.
If your charts show small dips at every deployment, do not assume it is traffic. Check whether instances are flushing their telemetry before they exit. It is one of the most common causes of mysterious gaps.
Why the missing data is biased towards trouble
The loss is not spread evenly across time. Instances are often replaced precisely because something was wrong with them: they were slow, they ran out of memory, or they started failing health checks. That means the final seconds of an instance’s life are disproportionately full of errors and slow requests. Imagine an instance that normally returns errors for one percent of requests, but in its final five seconds before an automatic restart, the error rate climbs to forty percent. If those final thousand spans are lost, the surviving data shows an instance that looked mostly healthy.
This produces a quiet optimism bias. The error rate you see in your dashboards is lower than the real error rate, because the failing moments are the ones that disappear. The effect is small for any single instance, but across a fleet that restarts regularly, it can hide real instability for a long time. Fixing shutdown does not only recover missing counts. It removes a systematic tendency for your data to look better than your system really is.
5Flush and Shutdown: Two Different Verbs
They sound similar, but they do different jobs and are used at different times.
The OpenTelemetry SDK specification gives each provider two related operations. The first is usually called force flush, and the second is called shutdown. People often use the words interchangeably, which leads to bugs. Understanding the difference is one of the most useful things you can learn about the SDK lifecycle.
Force flush
Asks the SDK to send everything it is currently holding right now, and to report back when that is finished or when a time limit passes. The SDK keeps running afterwards and continues to accept new telemetry.
Shutdown
Flushes remaining data and then stops the SDK for good, releasing resources such as threads and network connections. After shutdown, new telemetry is ignored.
| Aspect | Force flush | Shutdown |
|---|---|---|
| Sends pending data | Yes | Yes, as its first step |
| Keeps the SDK usable afterwards | Yes | No |
| Releases threads and connections | No | Yes |
| How often it may be called | As many times as you like | Intended to be called once |
| Typical moment to use | End of a short task, before a pause or freeze | Final step before the process exits |
Flushing is like emptying the post box into the van while the post office stays open for more letters. Shutdown is like emptying the box one last time, locking the door, and handing back the keys. You can empty the box as often as you like, but you can only lock up once.
Both are bounded by time
Both operations are meant to finish or give up within a limited time. A network that is slow or a backend that is down must not be allowed to freeze the application indefinitely. The specification therefore expects these calls to complete, fail, or time out in a reasonable period, and many SDKs accept an explicit timeout so that you can choose your own limit.
Calling shutdown at the end of every request or every function invocation is a mistake. After the first call, the SDK is stopped and later telemetry silently disappears. Use flush for repeated delivery points and reserve shutdown for the true end of the process.
6What Shutdown Actually Does, Step by Step
The orderly sequence that turns a running pipeline into a quiet, empty one.
Shutdown is not a single action. It is a chain of steps passed down from the provider to its processors, and from the processors to the exporters. Each layer is responsible for finishing its own work and then telling the layer below it to do the same. When the chain completes, nothing is left in memory and no background work is running.
The provider stops accepting new telemetry
From this point, new spans, measurements, and log records are ignored. This prevents the queue from refilling while it is being emptied.
Processors flush what they hold
Each processor sends its queued items to the exporter, in as many batches as needed. Metric readers perform a last collection of aggregated values and send it.
Exporters finish in-flight requests
Any network call already underway is given time to complete, so the final batch is not cut off in the middle.
Exporters release their resources
Connections are closed and background threads or tasks are stopped, which allows the process to exit cleanly.
The call returns a result
The caller learns whether shutdown succeeded, failed, or timed out. This result should be logged, since it tells you whether the final data really left.
Closing a shop at the end of the day follows the same order. First you stop letting new customers in. Then you serve those already at the counter. Then you count the till and bank the money. Finally you switch off the lights and lock the door. Doing these in a different order, such as locking the door before the last customer has paid, causes trouble.
What shutdown cannot do
Shutdown can only send what exists. It cannot send spans that have not ended yet, because unfinished spans are not complete records. It cannot save data that was already dropped because the queue overflowed. And it cannot succeed if the network or the backend is unreachable for the whole of the time limit. Knowing these limits helps you design around them rather than hoping shutdown will fix everything.
A span that is still open when shutdown runs is not exported. Any long-running operation that is cut off should end its span, ideally recording the reason, before the SDK is shut down.
What happens if shutdown is called twice
In real programs, more than one piece of code often believes it is responsible for shutting down the SDK. A framework hook may do it, an exit handler may do it, and the main function may do it as well. The specification says shutdown should be called only once, and that later calls should not do harm but may simply report failure or do nothing. That is helpful, because a second call is usually harmless.
It is still wise to give the responsibility to exactly one place. Two competing owners make it hard to know which call used the time budget and which one saw the result. A single, clearly named shutdown routine, called from the application’s single exit path, keeps the order predictable and makes the logged outcome trustworthy.
7Short-Lived Processes: Jobs, Scripts, and Command-Line Tools
When the whole program runs for less time than one batch delay.
The most dramatic victims of missing shutdown logic are programs that run briefly. A command-line tool might finish in two seconds. A scheduled job might run for a minute and exit. A migration script might process a file and stop. With a five second batch delay and a sixty second metric interval, such a program may finish before the first batch is ever sent, and it can end with every span, metric, and log record still in memory.
Scheduled batch jobs
A nightly job processes a day of orders and exits. Without a flush at the end, its traces and metrics never appear, and the team thinks the job did not run, or runs with no problems, because no data arrived either way.
Command-line tools
A deployment tool that talks to several services records spans for each step. If it exits right after the last step, the final and most informative spans, such as the failed step, are lost.
Continuous integration steps
Test runners and build steps that emit telemetry about durations and failures often run inside containers that are destroyed immediately after the step ends. Without an explicit flush, the telemetry about the build never leaves the container.
Data pipeline tasks
Short extract, transform, and load tasks in workflow engines may live for seconds. Their metrics about rows processed and their trace of the run are lost unless the task flushes before returning.
The rule for short-lived programs
The rule is simple and strict. Before the program exits, whether it succeeds or fails, it must shut down the telemetry providers and wait for the result. This includes failure paths. A program that crashes with an unhandled error needs a handler that still performs shutdown, because the failure is exactly the case you most want to see.
Do the work inside a root span
Wrap the program’s main work in a span so that the whole run is visible as one trace with its outcome recorded.
End the span on success and on failure
Make sure both paths record the status and end the span before anything else happens.
Shut down providers last
Call shutdown on the tracer, meter, and logger providers, and wait for them, with a sensible timeout.
Exit only after shutdown returns
Return the exit code and terminate the process only once the shutdown call has completed or timed out.
A short program that sometimes works because the timer happened to fire before exit has a hidden bug. It will fail on a faster machine or a shorter run. Always shut down explicitly.
8Serverless: When the Environment Gets Frozen
A different kind of ending, where the process does not exit but simply stops running.
Serverless platforms such as function services add a twist. After a function returns its response, the platform may freeze the execution environment instead of destroying it. A frozen environment is paused: background threads, timers, and network activity all stop. If the batch processor’s timer was due to fire a second after the response, it does not fire, and the data waits in a frozen memory until the next invocation wakes the environment, which might be minutes or hours later, or never.
sequenceDiagram
participant P as Platform
participant F as Function code
participant S as Telemetry SDK
participant B as Backend
P->>F: Invoke function
F->>S: Record spans and metrics
F->>S: Force flush before returning
S->>B: Send pending data
B-->>S: Acknowledge
F-->>P: Return response
Note over P,F: Environment may now be frozen, nothing is left waiting
Flush, do not shut down
In serverless, the same environment may be reused for many invocations, so the SDK must stay alive between them. Calling shutdown at the end of one invocation would stop the SDK, and every later invocation in the same environment would lose its telemetry. The correct tool is force flush, called at the end of each invocation before the response is returned. It empties the waiting room but leaves the SDK ready for the next call.
Time is billed
Flushing adds to the duration of every invocation, since the function must wait for the network call to complete before it can return. On platforms that charge by execution time, this has a direct cost, and it also affects the response time seen by callers. Some teams reduce the impact by sending telemetry to a nearby Collector or an extension running alongside the function, which accepts the data quickly and forwards it in the background, so the function only waits for a very short local handoff.
| Approach | Effect on telemetry | Effect on latency and cost |
|---|---|---|
| No flush at all | Frequent loss, delayed or missing data | No added time, but unreliable data |
| Flush directly to a remote backend | Reliable delivery | Adds network round trip time to every call |
| Flush to a local Collector or extension | Reliable handoff, background forwarding | Small, because the handoff is local and fast |
In serverless, treat the moment before returning as the end of the world for that invocation. Anything not delivered by then may never be delivered, so deliver it before you answer.
9Long-Running Services: Signals and Orchestration
How a service is told to stop, and what it should do in the time it is given.
A web service might run for weeks, but it still ends eventually, and the way it ends is almost always through a signal. When an orchestrator such as Kubernetes wants to remove an instance, it sends a termination signal to the main process, waits for a grace period, and then forcibly kills the process if it is still running. The termination signal is your opportunity to shut down cleanly. If the process ignores it, the forced kill follows and any waiting telemetry is destroyed.
The instance is marked as terminating
The orchestrator removes it from the list of instances that receive new traffic, so new requests stop arriving after a short delay.
A pre-stop step may run
If configured, a short command or pause runs first. Many teams use it to give load balancers time to notice that the instance is leaving.
The termination signal is sent
The main process receives the signal and the grace period countdown, commonly thirty seconds by default, is running.
The application stops and drains
It stops accepting new work and lets in-flight requests finish, ending their spans as they complete.
Telemetry is flushed and shut down
The providers send everything they hold and release their resources, within a strict time budget.
The process exits, or is killed
If everything finished in time, the process exits on its own. If the grace period ran out, a forced kill ends it immediately.
Budgeting the grace period
The grace period is a shared budget. Draining requests, flushing telemetry, closing database connections, and any other cleanup must all fit inside it. A common planning approach with a thirty second period is to give request draining about fifteen seconds, telemetry shutdown about five seconds, and leave the rest as a safety margin. If one stage overruns, the later stages are squeezed, and the telemetry stage is usually the last in line, which makes it the first casualty.
If your container starts the application through a shell script that does not pass signals on, the application never receives the termination signal at all. It only receives the final forced kill. Make sure the application process is the one that receives the signal, or that the wrapper forwards it.
The race with the load balancer
There is a subtle timing problem in the first steps of termination. When an instance is marked as terminating, the systems that route traffic need a moment to notice and stop sending requests to it. During that short window, new requests can still arrive at an instance that has already received its termination signal. If the application stops its server the instant the signal arrives, those late requests fail, and the telemetry about them is either missing or full of connection errors.
Many teams handle this with a brief pause before the application begins to stop, often implemented as a pre-stop step that simply waits a few seconds. During the pause, the instance keeps serving normally while traffic drains away on its own. Only after the pause does the real drain begin. This pause also takes part of the grace period, so it must be included in the time budget. A thirty second grace period with a five second pause, a fifteen second drain, and a five second telemetry shutdown leaves only a five second margin, which is why budgets should be written down and checked, not assumed.
10Order Matters: Shut Down Telemetry Last
The sequence of stopping things decides whether the final spans exist at all.
It is tempting to shut down telemetry first, because it feels like a background concern, and then take care of the real application. This is backwards. If the providers are shut down while requests are still being served, every span, metric, and log record created after that moment is ignored. The in-flight requests, which are usually the most interesting ones during a shutdown, produce no telemetry at all.
flowchart TD
A["Receive termination signal"] --> B["Stop accepting new requests"]
B --> C["Wait for in-flight requests to finish"]
C --> D["End any remaining open spans"]
D --> E["Force flush all providers"]
E --> F["Shut down all providers"]
F --> G["Close other resources and exit"]
Stop intake first
No new work should start once shutdown begins, otherwise the queue keeps refilling and the finish line keeps moving.
Let work finish
In-flight requests need time to complete so that their spans end properly and carry real results.
Telemetry goes last
Providers stay alive until everything that produces telemetry has stopped.
Dependencies outlive you
The Collector or agent that receives your data must still be running while you flush. Shutting it down first leaves your final batch with nowhere to go.
The sidecar ordering problem
Many deployments run a Collector or agent next to the application. When both are told to stop at the same time, the Collector may exit before the application has finished its final flush. The result is a failed export during shutdown, even though the application did everything right. The remedy is to keep the Collector alive for slightly longer than the application, using start-up and termination ordering features of the platform, or a short delay in the Collector’s own stop sequence.
Closing a restaurant in the right order means the cooks finish the last orders before the dishwasher is switched off, and the delivery driver is still waiting outside when the last parcel is ready. Switching off the driver first leaves the last parcel with no way out.
Stop the producers, then flush the pipe, and only then close the pipe. Telemetry should be the last thing alive in the process and the receiver should be the last thing alive outside it.
11Timeouts and Retries: Do Not Hang Forever
A shutdown that never finishes is nearly as bad as one that never starts.
Exporters talk to the network, and networks fail. A Collector may be down, a connection may be slow, or a backend may be rejecting requests. During normal operation, exporters deal with this by retrying failed sends after a growing delay. This is sensible when time is plentiful, but during shutdown time is exactly what you lack. A retry loop that waits thirty seconds between attempts can consume the entire grace period and still end in a forced kill.
Export timeout
The longest a single export attempt may take. A very long timeout lets a stuck connection hold up shutdown for its full duration.
Shutdown timeout
The overall budget you give to flushing and closing. Choose it so that it fits inside the platform’s grace period with a margin.
Retry with backoff
Exporters wait longer after each failure. During shutdown this means a failing backend can eat the whole budget in a few attempts.
Give up gracefully
When the budget is exhausted, shutdown should abandon remaining data, report that it did, and let the process exit rather than block it.
Choosing a shutdown budget
A good budget is long enough to send a normal final batch, which usually takes well under a second on a healthy network, and short enough that the process can never be blocked by an outage. For many services, a few seconds is plenty. For batch jobs that produce a lot of data at the end, a longer budget may be justified. The right value comes from measuring how long a normal final flush takes and adding a safety margin, not from guessing.
Measure a normal flush
Record how long a final flush takes under healthy conditions, using test runs or staging environments.
Multiply for safety
Allow several times the normal duration to cover a slower network, larger batches, and a retry or two.
Fit it inside the grace period
Check that the total of draining, flushing, and other cleanup stays comfortably below the platform’s limit.
Log the outcome
Write a line that records whether shutdown succeeded, failed, or timed out, so you can see over time whether data is being abandoned.
If shutdown has no timeout, a single unreachable Collector can prevent a process from ever exiting on its own. Orchestrators then wait out the whole grace period and kill the process anyway, so you lose both the data and the speed of the rollout.
When the backend itself is the problem
Sometimes the reason shutdown is slow has nothing to do with the application. The Collector or backend may be overloaded or down. In that situation, exporters have probably been failing for a while, and their queues may already be full, with newer items being dropped to make room. A graceful shutdown at that point can only try to send data that the pipeline could not send during normal operation either, so it will usually fail within the time budget.
The lesson is that shutdown is not a magic recovery tool. It is the final step of a pipeline that must be healthy throughout its life. Teams that need stronger guarantees for outage periods use a local agent on each machine that accepts data quickly and holds it on disk until the remote side recovers. They also record how many items were abandoned at shutdown, so that a rise in abandoned data becomes a visible signal about the health of the whole pipeline rather than a private secret of each process.
12Language and Runtime Notes
The concept is universal, but the details differ from one ecosystem to the next.
Every OpenTelemetry SDK implements the same idea of flush and shutdown, but they differ in how much the SDK does for you automatically and how much you must wire up yourself. Because these details change with versions, the safest approach is to treat the following as a map of what to look for rather than a guarantee, and to confirm the exact behaviour in the documentation of your SDK.
Automatic exit hooks
Some SDKs register a handler that runs when the program exits normally, calling shutdown for you. This covers simple cases but usually does not cover forced kills or crashes that skip the handler.
Explicit shutdown calls
Other SDKs expect you to call shutdown yourself, often as the last step of the main function or inside a deferred cleanup. Forgetting it means silent loss.
Asynchronous completion
In runtimes built on promises or tasks, shutdown returns something that must be awaited. Calling it without waiting lets the process exit before the work is done.
Agents and auto-instrumentation
Agents that attach to an application often install their own exit hooks and manage the providers for you. You may still need to handle signals and grace periods around them.
Context with deadline
Some ecosystems pass a context object that carries a deadline into shutdown, which gives you a natural way to enforce a timeout.
Disposal patterns
In ecosystems with disposable objects, providers are expected to be disposed at the end of their life, and disposal triggers the flush and shutdown.
A checklist for any language
Whatever language you use, ask the same questions. Does the SDK register an exit hook, and does it run in my deployment? Is there an explicit shutdown call, and who calls it? Does the call return something I must wait for? Can I give it a timeout? What happens when my process receives a termination signal? What happens when my process crashes? The answers to these questions, written down once per service template, prevent the majority of shutdown bugs.
| Question | Why it matters |
|---|---|
| Is an exit hook registered by default? | Tells you whether normal exits are already covered |
| Is there a way to flush without shutting down? | Needed for serverless and for checkpoints in long tasks |
| Is the call blocking or asynchronous? | Decides whether you must wait for completion |
| Can a timeout be set? | Prevents a hung shutdown from blocking exit |
| Are signals forwarded to the application? | Without them, no graceful path ever starts |
Automatic exit handlers only run if the process exits through a normal path. A forced kill, a crash in native code, or an out-of-memory termination skips them. Do not assume that the presence of an exit hook means your telemetry is safe in every situation.
13Traces, Metrics, and Logs at Shutdown
Each signal has its own quirks when the end arrives.
Although flush and shutdown apply to all three signals, each one behaves a little differently, and knowing the differences helps you decide what to expect after a clean shutdown. A common mistake is to flush the tracer provider but forget the meter provider and the logger provider, leaving two of the three signals exposed.
Traces
For traces, only finished spans are exported. A span that has started but not ended is still being built, and it is not in the queue. This means that a long-running operation interrupted by shutdown will leave no trace at all unless it ends its span. It is good practice for code that is interrupted to end its span with an error status describing the interruption, so that the trace tells the story of what was cut short.
Metrics
For metrics, the periodic reader normally performs one final collection when it is shut down, gathering the current aggregated values and sending them. This is why a clean shutdown recovers the partial interval at the end, which would otherwise be lost. With cumulative reporting, the final data point carries the latest totals. With delta reporting, it carries the activity since the last export, which is precisely the part that would have been lost.
Logs
For logs, the batch processor holds records in its queue until the next batch. The last records before an exit are usually the most important ones, such as a fatal error or a message announcing the shutdown itself. A clean shutdown sends them, while a kill leaves them behind. Teams that also write logs to standard output often keep that path as a second copy, because it is collected by the platform independently of the SDK.
| Signal | What a clean shutdown recovers | What it cannot recover |
|---|---|---|
| Traces | All finished spans waiting in the queue | Spans that never ended, and spans dropped by a full queue |
| Metrics | The final partial interval, through a last collection | Intervals already lost before this process started |
| Logs | Records waiting in the queue, including the last ones | Records dropped by a full queue |
Closing a business for the year involves three tasks: filing the finished invoices, counting the final stock, and posting the last letters. A tidy owner does all three. A hurried owner might post the letters and forget the stock count, leaving the books incomplete in a way that is hard to notice later.
Make the shutdown routine cover the tracer provider, the meter provider, and the logger provider together, and wait for all three to complete. A single helper that does this for the whole application removes the chance of forgetting one.
Seeing the loss with self-observability
You cannot fix what you cannot see, and lost telemetry is invisible by nature. Fortunately, the telemetry pipeline can report on itself. SDKs and Collectors expose their own metrics about how many items they have received, how many they have sent, how many are waiting in queues, and how many were dropped or failed to send. The exact names and the level of detail differ between languages and versions, and the conventions for them are still maturing, so check what your own components offer.
Treat these self-metrics as a first-class part of your dashboards. A steady, small queue and a flat line of zero drops are signs of a healthy pipeline. A queue that climbs, a rising count of failed exports, or a burst of dropped items at deployment times are signs that something in the ending is going wrong. Because these numbers come from the pipeline itself, they reveal losses that no ordinary application metric could ever show.
14The Collector Side: Draining the Pipeline
The same problem exists one hop later, inside the Collector itself.
The application is not the only place where data waits. The OpenTelemetry Collector has its own batching, its own queues in front of exporters, and its own retry logic. When a Collector is stopped, any data inside it that has not yet been sent shares the same fate as data inside an application. A perfect shutdown in the application is wasted if the Collector then loses the data a moment later.
flowchart LR
A["Applications flush on shutdown"] --> B["Agent Collector batches and queues"]
B --> C["Gateway Collector batches and queues"]
C --> D["Backend"]
E["Stop order: applications, then agents, then gateways"] -.-> A
Receivers stop first
On shutdown, the Collector stops its receivers first so that no new data enters, then lets the rest of the pipeline drain.
Batch processor flushes
The batch processor sends what it holds when the pipeline shuts down, so a clean stop recovers its partial batch.
Exporter queues
Exporters often have in-memory sending queues. A clean stop attempts to drain them, but a forced kill destroys whatever they hold.
Persistent queues
Where supported, a queue can be stored on disk so that unsent data survives a restart and is sent when the Collector returns.
Stop from the edge inward
In a layered design with agents near the applications and gateways further away, the order of stopping matters just as it does inside an application. Applications must finish first so that they can flush into the agents. Agents must stop next so that they can pass their data to gateways. Gateways stop last. If a gateway is replaced before the agents that feed it, those agents will fail their exports, retry, and may lose data when their own queues overflow.
Rolling updates of the Collector tier
Collectors are usually deployed with several replicas, and updates replace them one at a time. This works well for stateless forwarding, but stateful stages, such as tail sampling that buffers whole traces in memory, lose their buffered data when an instance is replaced. Plan these updates carefully, replace one instance at a time, and accept that a small number of traces may be incomplete during the change.
Teams often perfect the application’s shutdown and then restart Collectors carelessly, so the final data is lost one hop later. A pipeline is only as reliable as its weakest stopping procedure.
15When You Cannot Flush: Crashes and Hard Kills
Some endings give no warning, so the design must limit the damage in advance.
Graceful shutdown depends on the process having a chance to run code at the end. Some endings give no such chance. A process killed by a forced signal after the grace period, one terminated by the operating system for using too much memory, a machine that loses power, or a crash in native code all skip any cleanup. In those cases no flush is possible, and whatever was waiting in memory is gone.
| Kind of ending | Can the process run cleanup? | Typical telemetry outcome |
|---|---|---|
| Normal exit | Yes | Full flush is possible |
| Termination signal with handler | Yes, within the grace period | Full flush if the budget is respected |
| Unhandled error in managed code | Sometimes, through a global handler | Flush possible if the handler is installed |
| Forced kill after grace period | No | Everything in memory is lost |
| Out-of-memory termination | No | Everything in memory is lost |
| Power loss or machine failure | No | Everything in memory is lost |
Designing for the ending you cannot control
Since you cannot flush in these cases, the goal becomes making the amount of data at risk small, and making it more likely that important data left the process earlier. Several techniques help, and teams usually combine them.
Shorten the batch delay for critical signals
A smaller delay shrinks the window of data in memory. Balance it against network and CPU cost, and consider it mainly for services where the last seconds matter greatly.
Send to a local agent
Exporting to a nearby Collector on the same machine moves data out of the application’s memory very quickly, and the agent can hold and retry it even if the application is killed.
Use persistent queues in the Collector
Storing unsent data on disk protects it against Collector restarts and gives a second line of defence.
Install a last-chance error handler
Where the language allows it, install a global handler for unhandled errors that records the error, ends the active span, and flushes before the process dies.
Keep a second copy of the most vital logs
Writing important error lines to standard output or a file as well gives the platform a chance to capture them independently of the SDK.
Perfect delivery is not achievable. The goal is to make the common endings clean, make the rare endings cheap, and know which situations will always lose a few seconds of data so that nobody is surprised by them.
A worked estimate for a hard kill
Return to the instance that records about two hundred spans per second. With a five second batch delay, a hard kill can destroy up to about a thousand spans. If the team shortens the delay to one second, the exposure falls to about two hundred spans. That is a five-fold reduction, but the price is paid constantly, not only at the end. A time-triggered batch processor that sent twelve batches per minute now sends sixty, which is five times as many network calls, five times as many serialisations, and five times as many small requests for the Collector to handle.
| Batch delay | Spans at risk on a hard kill | Time-triggered batches per minute |
|---|---|---|
| 5 seconds | About 1,000 | About 12 |
| 1 second | About 200 | About 60 |
| 5 seconds plus a local agent | Small, because data leaves the process quickly | About 12 to the agent |
The comparison shows why a local agent is often the better answer. It lets the application keep a relaxed batch schedule while the data still leaves the process quickly, and the agent, which is much less likely to be killed at a random moment, takes over the job of holding and retrying it.
16Costs, Benefits, and Trade-offs
Doing shutdown well is cheap, but it is not entirely free.
Advantages
- Preserves the freshest and most valuable telemetry
- Removes mysterious dips around deployments
- Makes short jobs and serverless functions visible at all
- Keeps counts and totals closer to the truth
- Costs almost nothing in steady state
- Builds trust in the observability data
Disadvantages / Trade-offs
- Adds time to shutdown and, in serverless, to every invocation
- Needs coordination with the platform’s grace period
- Requires code in every service, or a shared template
- Can hang the process if no timeout is set
- Cannot help with hard kills and crashes
- Needs matching care in the Collector tier
The cost of the alternatives
Teams that skip graceful shutdown sometimes try to compensate by shortening batch delays to a fraction of a second. This reduces the loss window but increases the number of network calls many times over, raises CPU use for serialising small batches, and loads Collectors with a constant stream of tiny requests. Graceful shutdown solves the same problem at the exact moment it occurs, at a tiny fraction of the cost.
| Strategy | Data protected | Ongoing cost |
|---|---|---|
| Nothing, rely on luck | None of the final batch | None, but data is silently lost |
| Very short batch delay | Most of the data, but not all | High, many small network calls all the time |
| Graceful shutdown with flush | All of the final batch on clean endings | Very low, one flush per ending |
| Graceful shutdown plus local agent and persistent queue | Most data, even on hard kills | Moderate, extra component to run |
Locking the door when you leave takes a few seconds each time. Not locking it saves those seconds but occasionally costs far more. Graceful shutdown is the telemetry equivalent of locking the door: small, habitual, and very cheap compared with what it protects.
Deciding how much effort to invest
Not every service needs the full set of protections. A simple way to decide is to ask two questions. First, how costly is it if the final seconds of this service’s telemetry are missing? A payment service during an incident answers differently from an internal reporting tool. Second, how often does this service end? A service that is replaced a dozen times a day loses far more than one that runs for months.
For almost every service, the basics are worth doing: a shared shutdown routine, signal handling, and a time budget. The heavier measures, such as local agents with persistent queues and shorter batch delays, are best reserved for the services where both answers are high. Starting with the basics everywhere and adding more only where evidence demands it keeps the effort proportionate.
17Security and Privacy Considerations
The end of a process still needs the same care as the middle.
Shutdown looks like a purely operational topic, but it has security aspects worth remembering. During shutdown the process makes a final burst of network calls, sending whatever remains in its queues. Those calls must follow the same rules as any other export: they should use encryption, they should authenticate to the receiving Collector, and they should send only data that is allowed to leave the process.
Final data is sensitive too
The last spans and logs often contain error details, stack traces, and request data. Scrub sensitive values before export, not only during normal operation.
Encryption and credentials
Make sure credentials and certificates are still available at shutdown. Cleanup code that deletes them too early leads to failed exports.
Unbounded blocking
A shutdown without a timeout can be abused, or simply caused by a failing receiver, to keep processes alive and resources locked.
Error messages in logs
Shutdown failures should be logged without printing credentials, endpoints with secrets in them, or the contents of the data that failed to send.
Data left on disk
Persistent queues write unsent telemetry to disk so that it survives restarts. That data may include everything the application exports, so the directory that holds it needs the same protection as any other store of telemetry: restricted permissions, encryption at rest where required, and a clear retention rule so that old unsent data does not accumulate forever.
Problem
A shutdown routine skips redaction and sends raw error details, including request payloads and tokens, to speed up the final flush.
Why It’s Harmful
The final data is often the most detailed, because it describes a failure. Sending it unredacted places private information into long-lived telemetry stores that many people can read.
Correct Approach
Apply the same scrubbing at shutdown as during normal operation, and keep sensitive values out of span attributes and log records in the first place.
18Real-World Scenarios
Where teams most often discover the problem, usually by accident.
The following scenarios describe patterns that appear again and again across the industry. Each one starts with a puzzling symptom and ends with a missing flush as the cause.
The dip at every deployment
A team notices that request counts dip slightly at every release and recover a minute later. They suspect users are leaving during deployments. The real cause is that each replaced instance drops its last batch of spans and its last partial metric interval. After adding proper shutdown handling, the dips disappear.
The nightly job that never shows up
A scheduled report job appears in no dashboard, and its owners assume it is not running. In fact it runs for forty seconds and exits before the first metric interval ends. A final shutdown call makes its trace and metrics appear every night.
The missing crash trace
A service restarts several times a day because of an unhandled error. The traces from before each restart are always missing, so nobody can see which request caused it. A last-chance handler that ends the span, records the error, and flushes reveals the culprit within a day.
The serverless function with no traces
A function shows traces only when it happens to be invoked again soon after, because data is stuck in a frozen environment between invocations. Adding a flush before returning makes every invocation visible immediately.
The autoscaler scale-down gap
When traffic falls in the evening, the platform removes instances. Charts show brief gaps in error counts at those moments, hiding the errors that occurred just before scale-down. Handling the termination signal and flushing fixes the gaps.
The sidecar that left too early
An application flushes correctly but still loses data, because the Collector sidecar is stopped at the same moment and is already gone when the final export happens. Keeping the sidecar alive slightly longer solves it.
The spot instance warning
Cloud providers that sell spare capacity at a discount can reclaim machines with only a short warning, often a couple of minutes. Services running there need to treat that warning as a termination signal, start draining immediately, and flush telemetry within the short window, because after the deadline the machine simply disappears.
The database migration tool
A migration runs for ten minutes and fails on its last step. Its telemetry would have shown exactly which statement failed, but the tool exits with an error code straight away, skipping the cleanup path. Moving the shutdown call into a handler that runs on both success and failure restores the evidence.
The build pipeline with silent steps
Telemetry about build and test durations is missing for fast steps, because the containers running them are destroyed immediately after the step ends. A flush at the end of each step makes the measurements visible and lets the team find slow steps.
In every scenario, the system worked and the configuration was correct. The only thing missing was a deliberate, orderly ending. That is why this problem is so often discovered late.
19Design Patterns and Anti-patterns
Habits that keep endings clean, and traps that cause silent loss.
flowchart TD
A["What kind of process is this?"] --> B{"Does it run for a long time?"}
B -->|Yes| C["Handle termination signal, drain, flush, then shut down"]
B -->|No| D{"Is it a serverless function?"}
D -->|Yes| E["Force flush before every response, never shut down per call"]
D -->|No| F["Wrap work in a span, then shut down providers before exit"]
Patterns that work
One shared shutdown helper
Write a single routine that flushes and shuts down all three providers with a timeout, and reuse it in every service template.
Signal, drain, flush, close
Follow the same four-step order in every long-running service so that behaviour is predictable and reviewable.
Budgeted timeouts
Give telemetry shutdown a fixed share of the grace period and write down the numbers next to the deployment settings.
Flush at natural boundaries
In serverless and in long batch tasks, flush at the end of each unit of work, not only at the very end.
Outlive your receiver
Arrange that the Collector or agent stops after the applications that feed it.
Anti-patterns to avoid
Problem
Shutdown is called at the end of every serverless invocation or every request, because it seems the safest way to guarantee delivery.
Why It’s Harmful
Shutdown is meant to be used once. After the first call, the SDK stops, and every later piece of telemetry in the same environment is silently ignored.
Correct Approach
Use force flush at repeated delivery points and reserve shutdown for the true end of the process.
Problem
Telemetry providers are shut down first in the stop sequence, before the server has finished serving in-flight requests.
Why It’s Harmful
Everything those requests produce after that moment is ignored, so the spans and logs describing the final seconds of real work never exist.
Correct Approach
Stop intake, drain in-flight work, end open spans, then flush and shut down, so telemetry is the last component alive.
Problem
Shutdown has no timeout, or an extremely long one, and a failing Collector is allowed to keep the process alive during retries.
Why It’s Harmful
The orchestrator waits out the whole grace period and then kills the process anyway. The data is lost and rollouts become slow.
Correct Approach
Choose a budget that fits inside the grace period, abandon remaining data when it runs out, and log that this happened.
Problem
The asynchronous shutdown call is made but not awaited, so the program exits while the final export is still running.
Why It’s Harmful
The process appears to follow the correct procedure, yet the final batch is cut off. The bug is hard to notice because the code looks right.
Correct Approach
Wait for the result of shutdown, check whether it succeeded, and only then allow the process to exit.
20Testing and Monitoring Shutdown Behaviour
Because the failure is silent, you must test for it on purpose.
Shutdown bugs never announce themselves. The system looks healthy during normal operation, and the only symptom is missing data at the end. The only reliable way to find these bugs is to test the ending directly, in environments that behave like production, and to keep watching for signs of loss afterwards.
Emit a marker just before stopping
In a test environment, create a span, a metric increment, and a log line with a recognisable marker, then stop the process immediately and check that all three arrive.
Test every kind of ending
Try a normal exit, a termination signal, an unhandled error, and a rollout of the real deployment. Each exercises different code.
Test with a broken Collector
Make the Collector unreachable and confirm that shutdown finishes within its time budget and logs the abandonment instead of hanging.
Compare counts across a deployment
Compare request counts from the application’s own logs or the load balancer with telemetry counts across a rollout. A persistent gap suggests lost data.
Watch for drops and failures in production
Monitor the SDK’s and the Collector’s own metrics about queue sizes, dropped items, and failed exports, and alert on sustained non-zero drops.
| Symptom | Likely cause |
|---|---|
| Dips in counts at every deployment | Instances exit without flushing their final batch |
| Short jobs never appear | Program exits before the first batch or metric interval |
| Serverless traces appear late or never | Environment frozen with data still in memory |
| Shutdown logs show timeouts | Collector unreachable or budget too small |
| Rollouts take the full grace period | Shutdown hanging or signals not reaching the application |
| Data lost even with correct shutdown | Collector stopped too early or has no persistent queue |
Include a shutdown test in the checklist for every new service and run it again after SDK upgrades. A minute of testing on each upgrade prevents weeks of mysterious gaps later.
Rehearse a real rollout
Unit tests prove that your shutdown helper works. Only a rehearsal proves that your deployment works. Pick a staging environment that matches production, generate a steady stream of traffic that carries a countable marker, and trigger a rolling update. Afterwards, compare the number of requests your load generator sent with the number your telemetry recorded. The two should agree closely. If the telemetry is short by roughly one batch per instance, you have found a missing flush. If it is short by much more, look for signals that never arrive, grace periods that are too small, or a Collector that stops too early.
Repeat the rehearsal whenever something in the chain changes: a new SDK version, a new base image, a new startup script, or a new Collector release. Each of these can quietly break the path of the termination signal or change a default timeout. A rehearsal takes only a few minutes, and it turns shutdown from a hope into a measured property of your system.
21Frequently Asked Questions
Short answers to the questions people ask most often.
Because SDKs batch data in memory and send it later on a timer or when a size limit is reached. If the process ends while data is still waiting, nothing ever sends it.
Force flush sends pending data now and leaves the SDK running. Shutdown sends pending data and then stops the SDK permanently, releasing its resources. Shutdown is meant to be called once.
No. Use force flush before returning the response. Shutdown would stop the SDK, and later invocations in the same reused environment would lose their telemetry.
It depends on the language and on whether you use an agent. Some SDKs register an exit hook for normal exits, while others require an explicit call. Even where a hook exists, it does not cover forced kills or crashes.
No. Only spans that have ended are exported. Code that is interrupted should end its span, ideally with an error status that explains the interruption, before shutdown runs.
Roughly one batch delay of spans and logs, often about five seconds, and up to one export interval of metrics, often about a minute, plus any spans not yet ended. The exact amounts depend on your configuration.
Measure a normal final flush, allow several times that duration, and make sure it fits inside the platform’s grace period alongside request draining. A few seconds is enough for many services.
Often because the application is started through a shell wrapper that does not forward signals, so only the forced kill reaches it. Make the application the main process or ensure the wrapper passes signals along.
It can. A clean stop tries to drain its pipeline, but in-memory queues are lost on a forced kill. Persistent queues and careful stopping order, from applications to agents to gateways, reduce the loss.
Typically yes. Periodic readers usually perform one last collection and export when they are shut down, which recovers the partial interval at the end. Confirm this for your SDK version.
Not fully. Those endings skip cleanup code. You reduce the damage by shortening batch delays where needed, exporting to a local agent, using persistent queues, and installing a last-chance error handler where possible.
No. Shortening the delay shrinks the loss window but raises network and CPU cost all the time, and still loses data. Graceful shutdown handles the ending itself at almost no ongoing cost, and the two can be combined where needed.
22Summary and Key Takeaways
OpenTelemetry keeps recording cheap by batching telemetry in memory and sending it in the background. That design is excellent for performance, but it creates a waiting room where data sits until the next batch goes out. When a process ends before the waiting room is emptied, the data is lost silently. Because the newest data is the most likely to be waiting, the loss falls on exactly the moments you care about most: the final requests, the last errors, and the final partial interval of metrics.
Graceful shutdown and flushing are the discipline that closes this gap. Force flush empties the waiting room at natural boundaries, such as the end of a serverless call. Shutdown empties it one last time and then stops the SDK for good. Done in the right order, with intake stopped first, in-flight work drained, spans ended, and telemetry shut down last with a firm time budget, they preserve the story of every process’s final moments. Extend the same care to Collectors, plan for the endings that cannot be graceful, and test the ending on purpose. Like locking the door when you leave, it takes a moment and protects a great deal.
Key Takeaways
- Telemetry waits in memory — batching holds spans, metrics, and logs for seconds or a full interval, so anything still waiting when a process exits is silently lost.
- The newest data matters most — the final requests, errors, and partial metric interval are the most likely to be lost and the most valuable during an investigation.
- Flush and shutdown are different — force flush sends pending data and keeps the SDK running, while shutdown flushes and then stops it permanently and should be called only once.
- Order is everything — stop intake, drain in-flight work, end open spans, flush, and shut down telemetry last, with the Collector staying alive until after the application.
- Always set a time budget — fit the shutdown timeout inside the platform’s grace period so a failing Collector cannot hang the process or force a kill.
- Short-lived and serverless need special care — jobs must shut down before exiting, and serverless functions must flush before returning, because the environment may be frozen.
- Plan for the ending you cannot control — use local agents, persistent queues, last-chance handlers, and tested shutdown paths, and verify with a marker test that the final data really arrives.