Why Is Graceful Shutdown and Flushing Important in OpenTelemetry?

Why Is Graceful Shutdown and Flushing Important in OpenTelemetry?

Discover why the final seconds of a program's life hold some of its most valuable telemetry, how that data is silently lost when a process exits too quickly, and how to shut down in the right order so that nothing important is left behind.

Think about the last post collection of the day at a village post office. Throughout the afternoon, letters pile up in a bag behind the counter. A van arrives every evening to carry the bag to the sorting centre. If the post office closes early one day and nobody calls the van, every letter in the bag stays behind. The senders believe their letters were posted. The recipients never receive them. Nobody sees an error, because nothing technically failed. The letters were simply never delivered.

Telemetry in a running application behaves in much the same way. Spans, metrics, and logs are not sent the instant they are created. They wait in memory, gathered into batches, until a timer or a size limit triggers delivery. If the program ends while data is still waiting, that data vanishes without a sound. This guide explains why that happens, why it matters far more than people expect, and how graceful shutdown and flushing make sure the final bag always reaches the van.

1The Problem: The Last Seconds Are the Most Valuable

Why losing the end of a program’s telemetry hurts more than losing any other part.

When engineers investigate a problem, they usually care most about what happened just before something went wrong. A service that crashes leaves its most important clues in the final moments: the request that triggered the failure, the error that was raised, the metrics that spiked, and the log lines that explained the state of the system. These are exactly the pieces of telemetry that are youngest, and therefore most likely to still be sitting in a memory buffer when the process ends.

This creates a cruel irony. The telemetry system is designed to help you understand failures, but it is most fragile at the very moment a failure happens. A process that is killed, crashes, or simply finishes quickly may never get the chance to send its newest data. The dashboards then show a calm, clean picture right up to the moment the system died, and the cause of death is missing from the record.

!
The Silent Nature of the Problem

Lost telemetry produces no error message. The application works, the exporter is configured correctly, and data flows normally for hours. The only symptom is a mysterious gap at the end, which is easy to mistake for a quiet period or an unrelated glitch.

It is not only about crashes

It would be easy to assume that shutdown problems only matter during rare disasters. In reality, processes end constantly in healthy systems. Every deployment replaces running instances. Every scale-down event removes machines. Every scheduled job finishes and exits. Every serverless function is frozen after returning a response. In a busy organisation with automated deployments, instances are ending many times a day, and each ending is an opportunity to lose a little telemetry.

Simple Analogy

Imagine a news reporter who files stories by sending a courier at the end of each hour. If the reporter is called away in the middle of an hour, the last half hour of notes stays in the notebook. Over a month of such interruptions, the newspaper would be missing a surprisingly large share of events, and always the freshest ones.

i
Keep in Mind

Graceful shutdown and flushing are not advanced tuning features. They are basic hygiene, like closing a file before a program exits. Without them, telemetry is only best-effort for the final stretch of every process.

Who actually feels the loss

The people affected by missing end-of-life telemetry are rarely the ones who caused it. On-call engineers feel it first, because they open a trace during an incident and find that the request they need is not there. They lose precious minutes wondering whether the tracing system is broken, whether the request was sampled out, or whether the service really did nothing. Each of those doubts is a direct result of a small technical omission made months earlier by someone who was never on call that night.

Product and business teams feel it more quietly. They read dashboards that undercount activity around every release, and they may draw the wrong conclusions about traffic, conversion, or error rates. Capacity planners feel it when totals are slightly lower than reality. Finance and compliance teams feel it when usage-based metrics, such as the number of processed jobs, disagree with other records. In every case, the cost is a gradual erosion of trust in the data, and trust in observability data is much harder to rebuild than to protect.

2How Telemetry Leaves Your Application

An asynchronous pipeline with several waiting rooms.

To understand why shutdown matters, you first need a picture of what happens between the moment your code records something and the moment it reaches a backend. OpenTelemetry deliberately keeps this path off your critical path. Recording a span or a measurement must be fast, so the SDK does the cheap work immediately and defers the expensive work, such as network calls, to background threads or timers.

flowchart LR
  A["Application code records telemetry"] --> B["SDK creates span, metric, or log record"]
  B --> C["Processor or reader holds it in memory"]
  C --> D["Timer or size limit triggers a batch"]
  D --> E["Exporter sends the batch over the network"]
  E --> F["Collector or backend receives it"]
        
FIG 1 — The path from a recorded event to a backend, with a waiting room at step three
Component 1

Provider

The top-level SDK object for a signal, such as a tracer provider, meter provider, or logger provider. It owns the processors and exporters beneath it.

Component 2

Processor or Reader

The stage that receives finished telemetry and decides when to hand it onward. For traces and logs it is a processor, and for metrics it is a metric reader.

Component 3

Batch

A group of items collected together so that they can be sent in one network call, which is far cheaper than sending each item alone.

Component 4

Exporter

The part that actually serialises the data and sends it, commonly over the OpenTelemetry Protocol to a Collector or a backend.

Component 5

Queue

A bounded waiting area inside the processor. Items sit here until the next batch is sent, and items that arrive when it is full may be dropped.

Component 6

Background worker

A thread, task, or timer that wakes up periodically to send batches without blocking your application code.

The waiting room is the risk

Step three in the picture, where data sits in memory waiting for a batch to be triggered, is where all the danger lives. Anything in that waiting room is invisible to the outside world. It cannot be seen by the backend, it cannot be recovered by anyone else, and it exists only in the memory of a process that may be about to disappear.

Simple Analogy

Think of a restaurant kitchen with a pass where finished dishes wait for a waiter. The cook can keep cooking quickly because the waiter handles delivery. But if the restaurant suddenly closes, every dish on the pass is wasted, however well it was cooked.

3Batching and Its Defaults

How long data waits, and why the numbers matter.

Batching is the technique that makes telemetry affordable. Sending one network request per span would overwhelm both the application and the network, so SDKs gather items and send them in groups. The standard batch processors for traces and logs send a group when either a time limit passes or a size limit is reached, whichever comes first. Metrics work differently. A periodic reader collects and sends the aggregated values at a fixed interval.

SettingMeaningCommon default
Schedule delay for spans and logsHow often the batch processor sends what it holdsAbout 5 seconds
Maximum queue sizeHow many items may wait before new ones are droppedAbout 2,048
Maximum export batch sizeHow many items go out in a single network callAbout 512
Export timeoutHow long one export attempt may take before giving upAbout 30 seconds
Metric export intervalHow often aggregated metrics are collected and sentAbout 60 seconds
!
Check Your Own SDK

The numbers above are the commonly documented defaults in the specification, but individual SDKs, agents, and distributions can differ, and configuration may change them. Always confirm the real values for the version you run.

What those defaults mean in practice

With a five second schedule delay, a process that is killed at a random moment will, on average, have about two and a half seconds of spans and log records that were never sent, and up to five seconds in the worst case. With a sixty second metric interval, the same process will on average lose about thirty seconds of aggregated measurements, and up to a full minute in the worst case. These are not bugs. They are the intended trade-off that makes batching cheap.

~5 s
worst-case span and log data waiting in memory
~60 s
worst-case metric data waiting in memory
~2,048
items the queue holds before dropping new ones
Simple Analogy

A bus that leaves every five minutes saves fuel compared with a taxi for each passenger. But anyone who arrives just after the bus has gone must wait, and if the whole bus service is cancelled for the day, the waiting passengers never travel. Batching is the bus, and shutdown is the cancellation.

Why not simply send everything immediately

Some teams react to data loss by shortening the delay to almost nothing. That reduces the waiting room, but it multiplies network calls, increases CPU use for serialising tiny batches, and puts more load on Collectors. It can also hurt the application itself on busy systems. A better answer is to keep sensible batching and make sure the waiting room is emptied properly whenever the process ends.

Queue overflow: the other way to lose data

Abrupt exits are not the only way for items to disappear from the waiting room. The queue has a fixed size, and when it is full, new items are dropped rather than slowing the application down. This is a deliberate protection, because telemetry must never be allowed to stall the business logic. But it means that during a burst of activity, or while the exporter is struggling to reach a slow Collector, items can be lost long before any shutdown happens.

This matters for shutdown because a flush can only send what the queue still holds. If the queue overflowed a minute earlier, the final flush will deliver a clean, complete-looking batch that is missing a chunk of earlier data. Watching the drop counters of your processors, and sizing queues and batch sizes for your peak traffic rather than your average, is therefore part of the same discipline as shutting down properly.

4What Is Lost When a Process Exits Abruptly

A realistic estimate of the damage across a fleet.

Let us put numbers to the problem. Suppose a service runs on fifty instances, and each instance records about two hundred spans per second. The batch processor sends data every five seconds. At any moment, each instance holds roughly the last five seconds of spans in its queue, which is about a thousand spans. That is comfortably under the default queue size, so nothing has been dropped yet, but all thousand are waiting.

1,000
spans waiting in each instance at any moment
50
instances replaced during a rolling update
50,000
spans lost in the worst case of one careless rollout

During a rolling deployment, all fifty instances are stopped and replaced, one after another. If each one is stopped abruptly, up to a thousand spans disappear from each, giving a worst-case loss of fifty thousand spans in a single deployment. A team that deploys ten times a day could lose half a million spans daily, always from the end of each instance’s life, and nobody would see an alert.

The metric side of the story

Metrics lose data differently. A metric data point summarises everything that happened in an interval. If an instance exits thirty seconds into a sixty second interval, the half interval of increments is lost. With cumulative reporting, the next instance starts a new series from zero, and the lost half minute simply never appears in any total. With delta reporting, the lost interval is a permanent hole in the sum, because delta data points are never repaired by later reports.

SignalWhat waits in memoryTypical loss on abrupt exit
TracesFinished spans in the queue, plus any spans not yet endedUp to about one schedule delay of spans, and all open spans
MetricsAggregated values since the last exportUp to about one export interval of activity
LogsLog records in the queueUp to about one schedule delay of log lines

The cost is invisible but real

These losses distort the picture in subtle ways. Request counts around deployment times dip slightly, and graphs show small valleys exactly when new versions roll out, which can be mistaken for real changes in traffic. Error spans from instances that were struggling just before they were replaced never appear, so a problem that caused the replacement is hidden. Traces of long requests that were in flight when the instance stopped look incomplete or absent.

i
A Useful Habit

If your charts show small dips at every deployment, do not assume it is traffic. Check whether instances are flushing their telemetry before they exit. It is one of the most common causes of mysterious gaps.

Why the missing data is biased towards trouble

The loss is not spread evenly across time. Instances are often replaced precisely because something was wrong with them: they were slow, they ran out of memory, or they started failing health checks. That means the final seconds of an instance’s life are disproportionately full of errors and slow requests. Imagine an instance that normally returns errors for one percent of requests, but in its final five seconds before an automatic restart, the error rate climbs to forty percent. If those final thousand spans are lost, the surviving data shows an instance that looked mostly healthy.

This produces a quiet optimism bias. The error rate you see in your dashboards is lower than the real error rate, because the failing moments are the ones that disappear. The effect is small for any single instance, but across a fleet that restarts regularly, it can hide real instability for a long time. Fixing shutdown does not only recover missing counts. It removes a systematic tendency for your data to look better than your system really is.

5Flush and Shutdown: Two Different Verbs

They sound similar, but they do different jobs and are used at different times.

The OpenTelemetry SDK specification gives each provider two related operations. The first is usually called force flush, and the second is called shutdown. People often use the words interchangeably, which leads to bugs. Understanding the difference is one of the most useful things you can learn about the SDK lifecycle.

Operation 1

Force flush

Asks the SDK to send everything it is currently holding right now, and to report back when that is finished or when a time limit passes. The SDK keeps running afterwards and continues to accept new telemetry.

Operation 2

Shutdown

Flushes remaining data and then stops the SDK for good, releasing resources such as threads and network connections. After shutdown, new telemetry is ignored.

AspectForce flushShutdown
Sends pending dataYesYes, as its first step
Keeps the SDK usable afterwardsYesNo
Releases threads and connectionsNoYes
How often it may be calledAs many times as you likeIntended to be called once
Typical moment to useEnd of a short task, before a pause or freezeFinal step before the process exits
Simple Analogy

Flushing is like emptying the post box into the van while the post office stays open for more letters. Shutdown is like emptying the box one last time, locking the door, and handing back the keys. You can empty the box as often as you like, but you can only lock up once.

Both are bounded by time

Both operations are meant to finish or give up within a limited time. A network that is slow or a backend that is down must not be allowed to freeze the application indefinitely. The specification therefore expects these calls to complete, fail, or time out in a reasonable period, and many SDKs accept an explicit timeout so that you can choose your own limit.

!
Common Confusion

Calling shutdown at the end of every request or every function invocation is a mistake. After the first call, the SDK is stopped and later telemetry silently disappears. Use flush for repeated delivery points and reserve shutdown for the true end of the process.

6What Shutdown Actually Does, Step by Step

The orderly sequence that turns a running pipeline into a quiet, empty one.

Shutdown is not a single action. It is a chain of steps passed down from the provider to its processors, and from the processors to the exporters. Each layer is responsible for finishing its own work and then telling the layer below it to do the same. When the chain completes, nothing is left in memory and no background work is running.

1

The provider stops accepting new telemetry

From this point, new spans, measurements, and log records are ignored. This prevents the queue from refilling while it is being emptied.

2

Processors flush what they hold

Each processor sends its queued items to the exporter, in as many batches as needed. Metric readers perform a last collection of aggregated values and send it.

3

Exporters finish in-flight requests

Any network call already underway is given time to complete, so the final batch is not cut off in the middle.

4

Exporters release their resources

Connections are closed and background threads or tasks are stopped, which allows the process to exit cleanly.

5

The call returns a result

The caller learns whether shutdown succeeded, failed, or timed out. This result should be logged, since it tells you whether the final data really left.

Simple Analogy

Closing a shop at the end of the day follows the same order. First you stop letting new customers in. Then you serve those already at the counter. Then you count the till and bank the money. Finally you switch off the lights and lock the door. Doing these in a different order, such as locking the door before the last customer has paid, causes trouble.

What shutdown cannot do

Shutdown can only send what exists. It cannot send spans that have not ended yet, because unfinished spans are not complete records. It cannot save data that was already dropped because the queue overflowed. And it cannot succeed if the network or the backend is unreachable for the whole of the time limit. Knowing these limits helps you design around them rather than hoping shutdown will fix everything.

i
Spans Must End First

A span that is still open when shutdown runs is not exported. Any long-running operation that is cut off should end its span, ideally recording the reason, before the SDK is shut down.

What happens if shutdown is called twice

In real programs, more than one piece of code often believes it is responsible for shutting down the SDK. A framework hook may do it, an exit handler may do it, and the main function may do it as well. The specification says shutdown should be called only once, and that later calls should not do harm but may simply report failure or do nothing. That is helpful, because a second call is usually harmless.

It is still wise to give the responsibility to exactly one place. Two competing owners make it hard to know which call used the time budget and which one saw the result. A single, clearly named shutdown routine, called from the application’s single exit path, keeps the order predictable and makes the logged outcome trustworthy.

7Short-Lived Processes: Jobs, Scripts, and Command-Line Tools

When the whole program runs for less time than one batch delay.

The most dramatic victims of missing shutdown logic are programs that run briefly. A command-line tool might finish in two seconds. A scheduled job might run for a minute and exit. A migration script might process a file and stop. With a five second batch delay and a sixty second metric interval, such a program may finish before the first batch is ever sent, and it can end with every span, metric, and log record still in memory.

Scheduled batch jobs

A nightly job processes a day of orders and exits. Without a flush at the end, its traces and metrics never appear, and the team thinks the job did not run, or runs with no problems, because no data arrived either way.

Command-line tools

A deployment tool that talks to several services records spans for each step. If it exits right after the last step, the final and most informative spans, such as the failed step, are lost.

Continuous integration steps

Test runners and build steps that emit telemetry about durations and failures often run inside containers that are destroyed immediately after the step ends. Without an explicit flush, the telemetry about the build never leaves the container.

Data pipeline tasks

Short extract, transform, and load tasks in workflow engines may live for seconds. Their metrics about rows processed and their trace of the run are lost unless the task flushes before returning.

The rule for short-lived programs

The rule is simple and strict. Before the program exits, whether it succeeds or fails, it must shut down the telemetry providers and wait for the result. This includes failure paths. A program that crashes with an unhandled error needs a handler that still performs shutdown, because the failure is exactly the case you most want to see.

1

Do the work inside a root span

Wrap the program’s main work in a span so that the whole run is visible as one trace with its outcome recorded.

2

End the span on success and on failure

Make sure both paths record the status and end the span before anything else happens.

3

Shut down providers last

Call shutdown on the tracer, meter, and logger providers, and wait for them, with a sensible timeout.

4

Exit only after shutdown returns

Return the exit code and terminate the process only once the shutdown call has completed or timed out.

!
Do Not Rely on Luck

A short program that sometimes works because the timer happened to fire before exit has a hidden bug. It will fail on a faster machine or a shorter run. Always shut down explicitly.

8Serverless: When the Environment Gets Frozen

A different kind of ending, where the process does not exit but simply stops running.

Serverless platforms such as function services add a twist. After a function returns its response, the platform may freeze the execution environment instead of destroying it. A frozen environment is paused: background threads, timers, and network activity all stop. If the batch processor’s timer was due to fire a second after the response, it does not fire, and the data waits in a frozen memory until the next invocation wakes the environment, which might be minutes or hours later, or never.

sequenceDiagram
  participant P as Platform
  participant F as Function code
  participant S as Telemetry SDK
  participant B as Backend
  P->>F: Invoke function
  F->>S: Record spans and metrics
  F->>S: Force flush before returning
  S->>B: Send pending data
  B-->>S: Acknowledge
  F-->>P: Return response
  Note over P,F: Environment may now be frozen, nothing is left waiting
        
FIG 2 — Flushing before the response is returned, so a freeze cannot trap data

Flush, do not shut down

In serverless, the same environment may be reused for many invocations, so the SDK must stay alive between them. Calling shutdown at the end of one invocation would stop the SDK, and every later invocation in the same environment would lose its telemetry. The correct tool is force flush, called at the end of each invocation before the response is returned. It empties the waiting room but leaves the SDK ready for the next call.

Time is billed

Flushing adds to the duration of every invocation, since the function must wait for the network call to complete before it can return. On platforms that charge by execution time, this has a direct cost, and it also affects the response time seen by callers. Some teams reduce the impact by sending telemetry to a nearby Collector or an extension running alongside the function, which accepts the data quickly and forwards it in the background, so the function only waits for a very short local handoff.

ApproachEffect on telemetryEffect on latency and cost
No flush at allFrequent loss, delayed or missing dataNo added time, but unreliable data
Flush directly to a remote backendReliable deliveryAdds network round trip time to every call
Flush to a local Collector or extensionReliable handoff, background forwardingSmall, because the handoff is local and fast
i
A Mental Model

In serverless, treat the moment before returning as the end of the world for that invocation. Anything not delivered by then may never be delivered, so deliver it before you answer.

9Long-Running Services: Signals and Orchestration

How a service is told to stop, and what it should do in the time it is given.

A web service might run for weeks, but it still ends eventually, and the way it ends is almost always through a signal. When an orchestrator such as Kubernetes wants to remove an instance, it sends a termination signal to the main process, waits for a grace period, and then forcibly kills the process if it is still running. The termination signal is your opportunity to shut down cleanly. If the process ignores it, the forced kill follows and any waiting telemetry is destroyed.

1

The instance is marked as terminating

The orchestrator removes it from the list of instances that receive new traffic, so new requests stop arriving after a short delay.

2

A pre-stop step may run

If configured, a short command or pause runs first. Many teams use it to give load balancers time to notice that the instance is leaving.

3

The termination signal is sent

The main process receives the signal and the grace period countdown, commonly thirty seconds by default, is running.

4

The application stops and drains

It stops accepting new work and lets in-flight requests finish, ending their spans as they complete.

5

Telemetry is flushed and shut down

The providers send everything they hold and release their resources, within a strict time budget.

6

The process exits, or is killed

If everything finished in time, the process exits on its own. If the grace period ran out, a forced kill ends it immediately.

Budgeting the grace period

The grace period is a shared budget. Draining requests, flushing telemetry, closing database connections, and any other cleanup must all fit inside it. A common planning approach with a thirty second period is to give request draining about fifteen seconds, telemetry shutdown about five seconds, and leave the rest as a safety margin. If one stage overruns, the later stages are squeezed, and the telemetry stage is usually the last in line, which makes it the first casualty.

30 s
typical default grace period
15 s
example budget for draining requests
5 s
example budget for telemetry shutdown
!
The Hidden Signal Trap

If your container starts the application through a shell script that does not pass signals on, the application never receives the termination signal at all. It only receives the final forced kill. Make sure the application process is the one that receives the signal, or that the wrapper forwards it.

The race with the load balancer

There is a subtle timing problem in the first steps of termination. When an instance is marked as terminating, the systems that route traffic need a moment to notice and stop sending requests to it. During that short window, new requests can still arrive at an instance that has already received its termination signal. If the application stops its server the instant the signal arrives, those late requests fail, and the telemetry about them is either missing or full of connection errors.

Many teams handle this with a brief pause before the application begins to stop, often implemented as a pre-stop step that simply waits a few seconds. During the pause, the instance keeps serving normally while traffic drains away on its own. Only after the pause does the real drain begin. This pause also takes part of the grace period, so it must be included in the time budget. A thirty second grace period with a five second pause, a fifteen second drain, and a five second telemetry shutdown leaves only a five second margin, which is why budgets should be written down and checked, not assumed.

10Order Matters: Shut Down Telemetry Last

The sequence of stopping things decides whether the final spans exist at all.

It is tempting to shut down telemetry first, because it feels like a background concern, and then take care of the real application. This is backwards. If the providers are shut down while requests are still being served, every span, metric, and log record created after that moment is ignored. The in-flight requests, which are usually the most interesting ones during a shutdown, produce no telemetry at all.

flowchart TD
  A["Receive termination signal"] --> B["Stop accepting new requests"]
  B --> C["Wait for in-flight requests to finish"]
  C --> D["End any remaining open spans"]
  D --> E["Force flush all providers"]
  E --> F["Shut down all providers"]
  F --> G["Close other resources and exit"]
        
FIG 3 — A safe order that lets in-flight work produce telemetry before the SDK stops
Rule 1

Stop intake first

No new work should start once shutdown begins, otherwise the queue keeps refilling and the finish line keeps moving.

Rule 2

Let work finish

In-flight requests need time to complete so that their spans end properly and carry real results.

Rule 3

Telemetry goes last

Providers stay alive until everything that produces telemetry has stopped.

Rule 4

Dependencies outlive you

The Collector or agent that receives your data must still be running while you flush. Shutting it down first leaves your final batch with nowhere to go.

The sidecar ordering problem

Many deployments run a Collector or agent next to the application. When both are told to stop at the same time, the Collector may exit before the application has finished its final flush. The result is a failed export during shutdown, even though the application did everything right. The remedy is to keep the Collector alive for slightly longer than the application, using start-up and termination ordering features of the platform, or a short delay in the Collector’s own stop sequence.

Simple Analogy

Closing a restaurant in the right order means the cooks finish the last orders before the dishwasher is switched off, and the delivery driver is still waiting outside when the last parcel is ready. Switching off the driver first leaves the last parcel with no way out.

i
One-Line Rule

Stop the producers, then flush the pipe, and only then close the pipe. Telemetry should be the last thing alive in the process and the receiver should be the last thing alive outside it.

11Timeouts and Retries: Do Not Hang Forever

A shutdown that never finishes is nearly as bad as one that never starts.

Exporters talk to the network, and networks fail. A Collector may be down, a connection may be slow, or a backend may be rejecting requests. During normal operation, exporters deal with this by retrying failed sends after a growing delay. This is sensible when time is plentiful, but during shutdown time is exactly what you lack. A retry loop that waits thirty seconds between attempts can consume the entire grace period and still end in a forced kill.

Setting

Export timeout

The longest a single export attempt may take. A very long timeout lets a stuck connection hold up shutdown for its full duration.

Setting

Shutdown timeout

The overall budget you give to flushing and closing. Choose it so that it fits inside the platform’s grace period with a margin.

Behaviour

Retry with backoff

Exporters wait longer after each failure. During shutdown this means a failing backend can eat the whole budget in a few attempts.

Behaviour

Give up gracefully

When the budget is exhausted, shutdown should abandon remaining data, report that it did, and let the process exit rather than block it.

Choosing a shutdown budget

A good budget is long enough to send a normal final batch, which usually takes well under a second on a healthy network, and short enough that the process can never be blocked by an outage. For many services, a few seconds is plenty. For batch jobs that produce a lot of data at the end, a longer budget may be justified. The right value comes from measuring how long a normal final flush takes and adding a safety margin, not from guessing.

1

Measure a normal flush

Record how long a final flush takes under healthy conditions, using test runs or staging environments.

2

Multiply for safety

Allow several times the normal duration to cover a slower network, larger batches, and a retry or two.

3

Fit it inside the grace period

Check that the total of draining, flushing, and other cleanup stays comfortably below the platform’s limit.

4

Log the outcome

Write a line that records whether shutdown succeeded, failed, or timed out, so you can see over time whether data is being abandoned.

!
A Shutdown That Hangs

If shutdown has no timeout, a single unreachable Collector can prevent a process from ever exiting on its own. Orchestrators then wait out the whole grace period and kill the process anyway, so you lose both the data and the speed of the rollout.

When the backend itself is the problem

Sometimes the reason shutdown is slow has nothing to do with the application. The Collector or backend may be overloaded or down. In that situation, exporters have probably been failing for a while, and their queues may already be full, with newer items being dropped to make room. A graceful shutdown at that point can only try to send data that the pipeline could not send during normal operation either, so it will usually fail within the time budget.

The lesson is that shutdown is not a magic recovery tool. It is the final step of a pipeline that must be healthy throughout its life. Teams that need stronger guarantees for outage periods use a local agent on each machine that accepts data quickly and holds it on disk until the remote side recovers. They also record how many items were abandoned at shutdown, so that a rise in abandoned data becomes a visible signal about the health of the whole pipeline rather than a private secret of each process.

12Language and Runtime Notes

The concept is universal, but the details differ from one ecosystem to the next.

Every OpenTelemetry SDK implements the same idea of flush and shutdown, but they differ in how much the SDK does for you automatically and how much you must wire up yourself. Because these details change with versions, the safest approach is to treat the following as a map of what to look for rather than a guarantee, and to confirm the exact behaviour in the documentation of your SDK.

Pattern A

Automatic exit hooks

Some SDKs register a handler that runs when the program exits normally, calling shutdown for you. This covers simple cases but usually does not cover forced kills or crashes that skip the handler.

Pattern B

Explicit shutdown calls

Other SDKs expect you to call shutdown yourself, often as the last step of the main function or inside a deferred cleanup. Forgetting it means silent loss.

Pattern C

Asynchronous completion

In runtimes built on promises or tasks, shutdown returns something that must be awaited. Calling it without waiting lets the process exit before the work is done.

Pattern D

Agents and auto-instrumentation

Agents that attach to an application often install their own exit hooks and manage the providers for you. You may still need to handle signals and grace periods around them.

Pattern E

Context with deadline

Some ecosystems pass a context object that carries a deadline into shutdown, which gives you a natural way to enforce a timeout.

Pattern F

Disposal patterns

In ecosystems with disposable objects, providers are expected to be disposed at the end of their life, and disposal triggers the flush and shutdown.

A checklist for any language

Whatever language you use, ask the same questions. Does the SDK register an exit hook, and does it run in my deployment? Is there an explicit shutdown call, and who calls it? Does the call return something I must wait for? Can I give it a timeout? What happens when my process receives a termination signal? What happens when my process crashes? The answers to these questions, written down once per service template, prevent the majority of shutdown bugs.

QuestionWhy it matters
Is an exit hook registered by default?Tells you whether normal exits are already covered
Is there a way to flush without shutting down?Needed for serverless and for checkpoints in long tasks
Is the call blocking or asynchronous?Decides whether you must wait for completion
Can a timeout be set?Prevents a hung shutdown from blocking exit
Are signals forwarded to the application?Without them, no graceful path ever starts
!
Exit Hooks Are Not Magic

Automatic exit handlers only run if the process exits through a normal path. A forced kill, a crash in native code, or an out-of-memory termination skips them. Do not assume that the presence of an exit hook means your telemetry is safe in every situation.

13Traces, Metrics, and Logs at Shutdown

Each signal has its own quirks when the end arrives.

Although flush and shutdown apply to all three signals, each one behaves a little differently, and knowing the differences helps you decide what to expect after a clean shutdown. A common mistake is to flush the tracer provider but forget the meter provider and the logger provider, leaving two of the three signals exposed.

Traces

For traces, only finished spans are exported. A span that has started but not ended is still being built, and it is not in the queue. This means that a long-running operation interrupted by shutdown will leave no trace at all unless it ends its span. It is good practice for code that is interrupted to end its span with an error status describing the interruption, so that the trace tells the story of what was cut short.

Metrics

For metrics, the periodic reader normally performs one final collection when it is shut down, gathering the current aggregated values and sending them. This is why a clean shutdown recovers the partial interval at the end, which would otherwise be lost. With cumulative reporting, the final data point carries the latest totals. With delta reporting, it carries the activity since the last export, which is precisely the part that would have been lost.

Logs

For logs, the batch processor holds records in its queue until the next batch. The last records before an exit are usually the most important ones, such as a fatal error or a message announcing the shutdown itself. A clean shutdown sends them, while a kill leaves them behind. Teams that also write logs to standard output often keep that path as a second copy, because it is collected by the platform independently of the SDK.

SignalWhat a clean shutdown recoversWhat it cannot recover
TracesAll finished spans waiting in the queueSpans that never ended, and spans dropped by a full queue
MetricsThe final partial interval, through a last collectionIntervals already lost before this process started
LogsRecords waiting in the queue, including the last onesRecords dropped by a full queue
Simple Analogy

Closing a business for the year involves three tasks: filing the finished invoices, counting the final stock, and posting the last letters. A tidy owner does all three. A hurried owner might post the letters and forget the stock count, leaving the books incomplete in a way that is hard to notice later.

i
Do All Three

Make the shutdown routine cover the tracer provider, the meter provider, and the logger provider together, and wait for all three to complete. A single helper that does this for the whole application removes the chance of forgetting one.

Seeing the loss with self-observability

You cannot fix what you cannot see, and lost telemetry is invisible by nature. Fortunately, the telemetry pipeline can report on itself. SDKs and Collectors expose their own metrics about how many items they have received, how many they have sent, how many are waiting in queues, and how many were dropped or failed to send. The exact names and the level of detail differ between languages and versions, and the conventions for them are still maturing, so check what your own components offer.

Treat these self-metrics as a first-class part of your dashboards. A steady, small queue and a flat line of zero drops are signs of a healthy pipeline. A queue that climbs, a rising count of failed exports, or a burst of dropped items at deployment times are signs that something in the ending is going wrong. Because these numbers come from the pipeline itself, they reveal losses that no ordinary application metric could ever show.

14The Collector Side: Draining the Pipeline

The same problem exists one hop later, inside the Collector itself.

The application is not the only place where data waits. The OpenTelemetry Collector has its own batching, its own queues in front of exporters, and its own retry logic. When a Collector is stopped, any data inside it that has not yet been sent shares the same fate as data inside an application. A perfect shutdown in the application is wasted if the Collector then loses the data a moment later.

flowchart LR
  A["Applications flush on shutdown"] --> B["Agent Collector batches and queues"]
  B --> C["Gateway Collector batches and queues"]
  C --> D["Backend"]
  E["Stop order: applications, then agents, then gateways"] -.-> A
        
FIG 4 — Each hop holds data, so stopping must flow from the edge inward
Collector part

Receivers stop first

On shutdown, the Collector stops its receivers first so that no new data enters, then lets the rest of the pipeline drain.

Collector part

Batch processor flushes

The batch processor sends what it holds when the pipeline shuts down, so a clean stop recovers its partial batch.

Collector part

Exporter queues

Exporters often have in-memory sending queues. A clean stop attempts to drain them, but a forced kill destroys whatever they hold.

Collector part

Persistent queues

Where supported, a queue can be stored on disk so that unsent data survives a restart and is sent when the Collector returns.

Stop from the edge inward

In a layered design with agents near the applications and gateways further away, the order of stopping matters just as it does inside an application. Applications must finish first so that they can flush into the agents. Agents must stop next so that they can pass their data to gateways. Gateways stop last. If a gateway is replaced before the agents that feed it, those agents will fail their exports, retry, and may lose data when their own queues overflow.

Rolling updates of the Collector tier

Collectors are usually deployed with several replicas, and updates replace them one at a time. This works well for stateless forwarding, but stateful stages, such as tail sampling that buffers whole traces in memory, lose their buffered data when an instance is replaced. Plan these updates carefully, replace one instance at a time, and accept that a small number of traces may be incomplete during the change.

!
Do Not Forget the Middle

Teams often perfect the application’s shutdown and then restart Collectors carelessly, so the final data is lost one hop later. A pipeline is only as reliable as its weakest stopping procedure.

15When You Cannot Flush: Crashes and Hard Kills

Some endings give no warning, so the design must limit the damage in advance.

Graceful shutdown depends on the process having a chance to run code at the end. Some endings give no such chance. A process killed by a forced signal after the grace period, one terminated by the operating system for using too much memory, a machine that loses power, or a crash in native code all skip any cleanup. In those cases no flush is possible, and whatever was waiting in memory is gone.

Kind of endingCan the process run cleanup?Typical telemetry outcome
Normal exitYesFull flush is possible
Termination signal with handlerYes, within the grace periodFull flush if the budget is respected
Unhandled error in managed codeSometimes, through a global handlerFlush possible if the handler is installed
Forced kill after grace periodNoEverything in memory is lost
Out-of-memory terminationNoEverything in memory is lost
Power loss or machine failureNoEverything in memory is lost

Designing for the ending you cannot control

Since you cannot flush in these cases, the goal becomes making the amount of data at risk small, and making it more likely that important data left the process earlier. Several techniques help, and teams usually combine them.

1

Shorten the batch delay for critical signals

A smaller delay shrinks the window of data in memory. Balance it against network and CPU cost, and consider it mainly for services where the last seconds matter greatly.

2

Send to a local agent

Exporting to a nearby Collector on the same machine moves data out of the application’s memory very quickly, and the agent can hold and retry it even if the application is killed.

3

Use persistent queues in the Collector

Storing unsent data on disk protects it against Collector restarts and gives a second line of defence.

4

Install a last-chance error handler

Where the language allows it, install a global handler for unhandled errors that records the error, ends the active span, and flushes before the process dies.

5

Keep a second copy of the most vital logs

Writing important error lines to standard output or a file as well gives the platform a chance to capture them independently of the SDK.

i
Accept What Cannot Be Fixed

Perfect delivery is not achievable. The goal is to make the common endings clean, make the rare endings cheap, and know which situations will always lose a few seconds of data so that nobody is surprised by them.

A worked estimate for a hard kill

Return to the instance that records about two hundred spans per second. With a five second batch delay, a hard kill can destroy up to about a thousand spans. If the team shortens the delay to one second, the exposure falls to about two hundred spans. That is a five-fold reduction, but the price is paid constantly, not only at the end. A time-triggered batch processor that sent twelve batches per minute now sends sixty, which is five times as many network calls, five times as many serialisations, and five times as many small requests for the Collector to handle.

Batch delaySpans at risk on a hard killTime-triggered batches per minute
5 secondsAbout 1,000About 12
1 secondAbout 200About 60
5 seconds plus a local agentSmall, because data leaves the process quicklyAbout 12 to the agent

The comparison shows why a local agent is often the better answer. It lets the application keep a relaxed batch schedule while the data still leaves the process quickly, and the agent, which is much less likely to be killed at a random moment, takes over the job of holding and retrying it.

16Costs, Benefits, and Trade-offs

Doing shutdown well is cheap, but it is not entirely free.

Advantages

  • Preserves the freshest and most valuable telemetry
  • Removes mysterious dips around deployments
  • Makes short jobs and serverless functions visible at all
  • Keeps counts and totals closer to the truth
  • Costs almost nothing in steady state
  • Builds trust in the observability data

Disadvantages / Trade-offs

  • Adds time to shutdown and, in serverless, to every invocation
  • Needs coordination with the platform’s grace period
  • Requires code in every service, or a shared template
  • Can hang the process if no timeout is set
  • Cannot help with hard kills and crashes
  • Needs matching care in the Collector tier

The cost of the alternatives

Teams that skip graceful shutdown sometimes try to compensate by shortening batch delays to a fraction of a second. This reduces the loss window but increases the number of network calls many times over, raises CPU use for serialising small batches, and loads Collectors with a constant stream of tiny requests. Graceful shutdown solves the same problem at the exact moment it occurs, at a tiny fraction of the cost.

StrategyData protectedOngoing cost
Nothing, rely on luckNone of the final batchNone, but data is silently lost
Very short batch delayMost of the data, but not allHigh, many small network calls all the time
Graceful shutdown with flushAll of the final batch on clean endingsVery low, one flush per ending
Graceful shutdown plus local agent and persistent queueMost data, even on hard killsModerate, extra component to run
Simple Analogy

Locking the door when you leave takes a few seconds each time. Not locking it saves those seconds but occasionally costs far more. Graceful shutdown is the telemetry equivalent of locking the door: small, habitual, and very cheap compared with what it protects.

Deciding how much effort to invest

Not every service needs the full set of protections. A simple way to decide is to ask two questions. First, how costly is it if the final seconds of this service’s telemetry are missing? A payment service during an incident answers differently from an internal reporting tool. Second, how often does this service end? A service that is replaced a dozen times a day loses far more than one that runs for months.

For almost every service, the basics are worth doing: a shared shutdown routine, signal handling, and a time budget. The heavier measures, such as local agents with persistent queues and shorter batch delays, are best reserved for the services where both answers are high. Starting with the basics everywhere and adding more only where evidence demands it keeps the effort proportionate.

17Security and Privacy Considerations

The end of a process still needs the same care as the middle.

Shutdown looks like a purely operational topic, but it has security aspects worth remembering. During shutdown the process makes a final burst of network calls, sending whatever remains in its queues. Those calls must follow the same rules as any other export: they should use encryption, they should authenticate to the receiving Collector, and they should send only data that is allowed to leave the process.

Concern 1

Final data is sensitive too

The last spans and logs often contain error details, stack traces, and request data. Scrub sensitive values before export, not only during normal operation.

Concern 2

Encryption and credentials

Make sure credentials and certificates are still available at shutdown. Cleanup code that deletes them too early leads to failed exports.

Concern 3

Unbounded blocking

A shutdown without a timeout can be abused, or simply caused by a failing receiver, to keep processes alive and resources locked.

Concern 4

Error messages in logs

Shutdown failures should be logged without printing credentials, endpoints with secrets in them, or the contents of the data that failed to send.

Data left on disk

Persistent queues write unsent telemetry to disk so that it survives restarts. That data may include everything the application exports, so the directory that holds it needs the same protection as any other store of telemetry: restricted permissions, encryption at rest where required, and a clear retention rule so that old unsent data does not accumulate forever.

ANTI-PATTERN-01 Avoid
Problem

A shutdown routine skips redaction and sends raw error details, including request payloads and tokens, to speed up the final flush.

Why It’s Harmful

The final data is often the most detailed, because it describes a failure. Sending it unredacted places private information into long-lived telemetry stores that many people can read.

Correct Approach

Apply the same scrubbing at shutdown as during normal operation, and keep sensitive values out of span attributes and log records in the first place.

18Real-World Scenarios

Where teams most often discover the problem, usually by accident.

The following scenarios describe patterns that appear again and again across the industry. Each one starts with a puzzling symptom and ends with a missing flush as the cause.

The dip at every deployment

A team notices that request counts dip slightly at every release and recover a minute later. They suspect users are leaving during deployments. The real cause is that each replaced instance drops its last batch of spans and its last partial metric interval. After adding proper shutdown handling, the dips disappear.

The nightly job that never shows up

A scheduled report job appears in no dashboard, and its owners assume it is not running. In fact it runs for forty seconds and exits before the first metric interval ends. A final shutdown call makes its trace and metrics appear every night.

The missing crash trace

A service restarts several times a day because of an unhandled error. The traces from before each restart are always missing, so nobody can see which request caused it. A last-chance handler that ends the span, records the error, and flushes reveals the culprit within a day.

The serverless function with no traces

A function shows traces only when it happens to be invoked again soon after, because data is stuck in a frozen environment between invocations. Adding a flush before returning makes every invocation visible immediately.

The autoscaler scale-down gap

When traffic falls in the evening, the platform removes instances. Charts show brief gaps in error counts at those moments, hiding the errors that occurred just before scale-down. Handling the termination signal and flushing fixes the gaps.

The sidecar that left too early

An application flushes correctly but still loses data, because the Collector sidecar is stopped at the same moment and is already gone when the final export happens. Keeping the sidecar alive slightly longer solves it.

The spot instance warning

Cloud providers that sell spare capacity at a discount can reclaim machines with only a short warning, often a couple of minutes. Services running there need to treat that warning as a termination signal, start draining immediately, and flush telemetry within the short window, because after the deadline the machine simply disappears.

The database migration tool

A migration runs for ten minutes and fails on its last step. Its telemetry would have shown exactly which statement failed, but the tool exits with an error code straight away, skipping the cleanup path. Moving the shutdown call into a handler that runs on both success and failure restores the evidence.

The build pipeline with silent steps

Telemetry about build and test durations is missing for fast steps, because the containers running them are destroyed immediately after the step ends. A flush at the end of each step makes the measurements visible and lets the team find slow steps.

i
The Common Thread

In every scenario, the system worked and the configuration was correct. The only thing missing was a deliberate, orderly ending. That is why this problem is so often discovered late.

19Design Patterns and Anti-patterns

Habits that keep endings clean, and traps that cause silent loss.

flowchart TD
  A["What kind of process is this?"] --> B{"Does it run for a long time?"}
  B -->|Yes| C["Handle termination signal, drain, flush, then shut down"]
  B -->|No| D{"Is it a serverless function?"}
  D -->|Yes| E["Force flush before every response, never shut down per call"]
  D -->|No| F["Wrap work in a span, then shut down providers before exit"]
        
FIG 5 — Choosing the right ending for each kind of program

Patterns that work

Pattern 1

One shared shutdown helper

Write a single routine that flushes and shuts down all three providers with a timeout, and reuse it in every service template.

Pattern 2

Signal, drain, flush, close

Follow the same four-step order in every long-running service so that behaviour is predictable and reviewable.

Pattern 3

Budgeted timeouts

Give telemetry shutdown a fixed share of the grace period and write down the numbers next to the deployment settings.

Pattern 4

Flush at natural boundaries

In serverless and in long batch tasks, flush at the end of each unit of work, not only at the very end.

Pattern 5

Outlive your receiver

Arrange that the Collector or agent stops after the applications that feed it.

Anti-patterns to avoid

ANTI-PATTERN-02 Avoid
Problem

Shutdown is called at the end of every serverless invocation or every request, because it seems the safest way to guarantee delivery.

Why It’s Harmful

Shutdown is meant to be used once. After the first call, the SDK stops, and every later piece of telemetry in the same environment is silently ignored.

Correct Approach

Use force flush at repeated delivery points and reserve shutdown for the true end of the process.

ANTI-PATTERN-03 Avoid
Problem

Telemetry providers are shut down first in the stop sequence, before the server has finished serving in-flight requests.

Why It’s Harmful

Everything those requests produce after that moment is ignored, so the spans and logs describing the final seconds of real work never exist.

Correct Approach

Stop intake, drain in-flight work, end open spans, then flush and shut down, so telemetry is the last component alive.

ANTI-PATTERN-04 Avoid
Problem

Shutdown has no timeout, or an extremely long one, and a failing Collector is allowed to keep the process alive during retries.

Why It’s Harmful

The orchestrator waits out the whole grace period and then kills the process anyway. The data is lost and rollouts become slow.

Correct Approach

Choose a budget that fits inside the grace period, abandon remaining data when it runs out, and log that this happened.

ANTI-PATTERN-05 Avoid
Problem

The asynchronous shutdown call is made but not awaited, so the program exits while the final export is still running.

Why It’s Harmful

The process appears to follow the correct procedure, yet the final batch is cut off. The bug is hard to notice because the code looks right.

Correct Approach

Wait for the result of shutdown, check whether it succeeded, and only then allow the process to exit.

20Testing and Monitoring Shutdown Behaviour

Because the failure is silent, you must test for it on purpose.

Shutdown bugs never announce themselves. The system looks healthy during normal operation, and the only symptom is missing data at the end. The only reliable way to find these bugs is to test the ending directly, in environments that behave like production, and to keep watching for signs of loss afterwards.

1

Emit a marker just before stopping

In a test environment, create a span, a metric increment, and a log line with a recognisable marker, then stop the process immediately and check that all three arrive.

2

Test every kind of ending

Try a normal exit, a termination signal, an unhandled error, and a rollout of the real deployment. Each exercises different code.

3

Test with a broken Collector

Make the Collector unreachable and confirm that shutdown finishes within its time budget and logs the abandonment instead of hanging.

4

Compare counts across a deployment

Compare request counts from the application’s own logs or the load balancer with telemetry counts across a rollout. A persistent gap suggests lost data.

5

Watch for drops and failures in production

Monitor the SDK’s and the Collector’s own metrics about queue sizes, dropped items, and failed exports, and alert on sustained non-zero drops.

SymptomLikely cause
Dips in counts at every deploymentInstances exit without flushing their final batch
Short jobs never appearProgram exits before the first batch or metric interval
Serverless traces appear late or neverEnvironment frozen with data still in memory
Shutdown logs show timeoutsCollector unreachable or budget too small
Rollouts take the full grace periodShutdown hanging or signals not reaching the application
Data lost even with correct shutdownCollector stopped too early or has no persistent queue
i
Make It Routine

Include a shutdown test in the checklist for every new service and run it again after SDK upgrades. A minute of testing on each upgrade prevents weeks of mysterious gaps later.

Rehearse a real rollout

Unit tests prove that your shutdown helper works. Only a rehearsal proves that your deployment works. Pick a staging environment that matches production, generate a steady stream of traffic that carries a countable marker, and trigger a rolling update. Afterwards, compare the number of requests your load generator sent with the number your telemetry recorded. The two should agree closely. If the telemetry is short by roughly one batch per instance, you have found a missing flush. If it is short by much more, look for signals that never arrive, grace periods that are too small, or a Collector that stops too early.

Repeat the rehearsal whenever something in the chain changes: a new SDK version, a new base image, a new startup script, or a new Collector release. Each of these can quietly break the path of the termination signal or change a default timeout. A rehearsal takes only a few minutes, and it turns shutdown from a hope into a measured property of your system.

21Frequently Asked Questions

Short answers to the questions people ask most often.

Q1Why does telemetry get lost when a process exits?

Because SDKs batch data in memory and send it later on a timer or when a size limit is reached. If the process ends while data is still waiting, nothing ever sends it.

Q2What is the difference between force flush and shutdown?

Force flush sends pending data now and leaves the SDK running. Shutdown sends pending data and then stops the SDK permanently, releasing its resources. Shutdown is meant to be called once.

Q3Should I call shutdown after every serverless invocation?

No. Use force flush before returning the response. Shutdown would stop the SDK, and later invocations in the same reused environment would lose their telemetry.

Q4Does the SDK shut itself down automatically?

It depends on the language and on whether you use an agent. Some SDKs register an exit hook for normal exits, while others require an explicit call. Even where a hook exists, it does not cover forced kills or crashes.

Q5Are unfinished spans exported at shutdown?

No. Only spans that have ended are exported. Code that is interrupted should end its span, ideally with an error status that explains the interruption, before shutdown runs.

Q6How much data can I lose without flushing?

Roughly one batch delay of spans and logs, often about five seconds, and up to one export interval of metrics, often about a minute, plus any spans not yet ended. The exact amounts depend on your configuration.

Q7What timeout should I use for shutdown?

Measure a normal final flush, allow several times that duration, and make sure it fits inside the platform’s grace period alongside request draining. A few seconds is enough for many services.

Q8Why does my container never receive the termination signal?

Often because the application is started through a shell wrapper that does not forward signals, so only the forced kill reaches it. Make the application the main process or ensure the wrapper passes signals along.

Q9Does the Collector lose data when it is restarted?

It can. A clean stop tries to drain its pipeline, but in-memory queues are lost on a forced kill. Persistent queues and careful stopping order, from applications to agents to gateways, reduce the loss.

Q10Do metrics get a final export at shutdown?

Typically yes. Periodic readers usually perform one last collection and export when they are shut down, which recovers the partial interval at the end. Confirm this for your SDK version.

Q11Can graceful shutdown protect me from crashes and out-of-memory kills?

Not fully. Those endings skip cleanup code. You reduce the damage by shortening batch delays where needed, exporting to a local agent, using persistent queues, and installing a last-chance error handler where possible.

Q12Is it better to shorten the batch delay than to implement shutdown?

No. Shortening the delay shrinks the loss window but raises network and CPU cost all the time, and still loses data. Graceful shutdown handles the ending itself at almost no ongoing cost, and the two can be combined where needed.

22Summary and Key Takeaways

OpenTelemetry keeps recording cheap by batching telemetry in memory and sending it in the background. That design is excellent for performance, but it creates a waiting room where data sits until the next batch goes out. When a process ends before the waiting room is emptied, the data is lost silently. Because the newest data is the most likely to be waiting, the loss falls on exactly the moments you care about most: the final requests, the last errors, and the final partial interval of metrics.

Graceful shutdown and flushing are the discipline that closes this gap. Force flush empties the waiting room at natural boundaries, such as the end of a serverless call. Shutdown empties it one last time and then stops the SDK for good. Done in the right order, with intake stopped first, in-flight work drained, spans ended, and telemetry shut down last with a firm time budget, they preserve the story of every process’s final moments. Extend the same care to Collectors, plan for the endings that cannot be graceful, and test the ending on purpose. Like locking the door when you leave, it takes a moment and protects a great deal.

Key Takeaways

  • Telemetry waits in memory — batching holds spans, metrics, and logs for seconds or a full interval, so anything still waiting when a process exits is silently lost.
  • The newest data matters most — the final requests, errors, and partial metric interval are the most likely to be lost and the most valuable during an investigation.
  • Flush and shutdown are different — force flush sends pending data and keeps the SDK running, while shutdown flushes and then stops it permanently and should be called only once.
  • Order is everything — stop intake, drain in-flight work, end open spans, flush, and shut down telemetry last, with the Collector staying alive until after the application.
  • Always set a time budget — fit the shutdown timeout inside the platform’s grace period so a failing Collector cannot hang the process or force a kill.
  • Short-lived and serverless need special care — jobs must shut down before exiting, and serverless functions must flush before returning, because the environment may be frozen.
  • Plan for the ending you cannot control — use local agents, persistent queues, last-chance handlers, and tested shutdown paths, and verify with a marker test that the final data really arrives.