OOM Killer Behavior in Docker
How the Linux kernel decides which process to sacrifice when memory runs out inside a container — and why understanding this decision is the difference between a graceful restart and a baffling, silent crash.
Every so often, a container that was running perfectly well simply vanishes — no graceful shutdown log line, no clean error message from the application, just an abrupt stop and an exit code that leaves engineers staring at their terminal wondering what happened. More often than not, the culprit is a Linux kernel mechanism called the Out-Of-Memory killer, or OOM killer, quietly doing exactly what it was designed to do: picking a process to forcibly terminate the instant available memory runs out, in order to keep the rest of the system from grinding to a complete halt. In a containerized world, this same decades-old kernel mechanism operates at two different scales simultaneously — it can act within a single container’s memory limit, or across the entire host — and understanding exactly how Docker plugs into it is essential for anyone who has ever been confused by a container that died with no apparent cause. This guide explains what the OOM killer actually is, how it decides what to kill, and how to work with it deliberately rather than being surprised by it.
Chapter One
ACore Concepts
Before diving into how Docker interacts with it, it’s worth understanding why the OOM killer exists at all and what problem it was originally built to solve.
Why the kernel needs an “OOM killer” in the first place
Linux, like most operating systems, allows programs to request more memory than is immediately backed by physical RAM, a practice called overcommit, on the reasonable bet that not every process will actually use everything it asked for at the same moment. Most of the time this bet pays off. Occasionally, though, enough processes simultaneously try to actually use the memory they requested that the system genuinely runs out — no free RAM, no free swap space, nothing left to give. At that exact moment, the kernel faces an impossible situation: some process’s memory request cannot be satisfied, and it must intervene immediately or risk the entire system becoming unresponsive. The OOM killer is the kernel’s answer: rather than let the whole system stall or corrupt data, it selects one process, terminates it forcibly and instantly, reclaims its memory, and lets everything else continue running.
Picture a lifeboat that is over capacity and starting to take on water. There is no time for a vote or a careful discussion — someone has to go overboard immediately or everyone drowns. The OOM killer is the crew member forced to make that split-second call: pick one passenger to remove right now, based on a quick set of rules, so that the boat — the rest of the system — stays afloat for everyone else. It is not a graceful process, and it was never designed to be one; it exists specifically for the moment when graceful options have already run out.
Two different scopes: container-level and host-level
In a Docker environment, out-of-memory conditions can occur at two distinct scopes, and distinguishing between them is the single most important concept in this entire topic. A container-level OOM event happens when a specific container hits the memory limit Docker configured for it, even if the host machine as a whole still has plenty of free memory elsewhere. A host-level OOM event happens when the entire machine runs out of memory across all processes combined, regardless of any individual container’s configured limit. Both trigger the same underlying kernel mechanism, but they are scoped very differently, and confusing the two is a common source of misdiagnosed incidents.
A container hitting its own memory limit is, from the kernel’s point of view, almost identical to the whole host running out of memory — both trigger an OOM kill — but the trigger condition and the pool of candidate processes considered for termination are scoped differently: to the one container’s cgroup in the first case, and to the whole system in the second.
Where this connects to cgroups
The container-level scoping of OOM events is made possible by the same cgroup memory controller responsible for enforcing a container’s memory limit in the first place. When a container is started with a memory limit, Docker configures that limit through the cgroup memory controller, and it is that same controller which detects when the container’s cumulative memory usage would exceed its configured ceiling and triggers a scoped OOM kill confined to that container’s own cgroup, rather than triggering the broader, host-wide OOM killer.
Chapter Two
BArchitecture & Components
Several kernel and Docker components cooperate to detect memory pressure, choose a victim, and terminate it, all within a matter of milliseconds.
flowchart TB
subgraph HOST["Docker Host"]
MM["Kernel Memory Manager"]
HOOM["Host-Wide OOM Killer"]
subgraph C1["Container A cgroup (limit: 512MB)"]
CG1["cgroup Memory Controller"]
COOM["Container-Scoped OOM Killer"]
P1["Process using memory"]
end
end
P1 -->|exceeds cgroup limit| CG1
CG1 -->|triggers| COOM
COOM -->|SIGKILL| P1
MM -->|host-wide exhaustion| HOOM
HOOM -->|selects victim across all processes| P1
Global OOM Killer
The kernel’s system-wide mechanism, active whenever the entire host runs out of memory across every process, regardless of container boundaries or configured limits.
cgroup Memory Controller
Tracks memory usage for each cgroup (and therefore each container) and triggers a scoped OOM kill confined to that cgroup the moment its configured limit would be exceeded.
OOM Score Calculation
The kernel’s badness-scoring heuristic, which ranks every candidate process by estimated memory impact and adjustable priority hints to decide which one to terminate first.
Memory Limit Configuration
The interface through which an operator specifies a container’s maximum memory, which Docker translates into the corresponding cgroup memory controller settings at container start.
Container Exit Status Reporting
The mechanism by which Docker surfaces whether a stopped container was specifically killed due to an out-of-memory condition, distinct from an ordinary crash or a normal exit.
OOM Score Adjustment
A per-process adjustment value that shifts a process’s likelihood of being selected as the OOM killer’s victim relative to others, usable both at the host level and, with more limited effect, within a container.
Where the two scopes intersect
A subtle but important architectural point: a container-level OOM kill, confined to one container’s cgroup, happens entirely independently of whether the host as a whole is under memory pressure. A container can be killed for exceeding its own 512-megabyte limit on a host that otherwise has many gigabytes of free memory sitting idle, precisely because the limit exists to protect other containers and the host from that one container’s potential misbehavior, not because the physical memory itself was actually exhausted.
Chapter Three
CInternal Working
The mechanics of exactly how the kernel picks a victim, and what happens to that process in its final moments.
How the kernel scores candidate processes
When the OOM killer needs to choose a victim, it does not select randomly. It calculates a “badness” score for every candidate process, weighted heavily by how much memory that process is currently using — a process consuming a large share of the available memory is, all else being equal, a more attractive candidate for termination, since killing it frees up the most memory in a single action. This score can be further adjusted, either upward (making a process more likely to be chosen) or downward (making it less likely), through an explicit adjustment value that administrators or container runtimes can set for specific processes considered more or less critical.
What happens in the final moment
Once a victim is selected, the kernel sends it the SIGKILL signal, the most forceful and immediate process-termination signal Linux offers — one that cannot be caught, ignored, or handled by the target process in any way. This is a deliberate design choice: the entire point of the OOM killer is to reclaim memory as fast as possible, so it cannot afford to wait for a process to perform its own graceful cleanup, flush buffers, or close connections cleanly. The targeted process simply stops existing at the instant the signal is delivered, with no opportunity to log a final message, save unsaved state, or shut down its own child processes in an orderly way.
sequenceDiagram
participant P as Container Process
participant CG as cgroup Memory Controller
participant K as Kernel OOM Logic
participant OS as Docker Daemon
P->>CG: memory allocation request
CG->>CG: check against configured limit
alt Under limit
CG-->>P: allocation granted
else Would exceed limit
CG->>K: trigger OOM kill (scoped to this cgroup)
K->>K: score candidate processes in cgroup
K->>P: SIGKILL (no graceful handling possible)
K->>OS: kill event recorded
OS->>OS: container exits, marked OOMKilled
end
Why this can happen even with memory seemingly “available”
A common point of confusion: a container can be OOM-killed even when its process appeared, moments earlier, to be using noticeably less memory than its configured limit. This happens because memory usage can spike very quickly — a single large allocation, a burst of concurrent requests each allocating their own buffers, or a garbage collector temporarily holding more memory than usual right before cleanup — and the kernel’s accounting is checked at the moment of allocation, not on some slower periodic interval. By the time any external monitoring tool’s next measurement would have shown the spike, the kill may have already happened.
Chapter Four
DData Flow & Lifecycle
Tracing the full sequence from a container’s normal operation through to how the kill event is surfaced and, in orchestrated systems, potentially recovered from.
Normal Operation
The container runs with memory usage tracked continuously by its cgroup memory controller against its configured limit.
Memory Pressure Builds
Usage climbs, whether gradually through a memory leak or suddenly through a burst of legitimate demand, approaching the configured ceiling.
Allocation Would Exceed Limit
A specific memory allocation request, however small, would push cumulative usage past the configured limit, triggering the OOM condition at that exact instant.
Victim Selection and Kill
The kernel scores candidate processes within the container’s cgroup and sends SIGKILL to the selected victim, typically the container’s main process.
Container Exits, Marked OOMKilled
Docker records the container’s exit status, including a specific flag indicating the stop was due to an out-of-memory kill rather than a normal exit or an unrelated crash.
Restart Policy or Orchestrator Reacts
Depending on configuration, Docker’s restart policy or an orchestrator like Kubernetes may automatically restart the container, though repeated OOM kills in quick succession typically indicate an underlying issue rather than one the restart alone resolves.
What the exit code actually communicates
When a container is OOM-killed, Docker typically reports the container’s exit code as 137, a value derived from the convention of adding 128 to the numeric value of the signal that caused termination — in this case, signal 9, which is SIGKILL. This exit code alone does not definitively prove an OOM kill occurred, since SIGKILL can in principle be sent for other reasons, which is why Docker separately exposes a dedicated flag on the container’s inspection output specifically indicating whether the kernel’s OOM killer was the actual cause, rather than requiring an operator to infer it purely from the exit code.
Chapter Five
EAdvantages, Disadvantages & Trade-offs
The OOM killer is a blunt instrument by design, and understanding both sides of that trade-off shapes how teams should configure and respond to it.
Advantages
- Prevents a memory-exhausted host from becoming entirely unresponsive to every process, including critical system services
- Container-scoped OOM kills isolate the damage of a single runaway container from affecting healthy containers on the same host
- Provides a clear, distinctly identifiable exit signature (exit code 137, OOMKilled flag) that simplifies diagnosing memory-related failures
- Requires no application code changes to benefit from — the protection is enforced entirely at the kernel and cgroup level
- Combined with restart policies, allows a system to self-heal automatically from a single transient memory spike
Disadvantages
- Offers zero opportunity for the killed process to perform graceful shutdown, cleanup, or logging before termination
- Can select a process the operator would have preferred to keep running, since scoring is based on memory usage, not business importance
- A memory limit set too low relative to real application needs turns normal operation into recurring, disruptive kills
- Host-level OOM kills can select processes outside any specific container, including host system services, if not carefully isolated
- Diagnosing the root cause after the fact can be difficult, since the killed process leaves no self-generated error trail
Why “no graceful shutdown” is a feature, not an oversight
It is worth explicitly defending a design choice that often frustrates engineers the first time they encounter it: the OOM killer deliberately does not give the victim process a chance to clean up. Any delay introduced to allow graceful shutdown would itself require additional memory and processing time — exactly the resources that are, by definition, already exhausted at the moment the kill is triggered. Building in a grace period would risk the kernel being unable to reclaim memory quickly enough to prevent the broader system-wide stall the OOM killer exists to avoid in the first place, defeating its entire purpose.
Because the kill itself cannot be made gentler without undermining its purpose, the practical control operators actually have is setting an appropriate memory limit in the first place — one generous enough to accommodate genuine, expected peak usage, but tight enough to still protect the rest of the host from a truly runaway process.
Chapter Six
FSecurity
The OOM killer’s availability-protecting role means it intersects directly with denial-of-service risk on shared, multi-tenant Docker hosts.
Memory exhaustion as an attack or failure vector
A container without an explicit memory limit can, whether through a genuine bug or deliberate malicious intent, consume memory without bound until it threatens the entire host, at which point the host-wide OOM killer intervenes and may select any process on the machine as its victim — potentially including other, entirely unrelated containers, or even critical host-level services, not necessarily the offending container itself. This is precisely why configuring an explicit memory limit on every container is considered a baseline security and stability practice on any host running more than one workload: it confines the blast radius of a memory-exhaustion event to the offending container’s own cgroup rather than leaving the outcome to a host-wide, less predictable selection process.
An unlimited container on a shared host is like one apartment in a building with no circuit breaker of its own, wired directly into the building’s main electrical panel. If that one apartment draws too much power, the whole building’s breaker trips, cutting power to every other tenant, rather than just that one unit going dark. A per-container memory limit is the equivalent of giving every apartment its own dedicated circuit breaker, so one unit’s electrical fault only ever affects that unit.
Why the OOM killer is not, by itself, a security control
It is important not to overstate what the OOM killer protects against. It is a reactive availability mechanism, not an access-control or confidentiality mechanism — it does nothing to prevent an attacker from reading sensitive data or escalating privilege, and a well-behaved memory-exhaustion attack designed to specifically target another container’s process (rather than exceed its own limit) is outside what the OOM killer is built to address. Memory limits and the OOM killer belong squarely in the availability corner of the security triad, complementing, but never replacing, the isolation and access-control mechanisms covered by topics like capabilities dropping.
Context
An incident review found that a single unbounded container on a shared host had triggered a host-wide OOM event that killed an unrelated, business-critical container’s process instead of the offending one, causing a wider outage than the root cause alone should have justified.
Decision
Explicit memory limits were made mandatory for every container deployed to shared, multi-tenant hosts, enforced through deployment tooling that rejects container definitions lacking a configured limit.
Consequence
Subsequent memory-exhaustion incidents were confined to the specific offending container’s own cgroup, with no cross-container impact observed in the following incident reviews.
Chapter Seven
GMonitoring, Logging & Metrics
Because the killed process cannot log its own death, observability has to come from outside the process entirely.
Where to actually find the evidence
Since the terminated process is given no opportunity to write its own log line explaining what happened, evidence of an OOM kill has to be gathered from sources outside the process itself: Docker’s own container inspection data, which exposes a specific flag indicating the stop was OOM-related; the kernel’s own system log, which records OOM killer activity including which process was selected and why; and, for container-scoped kills specifically, the cgroup memory controller’s own accounting data showing usage approaching the configured limit in the moments before the kill.
| Source | What It Tells You | When to Check It |
|---|---|---|
| Docker container inspection | Whether this specific container’s stop was flagged as OOM-related | Immediately after noticing an unexpected container stop |
| Kernel system log | Which process was selected, its calculated score, and the memory state at kill time | When Docker’s flag confirms an OOM kill and root cause is needed |
| cgroup memory accounting | Usage trend leading up to the limit being hit | To distinguish a gradual leak from a sudden spike |
| Application-level metrics (pre-kill) | Business context for what the process was doing before the spike | To correlate the kill with a specific request pattern or workload |
Building alerts that catch this before it becomes a mystery
The most effective monitoring setups do not wait to detect an OOM kill after the fact; they alert on memory usage approaching a container’s configured limit well before the kill actually happens, giving operators a chance to intervene or at least be forewarned. Pairing this proactive threshold alert with a separate, explicit alert specifically for containers that have already been OOM-killed — rather than relying on a generic “container exited unexpectedly” alert — lets on-call engineers immediately understand the failure category without needing to dig through logs first.
A container with a memory limit set too low relative to its genuine needs, combined with an automatic restart policy, can enter a repeating cycle of starting, quickly hitting its limit, being OOM-killed, and restarting again — a pattern that can look like intermittent availability from the outside while actually being a completely predictable, continuous failure. Monitoring should specifically flag repeated OOM kills of the same container within a short window as a distinct, higher-priority signal rather than treating each occurrence as an isolated event.
Chapter Eight
HDesign Patterns & Anti-patterns
Recurring approaches for working with, rather than being ambushed by, the OOM killer’s behavior in production systems.
Benchmarked Memory Limits
Setting a container’s memory limit based on measured peak usage under realistic load testing, with deliberate headroom, rather than an arbitrary round number chosen without evidence.
Application-Level Memory Guardrails
Configuring the application’s own runtime (where its language or framework supports it) to stay comfortably under the container’s hard limit, so the application can degrade or shed load on its own terms before the kernel is forced to intervene.
Separate Limits for Sidecar and Main Process
Giving each container in a multi-container pod its own independently sized memory limit, so a runaway sidecar cannot consume the memory budget intended for the primary application.
No Memory Limit At All
Leaving a container’s memory unbounded, deferring the entire problem to the host-wide OOM killer, which offers no isolation guarantee about which process ultimately pays the price.
Treating Every OOM Kill as “Just Add More Memory”
Repeatedly raising a container’s memory limit in response to OOM kills without investigating whether the underlying cause is a genuine leak that will eventually exceed any limit given enough time.
Pattern in practice: memory limit as a leak detector
Some teams deliberately set a container’s memory limit only modestly above its measured normal peak usage, specifically so that a slow memory leak triggers a visible, alertable OOM kill relatively early — trading a small amount of resilience against legitimate usage spikes for much faster detection of a genuine application bug that would otherwise silently grow toward a far more disruptive, unpredictable failure later.
Chapter Nine
IBest Practices & Common Mistakes
Concrete guidance for configuring, monitoring, and responding to OOM kill behavior effectively.
Always set an explicit memory limit
Configure a deliberate memory ceiling for every container in shared or production environments, rather than relying on the host-wide OOM killer as an implicit, unpredictable backstop.
Check the OOMKilled flag before assuming a crash is an application bug
When a container stops unexpectedly, check Docker’s own OOM-specific status flag first, since it immediately rules in or rules out memory exhaustion as the cause before deeper application debugging begins.
Alert on approaching the limit, not just on the kill itself
Build monitoring that surfaces memory usage trending toward the configured ceiling, giving a chance to intervene before the kernel’s forceful, warning-free intervention occurs.
Setting limits based on guesswork
Choosing a memory limit without measuring real application behavior under realistic load risks either frequent unnecessary kills or a limit so generous it offers little real protection.
Ignoring repeated OOM kills as “just restart handled it”
A restart policy masking recurring OOM kills can hide a genuine, worsening memory leak behind an apparently healthy-looking uptime metric.
Assuming application-level exception handling can catch an OOM kill
Because the kill is delivered as an unblockable SIGKILL rather than a catchable exception or signal, no amount of application-level error handling can intercept or soften it.
Chapter Ten
JReal-World & Industry Examples
OOM killer behavior is one of the most universally encountered aspects of running containers at any meaningful scale, across every industry that operates production Docker workloads.
Kubernetes Pod Memory Limits
Kubernetes builds its own pod-level memory limit and eviction behavior directly on top of the same underlying cgroup mechanism this article describes, surfacing OOM kills through its own status reporting for pods and containers.
Language Runtime Memory Tuning
Many managed-memory language runtimes expose their own heap size configuration specifically so operators can keep the runtime’s own memory ceiling safely below a container’s hard limit, avoiding an abrupt kernel kill in favor of the runtime’s own, more controlled memory management.
“Mystery Crash” Postmortems
A recurring pattern across many organizations’ incident postmortems is an initially confusing, unexplained container crash that investigation eventually traces back to an OOM kill triggered by a limit that was too conservative for genuine peak load.
Managed Container Service Defaults
Major managed container hosting platforms apply conservative default memory limits to workloads unless explicitly overridden, directly exposing customers to this same OOM behavior as a routine part of the platform’s resource governance.
Chapter Eleven
KFrequently Asked Questions
Chapter Twelve
LSummary and Key Takeaways
The OOM killer is a blunt, warning-free kernel mechanism — and understanding it well turns confusing, mysterious container deaths into fully explainable, predictable events.
Key Takeaways
- The OOM killer is a Linux kernel mechanism that forcibly terminates a selected process the instant available memory is exhausted, in order to keep the rest of the system running.
- It operates at two distinct scopes in Docker: container-level, triggered by a cgroup memory limit, and host-level, triggered by total system memory exhaustion — and these are easy to confuse but behave very differently.
- Victim selection is based on a calculated “badness” score, heavily weighted by memory usage, and the chosen process receives an unblockable SIGKILL with no opportunity for graceful shutdown.
- An OOM-killed container typically exits with code 137, but the definitive confirmation is Docker’s dedicated OOM status flag, since exit code alone is not conclusive proof.
- Setting an explicit, well-benchmarked memory limit on every container is the primary lever operators actually control, since the kill behavior itself cannot be made gentler without undermining its entire purpose.
- Effective monitoring alerts on memory usage approaching a container’s limit, not just on confirmed kills, and treats repeated OOM kills of the same container as a strong signal of an underlying leak rather than a routine, self-resolving event.