Before hunting for evidence, it’s worth being precise about what event we’re actually looking for.

The one-sentence definition

An OOM kill (out-of-memory kill) happens when the Linux kernel forcibly terminates a process because the memory boundary it’s operating within — for a container, its cgroup memory limit — has been exhausted, with no more memory left to reclaim or swap out. It is not a crash the application chose, requested, or was even necessarily aware was coming; it’s an external, kernel-level decision made entirely outside the application’s own control flow.

Why this is a genuinely different debugging skill from application-level debugging

Most debugging a developer does day to day involves reasoning about code they can read, running in a process they can attach a debugger to, producing logs they control the content of. An OOM kill breaks all three of those assumptions at once: the decision is made entirely outside the application’s own code, the process is gone before any debugger could meaningfully attach, and the application never got a chance to write anything about its own final moments. This is precisely why treating an OOM kill investigation like an ordinary application bug — reading stack traces, adding print statements, stepping through code — tends to go nowhere, and why this tutorial instead focuses on the separate, kernel- and platform-level evidence trail covered in the chapters ahead.

Three pieces work together to make an OOM kill possible in the first place. Understanding each one separately makes the combined event much easier to reason about.

The limit: a promise the container can never break

When a container is started with a memory limit, Docker writes that value into the container’s cgroup — a kernel-managed accounting and enforcement boundary. Unlike a soft warning or a monitoring alert, this limit is a hard ceiling the kernel itself actively enforces: the cgroup’s total memory usage is simply never allowed to exceed it, no exceptions, no grace period.

The exhaustion: nowhere left to reclaim from

As covered in earlier tutorials on memory and storage internals, the kernel has several tools to free up memory under pressure before resorting to killing anything — reclaiming file-backed page cache, swapping out anonymous memory if swap is available. An OOM kill specifically happens once all of those options have been exhausted within the relevant scope, and a process is still requesting more memory than remains available.

The killer: a last-resort decision-maker

The OOM killer is the kernel component that steps in at exactly that point. Its job isn’t to prevent the system (or cgroup) from running out of memory — by the time it activates, that’s already happened — its job is to restore a workable state as quickly as possible by choosing one process to sacrifice, freeing its memory immediately.

Diagnosing an OOM kill is fundamentally an exercise in knowing which of several separate systems recorded what — no single place tells the whole story on its own.

graph TD
    A["Container memory exhausted"] --> B["Kernel OOM killer activates"]
    B --> C["Kernel log records full event"]
    B --> D["Process terminated with SIGKILL"]
    D --> E["Container exits with code 137"]
    E --> F["Docker records OOMKilled: true"]
    E --> G["Docker events stream logs death"]
    C -.->|"most detailed source"| H["Full diagnostic picture"]
    F -.->|"fast confirmation"| H
    G -.->|"timing context"| H
        

Why no single source is sufficient on its own

It’s worth being explicit about why this tutorial insists on checking multiple sources together rather than picking one “best” place to look. The kernel log is the most detailed, but it’s host-wide and requires correctly matching the right cgroup to the right container. The OOMKilled flag is the fastest and most unambiguous, but it tells you nothing about why the memory grew in the first place. The events stream gives timing context but no memory detail at all. Historical cgroup statistics show the trend but require monitoring to have already been capturing them before the event happened. A complete diagnosis typically draws on at least two or three of these sources together, each compensating for what the others lack.

Understanding the selection logic behind an OOM kill turns the kernel log’s cryptic-looking numbers into something genuinely readable.

The badness score

When the OOM killer activates within a given scope, it evaluates every candidate process in that scope and assigns each one a numeric “badness” score, primarily driven by how much memory that process is currently using — a process consuming more memory generally scores as a “worse” candidate to keep alive, and thus a more likely candidate to be killed. The process with the highest resulting score is the one selected for termination.

oom_score_adj: a deliberate thumb on the scale

Every process also carries an adjustable value, oom_score_adj, that shifts its computed badness score up or down before the final comparison — a way for the system (or occasionally an administrator) to declare certain processes more or less expendable than raw memory usage alone would suggest. In container platforms, this value is sometimes set deliberately based on a container’s configured priority, so that, when multiple containers share a host and one has to be sacrificed, a lower-priority workload is more likely to be chosen over a critical one.

Scoped to the cgroup, not always the whole host

Critically, when a container has its own memory limit configured, an OOM event triggered by that container hitting its own ceiling is scoped to just that container’s cgroup — the kernel only considers processes within that boundary as candidates, even if the host machine as a whole has abundant free memory elsewhere. This cgroup-aware OOM behavior is what makes per-container memory limits meaningful in the first place: it contains the failure to the one workload that actually exceeded its own budget.

sequenceDiagram
    participant K as Kernel OOM killer
    participant C as Container's cgroup
    participant P1 as Process A (high memory)
    participant P2 as Process B (low memory)
    K->>C: Memory limit exceeded, no reclaim left
    K->>P1: Compute badness score
    K->>P2: Compute badness score
    Note over K: P1's score is higher (more memory used)
    K->>P1: Send SIGKILL
    P1-->>K: Terminated immediately
    Note over C: Memory freed, cgroup returns under its limit
        

Why memory usage, not memory allocation history, drives the score

It’s worth being precise that the badness score is driven primarily by a process’s current, resident memory usage at the moment of the OOM event, not by how much memory it has allocated and freed over its lifetime, nor by how quickly it grew. Two processes with identical current memory footprints score similarly regardless of whether one got there gradually over hours and the other spiked there in seconds — the kernel’s decision is a snapshot judgment, not a historical one. This is exactly why the historical trend, covered in a later chapter on cgroup statistics, has to come from separate monitoring rather than from the kernel’s own OOM report, which only ever captures the single moment of the kill itself.

Tracing the full sequence from rising memory usage to a container’s final exit state shows exactly where each piece of diagnostic evidence gets written.

Before digging into detailed kernel logs, there’s a fast, reliable first check worth running on any suspicious container exit.

Why 137, specifically

A container’s exit code following a signal-based termination is conventionally reported as 128 plus the signal number. The OOM killer terminates a process with signal 9 (SIGKILL, the uncatchable, unblockable kill signal) — and 128 plus 9 is 137. Seeing exit code 137 is a strong, fast hint that a process was killed by SIGKILL, though it’s worth being precise: 137 alone confirms a SIGKILL, not specifically that the OOM killer was the one that sent it.

The definitive confirmation: the OOMKilled flag

Docker itself tracks, separately from the raw exit code, an explicit boolean field on the container’s state specifically recording whether the kernel’s OOM killer was the cause of that container’s termination. This field is the single most direct, unambiguous confirmation available — where exit code 137 is a strong hint, this flag is Docker’s own explicit statement that it detected an OOM kill.

Why relying on exit code alone leads to false positives

It’s worth walking through a concrete false-positive scenario to make this concrete. An orchestrator performing a rolling update might forcefully terminate an old container instance that’s taking too long to shut down gracefully, sending exactly the same SIGKILL an OOM kill would use — producing the identical exit code 137, with no memory problem involved at all. Someone reading only the exit code in this scenario would reasonably, but incorrectly, suspect a memory issue, potentially spending real investigation time looking for a leak that was never there. Checking the OOMKilled flag first would have ruled this out in seconds.

This is the richest evidence source available, and it’s worth understanding its structure conceptually, section by section, even without reproducing exact log syntax here.

Mapping a kernel-level process back to a specific container

Because the kernel log reports process and cgroup identifiers rather than Docker container names directly, correctly diagnosing which container was affected sometimes requires matching the cgroup path or process ID mentioned in the kernel log against Docker’s own container listing from around that same timestamp — a small but important translation step between the kernel’s vocabulary and Docker’s own naming.

A container that “just disappeared” has several plausible explanations, and confusing them wastes real debugging time chasing the wrong root cause.

Once an OOM kill is confirmed, the next question is always “why did memory usage actually get this high” — and the answer usually falls into a small number of recurring patterns.

Once the cause is understood, the practical response usually involves a memory-limit decision — and every choice here carries a real trade-off, not a universally correct number.

Why “just raise the limit” is a diagnosis, not always a fix

Increasing a memory limit after an OOM kill is sometimes the genuinely correct response — when investigation confirms the workload’s real, legitimate peak usage simply needed more headroom. But raising the limit without first confirming the cause risks masking a real leak behind a larger allowance, buying time before the same problem resurfaces at a higher, more resource-costly ceiling. The diagnostic steps in earlier chapters exist precisely to make this distinction before committing to either response.

The role of a soft reservation alongside a hard limit

Beyond the binary choice of raising or lowering a hard limit, a separate soft reservation value — a target the kernel leans toward respecting under general host pressure without being a hard ceiling the container can never exceed — offers a middle path for workloads with genuinely variable usage. Setting a reservation close to typical usage, with a hard limit further above it for real spikes, gives a workload legitimate room to breathe during normal peaks while still keeping an absolute, enforced boundary against runaway growth.

OOM kills aren’t purely an operational nuisance — a few genuine security angles are worth understanding.

The best diagnosis is the one that happens before a kill, not after — proactive memory monitoring turns an OOM kill from a mystery into a confirmed, predicted outcome.

Setting sensible alert thresholds

A common early mistake in building proactive monitoring is alerting only at the same threshold the kernel itself enforces — effectively duplicating the OOM kill’s own timing rather than providing any real warning ahead of it. Setting a meaningfully earlier threshold, with enough margin to actually investigate and respond before the hard limit is reached, is what turns monitoring into genuine prevention rather than a slightly faster notification of the same failure.

A handful of recurring mistakes account for most of the frustration teams experience around OOM kills — usually because the underlying diagnosis was skipped in favor of a quick-seeming fix.

Good pattern: treating the OOMKilled flag as a first-class alert condition

Wiring monitoring specifically around Docker’s own OOMKilled state — rather than only generic “container restarted” alerts — lets a team immediately distinguish a memory-related failure from every other kind of restart, skipping straight to the right diagnostic chapter of this tutorial instead of starting an investigation from scratch each time.

A practical checklist distilled from everything covered so far.

Pulling every earlier chapter together into one repeatable investigation sequence for the next unexplained container exit.

The diagnostic patterns covered here show up consistently across the broader container ecosystem, not just plain Docker hosts.

Beyond the kernel’s OOM report itself, the cgroup’s own detailed memory statistics — when captured by monitoring before an event — provide the richest picture of exactly how usage built up over time.

Breaking total usage down by category

A cgroup’s memory accounting doesn’t just report one aggregate number; it breaks usage down into distinct categories, each telling a different part of the story. Anonymous memory usage reflects the application’s own heap and stack allocations — the category most directly tied to a genuine leak. File-backed page cache usage reflects data read from files that the kernel is holding onto for efficiency, which is normally reclaimable relatively cheaply and thus less concerning on its own. Swap usage, where applicable, shows how much of the container’s memory pressure has already been offloaded to disk-backed swap space before the limit was finally reached.

Why this breakdown matters more before the kill than after

The kernel’s own OOM report captures a snapshot at the moment of the kill, but a cgroup’s detailed statistics, sampled repeatedly over time by ordinary monitoring, show the trend leading up to that moment — which category was actually driving the climb, and how long it had been climbing. This historical view is often more useful for distinguishing a leak from a legitimate spike than the single final snapshot alone, which is why proactive metric collection, covered earlier, pays off directly here.

When an application is deployed alongside a sidecar — as covered in earlier tutorials on the sidecar pattern — diagnosing an OOM kill gains an extra layer worth understanding specifically.

Which container’s cgroup actually hit the limit?

Depending on how memory limits are configured across a paired application-and-sidecar deployment, each container may have its own independent cgroup and limit, or in some deployment models the two may effectively share a combined budget. The very first question worth answering in a multi-container investigation is which specific container’s cgroup the kernel log actually references — it’s entirely possible for a sidecar’s own memory usage, not the application’s, to be the actual trigger behind an OOM event that superficially looks like “the service went down.”

Sidecar overhead as a quiet contributor to application-side limits

As mentioned earlier in the common-causes chapter, teams sizing a paired deployment’s memory budget sometimes account generously for the application’s own needs while underestimating the sidecar’s baseline footprint, or vice versa — leading to an OOM kill that traces back to a budgeting gap between the two, rather than a genuine problem in either container’s own code.

Why single-cgroup diagnosis skills still apply directly

None of this changes the fundamental diagnostic approach covered throughout this tutorial — every technique here, from reading the kernel log to distinguishing a leak from a spike, applies identically to each individual container’s cgroup. The only addition in a multi-container setup is the extra step of first confirming precisely which of the paired containers the kernel log is actually reporting on, before applying everything else.

Not necessarily — 137 confirms a SIGKILL was sent, which is consistent with an OOM kill, but SIGKILL can also come from a manual forceful stop or another external source. Always confirm with the OOMKilled flag and kernel log before concluding it’s memory-related.

Because SIGKILL is immediate and uncatchable, the application has no real opportunity to log a final message before termination — this is expected, not a sign that logging is broken.

Per-container memory limits are enforced by the container’s own cgroup boundary, independent of the host’s total available memory. A container can be OOM-killed purely for exceeding its own configured limit, even while the host as a whole has abundant free memory elsewhere.

Only after confirming the cause. If the workload’s genuine, healthy peak usage simply needed more headroom, raising the limit is correct; if the real cause is a leak or unbounded cache, raising the limit only delays the same failure at a higher cost.

The kernel log’s victim section identifies the specific process by name and ID within the container’s cgroup — the killed process isn’t necessarily the container’s designated main process, just whichever process scored highest in the kernel’s badness calculation.

Yes — swap only delays an OOM kill by providing additional reclaimable capacity; once both RAM and the available swap allowance within the cgroup are genuinely exhausted, the OOM killer still activates as the final resort.

A single, isolated event tied to a known, unusual spike (a one-off large batch job, for instance) may not warrant deep investigation. A repeating pattern, or an OOM kill with no obvious explanation, is worth the full diagnostic walkthrough covered in this tutorial.

The kernel log’s cgroup reference identifies exactly which container’s boundary triggered the event — always confirm this directly rather than assuming based on which of the two seems more likely at a glance.

Yes, it’s counted as part of the cgroup’s total usage, though it’s normally reclaimed relatively cheaply before an OOM kill becomes necessary — a cgroup dominated by reclaimable cache rather than anonymous memory is a different, generally less concerning situation than one dominated by anonymous memory alone.

Diagnosing an OOM kill is less about any single command and more about knowing where the evidence for an abrupt, kernel-level decision actually gets recorded — because the process itself almost never gets a chance to explain what happened on its own. Confirming the event through Docker’s own OOMKilled flag, reading the kernel’s detailed report of exactly who was killed and why, and tracing the memory trend that led up to it together turn what initially looks like a mysterious, unexplained failure into a concrete, investigable root cause. From there, the right fix — a code change, a limit adjustment, or both — follows naturally from the evidence, rather than from a guess. The underlying skill transfers well beyond any single incident: once the evidence-gathering habit is built, every future unexplained container exit becomes a five-minute check rather than an open-ended mystery.