To understand a liveness probe we need a handful of simple words: container, pod, kubelet, probe and restart.

So what exactly is a liveness probe?

A liveness probe is a rule you attach to a container that tells the kubelet how to check, again and again, whether the container is still alive. If the check keeps failing, the kubelet decides that the container is beyond saving and restarts it.

The purpose is very specific. A liveness probe is not about sending traffic, or about slow starts, or about a temporary hiccup. It is about detecting the situations where the only sensible cure is a restart, and then doing that restart automatically, at any hour, without a human being woken up.

A short history

Google released Kubernetes as open source in 2014. Health checking was part of the thinking from the very beginning, because Google’s own internal systems had long relied on automatically replacing broken programs. The liveness probe was one of the earliest features.

Later, two companions were added. The readiness probe decides whether a container should receive traffic. The startup probe, which became stable in the version 1.20 era, protects slow-starting programs. Together the three make up the family of Kubernetes health checks, and we will see how they differ.

Kubernetes already restarts containers that crash. So why do we need to teach it to notice anything else?

What Kubernetes knows without help

By default, Kubernetes watches one simple fact: is the main process of the container still running? If the program exits, whether by crashing or by finishing, Kubernetes notices at once and, depending on the pod’s restart rule, starts it again. This handles loud failures very well.

But many failures are silent. The process is still there, breathing, yet it cannot do useful work. Here are some real examples that engineers meet again and again.

Without a liveness probe, such a container stays in this half-dead state until a human notices the complaints and restarts it by hand. That could take minutes or hours. Every request sent to it fails in the meantime.

flowchart TB
  subgraph WITHOUT["Without a liveness probe"]
    direction LR
    A1["Application freezes but the process stays alive"] --> A2["Kubernetes sees a running process"]
    A2 --> A3["Customers get errors or timeouts"]
    A3 --> A4["A human finally notices and restarts it"]
  end
  subgraph WITH["With a liveness probe"]
    direction LR
    B1["Application freezes but the process stays alive"] --> B2["Probe stops getting answers"]
    B2 --> B3["Kubelet restarts the container automatically"]
    B3 --> B4["Service recovers in seconds or minutes"]
  end
    

A probe has two decisions to make: how to check, and how patient to be. Learn these building blocks and every probe you meet will make sense.

The four ways to check

The kubelet can perform four kinds of check. You choose one per probe, depending on what your application offers.

The settings that control patience

How often to check, how long to wait, and how many failures to tolerate are all adjustable. These settings have sensible defaults, but understanding them is the key to good probes.

From these settings we can work out an important number: how long a frozen container survives before it is restarted. Roughly, it is the period multiplied by the failure threshold, plus the time the last check waits for its timeout. With the defaults of a ten-second period and three failures, that is about thirty seconds, plus a moment.

Three probes, three jobs

Probes are not run by some distant central system. They are run right next to your container, by a small agent on the same computer. Let us meet the players.

flowchart LR
  SPEC["Pod definition with probe settings"] --> API["Kubernetes API Server"]
  API --> KUB["Kubelet on the node"]
  KUB -->|"HTTP request"| APP["Container: your application"]
  KUB -->|"TCP connection attempt"| APP
  KUB -->|"Run a command"| APP
  KUB -->|"gRPC health call"| APP
  KUB -->|"Too many failures"| RT["Container runtime"]
  RT -->|"Stop and start the container"| APP
  KUB -->|"Events and status"| API
    

Piece by piece

The pod definition is where you write the probe rules. Each container in the pod can have its own liveness, readiness and startup probes. The definition is stored in the cluster through the API server, the front desk of Kubernetes.

The kubelet reads the definition of every pod assigned to its computer and sets up a small timer-driven worker for each probe. This is important: the checks come from the same computer as the container, not from a faraway machine. That keeps them fast and means they keep working even if the computer loses contact with the control centre.

The container runtime is the software that actually starts and stops containers. The kubelet does not kill containers itself. It asks the runtime to stop the old one and start a new one.

Your application must offer something to check. That may be a web address such as a health endpoint, a listening network port, a command that can be run, or a gRPC health service. The quality of your probe depends entirely on how well this piece is designed.

The API server and events receive the kubelet’s reports. When a probe fails, the kubelet records an event such as “Liveness probe failed”, and after the restart it updates the restart counter. Those records are how you, as a human, learn what happened.

Here is what the kubelet does, second by second, after your container starts.

flowchart TD
  A["Timer fires every period"] --> B["Kubelet runs the check"]
  B --> C{"Did it succeed before the timeout?"}
  C -->|"Yes"| D["Reset the failure count to zero"]
  C -->|"No"| E["Add one to the failure count"]
  E --> F{"Has the count reached the threshold?"}
  F -->|"No"| A
  F -->|"Yes"| G["Kill the container and start a new one"]
  D --> A
  G --> H["Restart counter goes up by one"]
  H --> A
    

Doing the arithmetic

Suppose the period is ten seconds, the timeout is one second, and the failure threshold is three. A container freezes just after a successful check. The next check, ten seconds later, fails after waiting one second. That is failure one. The second failing check comes ten seconds after that, and the third ten seconds later. The restart begins about thirty seconds after the container froze, and up to ten seconds either side depending on exact timing.

If your business can accept thirty seconds of trouble, these defaults are fine. If not, you shorten the period. If your application often pauses for ten seconds during heavy work, you lengthen the timeout or raise the threshold, otherwise healthy but busy containers will be killed by mistake.

The restart backoff

If a container keeps failing and restarting, Kubernetes does not restart it instantly forever. It inserts a growing delay between attempts: about ten seconds, then twenty, then forty, and so on, up to a maximum of five minutes. If the container then runs happily for about ten minutes, the delay resets. During the waiting time, the pod shows the status CrashLoopBackOff. This protects the computer from wasting effort on a container that will not recover.

Shutdown timing and the grace period

When the kubelet decides to restart a container, it does not pull the plug instantly. It first sends a polite termination signal and then waits for the grace period, which is thirty seconds unless changed. A well-written program uses this time to finish the request it is handling, close its files and say goodbye to its connections. If the container is still running after the grace period, it is stopped by force.

This matters for liveness probes in a special way. A container that has frozen may never react to the polite signal, so the restart takes the full grace period before the force-stop arrives. If your grace period is a long one hundred and twenty seconds, a frozen container could be lingering for two minutes after the probe decided it was dead. Newer versions of Kubernetes let you set a shorter grace period just for probe-triggered restarts, so frozen containers are replaced swiftly while normal shutdowns stay gentle.

Now let us follow one container through its whole life, from start-up, through healthy service, to a freeze and a restart.

sequenceDiagram
  participant C as Container
  participant K as Kubelet
  participant S as Service Endpoints
  K->>C: Start the container
  K->>C: Startup probe repeated until it passes
  C-->>K: Startup probe passes
  K->>C: Begin liveness and readiness probes
  C-->>K: Readiness passes
  K->>S: Pod is added to the list of receivers
  Note over C: The application freezes
  K->>C: Liveness check gets no answer
  K->>C: Liveness check gets no answer again
  K->>C: Third failure in a row
  K->>C: Stop the container and start a fresh one
  K->>S: Pod removed until ready again
    

The stages of life

What a restart keeps and what it loses

A restart replaces only the container. The pod, its name, its network address, its attached storage volumes and the contents of its shared scratch folders all remain. What is lost is everything inside the old container’s memory and its own writable layer. Connections open to the old container are cut, which is why clients should be ready to retry.

This is one reason liveness probes suit stateless programs so well. A stateless program keeps nothing important in memory, so being restarted costs only a few seconds. A program that holds a lot of unsaved data in memory needs more care, because a wrongly triggered restart could lose that data.

How you see it happening

When a liveness probe fails, the kubelet writes an event to the pod, along the lines of “Liveness probe failed” followed by the reason, such as an HTTP status code or a timeout. When the threshold is reached, another event records that the container failed its probe and will be restarted. The pod list shows the RESTARTS number growing. The pod’s detailed description also shows the last state of the container, including the reason and exit code of the previous run.

Exit code 137 is a common one to recognise. It means the container was force-stopped with a kill signal after it failed to shut down politely. Exit code 143 means it stopped in response to the polite termination signal.

Most real-world liveness probes fall into a small set of patterns. Learn to recognise them and you can design a good probe for almost any program.

A worked scenario, told in words

Let us design a probe for a small service that converts uploaded pictures into thumbnails. It is a web service with a pool of worker threads. First we ask: how can it fail silently? The answer is that all worker threads might get stuck on a broken image, leaving none free. New uploads then wait forever, while the process remains alive.

Next we choose a check. A web endpoint that is served by the same worker pool would also get stuck, which is exactly what we want to detect. But it must be light: it returns a plain “OK” without touching storage or other services. We serve it on the normal port, so that a jammed worker pool makes it time out.

Then we choose numbers. The service starts in about eight seconds, so a startup probe with a thirty-second window is plenty. Normal answers take a few milliseconds, but occasional memory clean-ups pause it for up to two seconds. We set the timeout to four seconds, the period to ten, and the failure threshold to three. A real jam is therefore repaired in roughly forty seconds, and a two-second pause never causes trouble.

Finally, we test. In a test cluster we send the service an image built to jam its workers, watch the health check time out three times, and see the container restart. We also run a load test to confirm that busy but healthy containers are left alone. Only then do we roll the probe out to production.

A health check looks harmless, yet it is a doorway into your application and a lever that can restart it. Both deserve care.

The restart lever as a weakness

An attacker who can make your health check fail can force your containers to restart repeatedly. If the probe is expensive or depends on outside services, a flood of requests to it can cause exactly that. This is a form of denial of service. Simple, cheap, local health checks are naturally resistant.

Similarly, a probe that checks a shared dependency turns one outage into many. If ten thousand containers all restart because the database slowed down, the sudden burst of new connections can crush a database that was only struggling. Good design keeps such decisions out of the liveness path.

Good probes are chosen with the application in mind. Here is how to decide, and how to find sensible numbers.

Which check method fits?

Finding the right numbers

There are no magic numbers, but there is a sound method. Start from what you can tolerate. How long can your service be broken before the damage matters? That gives an upper limit for period times threshold. Then look at how your program behaves under load. How long do its slowest normal responses take? How long is its worst pause when it cleans up memory? The timeout must be comfortably longer than those pauses.

Probes and cloud load balancers are different things

On Amazon EKS, an Application Load Balancer also performs health checks on its targets to decide where to send traffic. These are separate from Kubernetes probes. The load balancer’s check decides whether it sends web requests to a pod. The liveness probe decides whether Kubernetes restarts a container. It is wise to make them agree in spirit, but never assume one replaces the other. Readiness, not liveness, is the probe that ties in with traffic decisions during updates.

No tool is perfect. A good engineer knows both what liveness probes give and what they cost.

The trade-offs behind the lists

Quick detection versus false alarms. A short period and low threshold find failures quickly, but a healthy program that hiccups for a few seconds gets killed. A long period and high threshold are calm, but users suffer longer before recovery. The right balance depends on how costly a restart is versus how costly a freeze is for your service.

Shallow checks versus deep checks. A shallow check, such as “the web server answers”, rarely gives a false alarm but may miss subtle breakage. A deep check, such as “can I reach the database and the cache”, catches more but blames the wrong thing when a neighbour fails. For liveness, the safe rule is to stay shallow, and to leave deep checks to readiness and monitoring.

Restarting versus understanding. Automatic restarts hide problems. A memory leak that causes a restart every six hours may go unnoticed for months. Restarts should be counted, graphed and reviewed, not merely enjoyed.

One probe versus none. A pointless probe adds risk. If a program either works or exits, and has never been observed to hang, skipping the liveness probe is a legitimate choice.

Liveness probes compared with other recovery tools

A liveness probe is one layer in a stack of recovery tools, and it helps to see where it fits. The lowest layer is the automatic restart of a container that exits. The liveness probe extends this to containers that are stuck but not exited. Above that, a deployment notices when a whole pod disappears, for example because its computer failed, and creates a replacement on another computer. Above that, an autoscaler adds or removes pods as load changes. And above everything, monitoring and alerts tell humans about patterns the automation cannot fix.

Each layer handles a different size of problem. A liveness probe solves the small, local one: this one container is stuck. It cannot solve a broken release that fails everywhere, or a database that is down. Trying to make it do so is the source of most probe disasters.

Probes fix problems quietly. That is exactly why you must watch them, so the quiet fixes do not hide the loud causes.

Failure scenarios worth rehearsing

A pattern is a proven way of solving a common problem. An anti-pattern is a habit that looks sensible at first and causes pain later.

Patterns that work

Anti-patterns to avoid

These habits come from many teams running probes in production. Follow them and you will avoid most of the painful surprises.

Common mistakes and their cures

Reviewing probes after every incident

Probes are not “set and forget”. Each incident is a chance to learn whether they helped or hurt. After any outage, ask three questions. Did a probe notice the problem sooner than a human would have? Did any probe make things worse by restarting healthy containers? And did the alerts on restart counts fire early enough to be useful?

Write the answers down and adjust the numbers. Over a few months, a team that does this ends up with probes that are boring, quiet and trusted, which is exactly what a safety net should be.

Organisations of every size use liveness probes. The stories below are simplified illustrations of common situations, not descriptions of any one company’s private setup.