A running program is not always a working program.

When you start an application in a container, the simplest thing Kubernetes can observe is whether the main process is running. If the process exits, Kubernetes knows something is wrong and can restart it. But many failures are quieter. A web server may be running yet stuck in a deadlock, unable to answer any request. A database client may have lost its connection and be waiting forever. A new application may be running but still loading data into memory and not yet able to serve anyone. From the outside, the process looks fine, while users see errors or endless waiting.

Probes are Kubernetes’ answer to this gap. A probe is a small, repeated check that the kubelet performs against a container to learn something the process status alone cannot tell. By asking the container itself, through a web request, a network connection, or a small command, Kubernetes can tell whether the application is truly healthy and can act automatically: restart it, stop sending it traffic, or wait patiently for it to finish starting.

What can go wrong without probes

Without probes, Kubernetes assumes that a running container is healthy and ready the moment it starts. Traffic is sent to pods that are still loading. Stuck pods stay in service forever, quietly failing requests. Rolling updates proceed to replace healthy old pods with new pods that cannot yet answer, causing an outage during what should have been a safe release.

Without probes, “running” means “healthy” and “started” means “ready”.

By default, Kubernetes trusts the container’s main process. If the process is alive, the container is considered running. If the process exits, the kubelet restarts it according to the pod’s restart policy. That is the entire health model when no probes are defined. It handles crashes well, since a crashed process is gone, but it is blind to every failure in which the process stays alive.

The same default applies to readiness. A container with no readiness probe is treated as ready as soon as it starts. The pod is added to the Service’s list of endpoints immediately, and traffic begins to flow, whether or not the application inside has finished initialising. For a small program that starts in milliseconds, this is harmless. For a large application that needs half a minute to warm up, it means the first users of every new pod receive errors.

flowchart TD
    A[Container process starts] --> B[Kubernetes marks it running and ready]
    B --> C[Service starts sending traffic]
    C --> D{Is the application really able to serve?}
    D -->|Yes| E[Users are served]
    D -->|No, still loading or stuck| F[Users see errors]
    F --> G[Nothing detects the problem]
        

Each probe asks a different question and triggers a different action.

The three probes are easy to confuse because they use the same mechanisms and the same settings. What differs is the question each asks and, crucially, what Kubernetes does when the answer is no. Keeping this distinction clear removes most of the confusion around probes.

The last line of defence against a container that cannot recover on its own.

A liveness probe tells Kubernetes whether a container is still functioning at a basic level. If the probe fails several times in a row, the kubelet concludes that the container is stuck in a state it will never leave by itself, kills it, and starts a new one according to the pod’s restart policy. The whole point is to turn “stuck forever” into “restarted and hopefully healthy.”

Liveness probes are meant for one particular kind of failure: the process is running but cannot make progress. Classic examples are a deadlock between threads, an infinite loop, or an internal component that has crashed while the main process survives. A restart fixes these problems because it resets the program to a clean state. A restart does not fix problems that live outside the container, such as a database outage, and this distinction is the key to using liveness safely.

sequenceDiagram
    participant K as Kubelet
    participant C as Container
    loop Every period
        K->>C: Liveness check
        alt Healthy
            C-->>K: Success
        else Stuck
            C-->>K: Failure or timeout
            K->>K: Count consecutive failures
        end
    end
    K->>C: Failure threshold reached, stop container
    K->>C: Start a new container instance
    Note over K,C: Restart delay grows if the container keeps failing
        

The gate that decides whether a pod may receive traffic.

A readiness probe tells Kubernetes whether a container is ready to serve requests at this moment. While the probe succeeds, the pod is listed among the endpoints of every Service that selects it, and traffic flows to it. When the probe fails, the pod is removed from those lists. The container keeps running and is not restarted; it is simply left alone until it reports ready again.

This makes readiness the right tool for anything temporary. A new pod that is still warming its caches is not ready yet. A pod that is busy with an overload may report not ready for a short while to shed load. A pod that has lost a needed connection may become ready again once it reconnects. In all of these cases, restarting would be unnecessary and even harmful, but withholding traffic is exactly right.

stateDiagram-v2
    [*] --> NotReady: container starts
    NotReady --> Ready: readiness probe succeeds
    Ready --> NotReady: readiness probe fails
    NotReady --> Ready: probe succeeds again
    Ready --> [*]: pod terminating, removed from endpoints
        

A patient guardian for applications that take a long time to start.

The startup probe was added to solve a specific problem. Some applications need a long and unpredictable time to start, perhaps because they load large data sets, run migrations, or initialise heavy frameworks. A liveness probe that is strict enough to catch deadlocks quickly will kill such an application while it is still starting, because it cannot answer yet. Loosening the liveness probe to allow for slow starts makes it slow to detect real deadlocks later. It is a painful trade-off.

The startup probe resolves the trade-off by separating the two phases. While the startup probe is running, the liveness and readiness probes are switched off. Once the startup probe succeeds, it stops for good, and the liveness and readiness probes take over with their normal, stricter settings. If the startup probe never succeeds within its allowed budget, the container is killed and restarted.

sequenceDiagram
    participant K as Kubelet
    participant C as Container
    K->>C: Startup check
    C-->>K: Not yet
    K->>C: Startup check
    C-->>K: Not yet
    Note over K,C: Liveness and readiness are paused
    K->>C: Startup check
    C-->>K: Success
    Note over K,C: Startup probe stops, liveness and readiness begin
    loop From now on
        K->>C: Liveness and readiness checks
    end
        

The start-up budget

The maximum time an application is given to start is the failure threshold multiplied by the period. With a period of ten seconds and a failure threshold of thirty, the application has up to five minutes to become healthy. If it starts in twenty seconds, the startup probe succeeds at the next check and the liveness probe begins at once, so a generous budget costs nothing when start-up is quick.

A container’s life is a sequence of phases, and each probe owns one of them.

When all three probes are defined, they form a sequence. The startup probe guards the first phase, during which nothing else runs. When it succeeds, the liveness and readiness probes begin in parallel and keep running for the remainder of the container’s life. Liveness watches for a permanently stuck container, while readiness watches for a temporarily unfit one.

flowchart TD
    S[Container starts] --> SP{Startup probe defined?}
    SP -->|Yes| W[Wait for startup probe to succeed]
    SP -->|No| LR
    W -->|Fails beyond budget| R[Container restarted]
    W -->|Succeeds| LR[Liveness and readiness start]
    LR --> L{Liveness failing repeatedly?}
    L -->|Yes| R
    L -->|No| RD{Readiness passing?}
    RD -->|Yes| T[Pod receives traffic]
    RD -->|No| N[Pod removed from endpoints, keeps running]
    N --> RD
        

Four mechanisms, each with its own strengths and blind spots.

Every probe, whichever of the three kinds it is, must perform its check in one of four ways. The kubelet runs the check from the node where the pod lives. The choice of mechanism decides what the probe can actually detect, how much it costs, and what can go wrong with it.

flowchart LR
    K[Kubelet on the node] -->|web request to pod address| H[HTTP probe]
    K -->|open connection| T[TCP probe]
    K -->|run command inside container| E[Command probe]
    K -->|gRPC health call| G[gRPC probe]
    H --> APP[Application in the container]
    T --> APP
    E --> APP
    G --> APP
        

Five numbers decide how quickly a probe reacts and how forgiving it is.

All three probes share a small set of timing settings. Learning what each one controls, and how they combine, lets you predict exactly how long Kubernetes takes to notice a problem. Most probe incidents come from settings that were copied without understanding.

A worked example

Consider a liveness probe with a period of ten seconds, a timeout of two seconds, and a failure threshold of three. If the application becomes stuck, the first failing check may come up to ten seconds later. Three consecutive failures are needed, each taking up to two seconds to time out, so the container is restarted roughly thirty to forty seconds after the application gets stuck. If you instead used a period of five seconds and a threshold of two, the reaction would take about ten to fifteen seconds, but the probe would be far more sensitive to short glitches.

sequenceDiagram
    participant A as Application
    participant K as Kubelet
    Note over A: Application becomes stuck
    K->>A: Check 1, times out
    Note over K: Failure count 1
    K->>A: Check 2, times out
    Note over K: Failure count 2
    K->>A: Check 3, times out
    Note over K: Failure count 3, threshold reached
    K->>A: Restart container
        

A probe is only as good as the check behind it.

The probe settings tell Kubernetes when and how to ask. The health endpoint decides what the answer means. A poorly designed endpoint makes even perfect settings useless or dangerous. The most important principle is that liveness and readiness answer different questions, so they should usually be different endpoints with different logic.

The most dangerous mistake in probe design is checking things outside the container.

It is tempting to make a health check verify that the database is reachable, the cache is up, and the other services respond. After all, if the database is down, the application is not really healthy. But what should Kubernetes do about it? If the liveness probe fails because the database is down, Kubernetes restarts the application container. The restart does nothing to fix the database. The restarted application fails the same check and is restarted again. Now every pod of the application is in a restart loop, the application is completely unavailable, and the database, once it recovers, is met by a thundering herd of freshly started pods all connecting at once.

sequenceDiagram
    participant D as Shared database
    participant P as Application pods
    participant K as Kubelet
    D->>D: Becomes unavailable
    P->>K: Liveness checks fail, dependency down
    K->>P: Restart all pods
    P->>D: Reconnect, still down
    K->>P: Restart again
    Note over P: Application fully down, restart loops
    D->>D: Recovers
    P->>D: All pods reconnect at once
    Note over D: Surge of connections may overload it again
        

Readiness and dependencies

Readiness is a better place to consider dependencies, because its failure action, withholding traffic, is gentle and reversible. If a pod cannot reach a database it truly needs, it is reasonable to stop sending it requests until the connection returns. But even here, care is needed. If every pod depends on the same shared service, and that service has a brief outage, every pod becomes not ready at once. The Service then has no endpoints at all, and a partial degradation becomes a total outage, even for requests that did not need the failing service.

Giving heavy applications the time they need, without slowing everything else.

Many real applications take a long time to start. Large Java applications, applications that load machine learning models, and legacy systems that run migrations at boot can all take minutes. If probes are not designed with this in mind, Kubernetes will kill the application before it finishes starting, and the application will be stuck in a loop of starting and being killed.

Readiness quietly steers several other Kubernetes features.

Readiness does much more than open and close the door to traffic. Several higher-level features use it as their definition of a pod being usable. Understanding these links explains why a poor readiness probe can cause problems in places that seem unrelated to health checks.

sequenceDiagram
    participant D as Deployment
    participant N as New pod
    participant O as Old pod
    participant S as Service
    D->>N: Create new pod
    N->>N: Start, readiness not yet passing
    Note over D,N: Rollout waits
    N->>S: Readiness passes, joins endpoints
    D->>O: Now safe to remove an old pod
    O->>S: Leaves endpoints
    Note over D: Repeat until all pods are replaced
        

Stopping a pod politely takes cooperation between probes, the network, and the application.

When a pod is told to stop, it is marked as terminating and removed from Service endpoints, and the container receives a termination signal. But removal from endpoints and the shutdown signal happen at about the same time, and it takes a moment for load balancers and proxies across the cluster to learn that the pod is gone. During that moment, new requests may still arrive at a pod that has begun shutting down.

sequenceDiagram
    participant K as Kubelet and controllers
    participant E as Endpoint lists
    participant L as Load balancers and proxies
    participant P as Terminating pod
    K->>P: Mark terminating
    par Endpoint removal
        K->>E: Remove pod
        E->>L: Propagate change
    and Shutdown begins
        P->>P: Optional pre-stop delay
        P->>P: Receive termination signal
    end
    L-->>P: Late requests may still arrive
    P->>P: Finish in-flight requests
    P->>K: Exit within grace period
        

Probe problems tend to appear in a few recognisable patterns.

flowchart TD
    A[Traffic spike] --> B[Application slows down]
    B --> C[Health checks exceed timeout]
    C --> D[Liveness fails repeatedly]
    D --> E[Kubelet restarts pods]
    E --> F[Fewer healthy pods]
    F --> G[Remaining pods get even more load]
    G --> B
        

The same settings behave differently on a quiet laptop and a busy production node.

Probes tuned on an idle test cluster often fail in production. Real nodes are shared with many other pods, disks are slower, networks are busier, and applications pause for memory cleanup. Tuning probes means choosing values that hold up under these conditions while still reacting quickly enough to real trouble.

Probes are cheap individually, but they add up across a large cluster.

Every probe is a small piece of work performed on a schedule. With a few pods this is invisible. In a cluster with thousands of pods, each with two or three probes running every few seconds, the total volume of checks becomes significant, both for the kubelets that run them and for the applications that answer them.

Health endpoints are small doors into your application, and they should be treated with care.

Health checks exist for automation, but the endpoints they call are usually reachable by other workloads inside the cluster network. A careless health endpoint can leak information or provide an easy way to overload the application. Security thinking for probes is mostly about limiting what they reveal and what they can do.

The cluster records why a probe failed, if you know where to look.

When a pod restarts unexpectedly or never becomes ready, probes are a prime suspect. Fortunately, Kubernetes leaves clear evidence. Each failed probe produces an event on the pod that names the probe type and the reason, such as an unexpected status code, a connection refused, or a timeout. Reading these events is the fastest way to understand what is happening.

Health checks should be watched, not just trusted.

Probes work quietly in the background, which is exactly why problems with them can go unnoticed for a long time. A pod that restarts once a day because of a slightly too strict liveness probe may never trigger an alarm, yet it steadily degrades user experience. Good monitoring turns probe behaviour into visible numbers and alerts.

flowchart TD
    M[Monitoring system] --> A1[Alert: restarts rising for a workload]
    M --> A2[Alert: ready pods below desired for several minutes]
    M --> A3[Alert: probe failures rising]
    M --> A4[Alert: Service with zero endpoints]
    M --> A5[Alert: rollout not progressing]
    A1 --> R[Investigate probes and application]
    A2 --> R
    A3 --> R
    A4 --> R
    A5 --> R
        

Reliable arrangements used across many production services.

flowchart TD
    A[Application] --> H[Health port, separate thread pool]
    H --> L[Live endpoint
process alive and progressing]
    H --> R[Ready endpoint
reads cached dependency status]
    H --> S[Start endpoint
initialisation finished]
    B[Background dependency checker] --> C[Cached status]
    C --> R
        

Habits that turn health checks into a source of outages.

A short checklist for dependable health checks.

How teams use, and misuse, probes in daily operations.

Liveness asks whether a container is stuck and should be restarted. Readiness asks whether a pod can serve traffic right now, and if not, removes it from Service endpoints without restarting it.

When an application takes a long or unpredictable time to start. The startup probe holds back liveness and readiness until start-up completes, so a slow start is not mistaken for a failure.

Kubernetes treats a running container as healthy and ready. Traffic reaches pods before they can serve, stuck pods are never restarted, and rolling updates cannot detect broken releases.

No. A restart cannot fix a database problem, so a liveness check on a dependency causes restart loops and turns a partial outage into a full one. Keep liveness internal to the process.

Initial delay is the wait before the first check, period is how often checks run, timeout is how long to wait for an answer, and thresholds are how many consecutive successes or failures change the state. Failure threshold times period sets the reaction time.

The failure threshold multiplied by the period. For example, thirty failures at ten-second intervals allow up to five minutes. If the application succeeds sooner, the startup probe ends immediately.

No. They run for the whole life of the container, so a pod can become not ready and ready again many times, which is useful for handling temporary trouble.

Use an HTTP check for web services, gRPC for services that speak it, a TCP check for simple servers, and a command only when nothing else fits, since commands start a process on each check.

The health check probably becomes slow under load and exceeds its timeout, so liveness fails. Increase the timeout, lighten the endpoint, and consider a dedicated health port and thread pool.

Yes, indirectly. Failing readiness when draining, together with a short pre-stop delay and proper signal handling, helps traffic move away before the container stops.

Usually because new pods never become ready. Check the readiness probe events and the new version’s behaviour, then fix the cause or roll back.

Watch container restart counts, the gap between ready and desired pods, and probe failure counters, and read probe failure events on the pods. Rising numbers usually point to a probe that is too strict or misdesigned.

Probes give Kubernetes the ability to look inside a running application and act on what it finds. The liveness probe detects a container that is stuck beyond recovery and restarts it. The readiness probe decides whether a pod may receive traffic, and it can switch on and off throughout the container’s life without any restart. The startup probe protects slow-starting applications by holding the other two back until initialisation is complete.

The power of probes comes with responsibility. A liveness check that depends on external services can turn a small problem into a total outage. A readiness check that is missing lets broken releases replace healthy pods. Timeouts that are too tight cause restart storms under load. Good probe design means separate, cheap, honest endpoints, timing based on measurement, careful treatment of dependencies, graceful shutdown, and monitoring that shows restarts and readiness clearly. Done well, probes make an application self-healing and safe to release. Done badly, they become the cause of the outages they were meant to prevent.