Two small operations that let you retire or repair a machine in a controlled way.

A node is a single machine, physical or virtual, that runs your containers in a Kubernetes cluster. Sooner or later every node needs attention. It may need a new operating system patch, a kernel upgrade, a hardware replacement, a new version of the Kubernetes node software, or simply to be removed because you no longer need that much capacity. Doing any of these while the node is full of running applications would interrupt those applications. Cordoning and draining exist to avoid that interruption.

Cordoning a node marks it as unschedulable. The scheduler, which is the part of Kubernetes that decides where new Pods go, will stop placing new Pods on that node. Importantly, cordoning does not touch anything already running there. Existing Pods carry on exactly as before. Think of it as putting up a “No Vacancy” sign.

Draining a node goes a step further. It first cordons the node, and then asks every eligible Pod on that node to leave, politely and gradually. Pods that belong to a controller, such as a Deployment, are recreated on other nodes by that controller. When the drain finishes, the node is empty of ordinary workloads and safe to work on. Uncordoning is the reverse of cordoning: it removes the “No Vacancy” sign and lets new Pods arrive again.

These three operations are the daily bread of anyone who maintains a cluster. Even if you never type the commands yourself, managed upgrades, autoscalers, and automatic repair tools all perform cordon and drain on your behalf. Understanding them helps you design applications that survive these events gracefully.

Maintenance is not an exception in a cluster. It is a constant, planned part of life.

Nodes do not live forever. Operating systems need security patches. Cloud providers retire underlying hardware. New versions of the node agent bring fixes and features. Teams resize clusters as demand grows and shrinks. Some nodes simply misbehave, with failing disks or flaky networks, and need to be replaced. Because there are so many nodes, these events happen almost every day in a busy organization.

If you simply power off a node without draining it, Kubernetes will eventually notice that the node has stopped reporting, mark it as not ready, and then reschedule its Pods elsewhere. But that discovery takes time, often several minutes, during which users may see errors. A drain moves the work proactively and in a controlled order, so the applications keep serving traffic throughout.

A short refresher on the building blocks that cordon and drain rely on.

How Pods Are Normally Placed

When you create a workload, the controller creates Pods, and each new Pod waits without a node. The scheduler examines all nodes, filters out those that cannot host the Pod because of resource shortages, rules, or marks like taints, and then ranks the remaining nodes before choosing one. The key point for us is that a node marked unschedulable is filtered out for ordinary new Pods. That single mark is what cordoning sets.

Controllers Bring Pods Back

A Pod managed by a Deployment, StatefulSet, ReplicaSet, or Job is not precious. If it is removed, its controller notices that the count is short and creates a replacement. The replacement is a brand-new Pod and it is placed by the scheduler on any suitable node, which, after a cordon, excludes the cordoned one. This is why draining works so well: it simply removes Pods and lets the controllers do what they always do.

Cordoning is the gentlest operation. It changes where future Pods go, not where current ones are.

When you cordon a node, Kubernetes marks that node as unschedulable. In practice, it sets a flag on the node object and also applies a built-in taint that tells the scheduler to avoid it for ordinary workloads. From that moment, no new Pod will be placed there, unless it is a special kind that tolerates the mark, such as certain system components.

When Cordon Alone Is Enough

Draining is cordoning plus a careful, rule-respecting removal of the Pods.

A drain does two things in sequence. It first cordons the node so that no replacement Pods land there, and then it goes through every Pod on the node and asks each eligible Pod to leave using the eviction mechanism. The word “eviction” here is important, because it is different from simply deleting. An eviction request is checked against the rules you have set for your applications, particularly disruption budgets, and may be refused if granting it would hurt availability. The drain tool keeps retrying until the request is allowed or a time limit is reached.

flowchart TD
    S[Start drain on node] --> C[Cordon the node]
    C --> L[List Pods on the node]
    L --> F{Pod eligible for eviction}
    F -->|No| K[Skip or stop with a message]
    F -->|Yes| E[Send eviction request]
    E --> B{Disruption budget allows it}
    B -->|Yes| T[Pod terminates gracefully]
    B -->|No| R[Wait and retry]
    R --> E
    T --> N[Controller creates replacement on another node]
    N --> D{More Pods left}
    D -->|Yes| F
    D -->|No| Z[Node is empty and ready for work]
        

Which Pods Get Moved and Which Do Not

Not every Pod on a node can or should be evicted. The drain tool applies sensible rules, and stops with an explanation when it meets something it should not touch without your explicit permission.

Once maintenance is complete, the node needs to be told it may receive work again.

Uncordoning removes the unschedulable mark and the built-in taint. The scheduler then considers the node again for new Pods. It is easy to forget this step, and a forgotten cordon is a classic cause of mysterious capacity problems: the cluster looks as if it has the right number of nodes, yet some are silently refusing work.

stateDiagram-v2
    [*] --> Schedulable: Node is healthy and open
    Schedulable --> Cordoned: Cordon applied
    Cordoned --> Draining: Drain begins
    Draining --> Empty: Pods evicted
    Empty --> Maintenance: Patch or upgrade
    Maintenance --> Cordoned: Node returns and is checked
    Cordoned --> Schedulable: Uncordon applied
    Empty --> Removed: Node deleted
    Removed --> [*]
        

Verify Before You Reopen

Before uncordoning, confirm that the node is truly healthy: that its status is ready, that the node agent is running the expected version, that networking and storage plugins are working, and that system Pods on it are healthy. Reopening a node that is only half fixed can cause a new wave of failing Pods. A good habit is to uncordon, watch the first few Pods that arrive, and be ready to cordon again if something looks wrong.

Let us open the hood and follow what happens to a single Pod during a drain.

Kubernetes offers two ways to remove a Pod. Plain deletion removes it immediately, regardless of consequences. The eviction API is a more careful way: it creates an eviction request for the Pod, and the API server checks whether granting it would violate any disruption budget that covers the Pod. If it would not, the Pod is deleted gracefully. If it would, the API server refuses with a “too many requests” style response, which tells the caller to try again later. The drain tool uses eviction for every Pod, which is why drains respect your availability rules.

sequenceDiagram
    participant Admin as Administrator
    participant Drain as Drain tool
    participant API as API Server
    participant Kubelet as Kubelet on node
    participant Ctrl as Controller
    Admin->>Drain: Request drain of node
    Drain->>API: Mark node unschedulable
    Drain->>API: Evict Pod A
    API->>API: Check disruption budget
    API-->>Drain: Allowed
    API->>Kubelet: Pod A marked for deletion
    Kubelet->>Kubelet: Run pre-stop hook and send termination signal
    Kubelet->>API: Pod A is gone
    Ctrl->>API: Create replacement Pod
    Drain->>API: Evict Pod B
    API-->>Drain: Refused for now, budget exhausted
    Drain->>Drain: Wait and retry
        

The Life of a Terminating Pod

A disruption budget tells Kubernetes how much voluntary loss an application can tolerate at one time.

Kubernetes distinguishes two kinds of disruption. Involuntary disruptions are accidents, such as a machine crashing or a network failure. Voluntary disruptions are deliberate actions by an administrator or a tool, such as draining a node for an upgrade. A Pod Disruption Budget, usually shortened to PDB, applies only to voluntary disruptions that go through the eviction API. It says either “at least this many Pods must always be available” or “at most this many Pods may be unavailable at once”.

flowchart LR
    A[Eviction request arrives] --> B{Is Pod covered by a budget}
    B -->|No| C[Allow eviction]
    B -->|Yes| D{Would available count drop below minimum}
    D -->|No| C
    D -->|Yes| E[Refuse and ask caller to retry]
    C --> F[Pod terminates gracefully]
    E --> G[Wait for replacement to become ready]
    G --> A
        

Choosing Good Budget Values

The best value depends on how many replicas you have and how much capacity the application needs. For an application with five replicas that can run on three, a budget allowing two unavailable gives drains room to work without risking the service. For a single-replica application, any budget that requires that replica to stay available makes it impossible to drain its node at all, because the one Pod can never leave without breaking the rule. This leads to one of the most common reasons for stuck drains, covered later.

Cordoning is built on the same mechanism that Kubernetes uses to repel Pods from nodes in many other situations.

A taint is a label on a node with an effect that discourages Pods. A toleration is a permission slip on a Pod that says “I am allowed on nodes with this taint”. There are three effects. The first prevents new Pods from being scheduled onto the node unless they tolerate it. The second prefers to avoid the node but does not forbid it. The third not only prevents new Pods but also removes existing Pods that do not tolerate the taint.

Where Cordon Fits

When a node is cordoned, Kubernetes sets its unschedulable flag and adds a built-in no-schedule taint for unschedulable nodes. Ordinary Pods lack a toleration for it, so they stay away. Certain system Pods, such as those from DaemonSets, carry tolerations for this taint so they can still run on cordoned nodes, which is why per-node agents continue working during maintenance.

Cordon Versus a Custom Taint

The drain tool has a few important switches. Knowing what each does protects you from both surprises and stuck drains.

A Careful Manual Routine

A drain that hangs is one of the most common maintenance frustrations. Almost every case falls into a handful of causes.

Not every Pod is a simple stateless web server. Some need extra thought during a drain.

The best drain is one nobody notices. That depends mostly on how your applications are built.

The Race Between Routing and Shutdown

There is a subtle timing issue. When a Pod is marked for deletion, the system begins removing it from the lists that route traffic to it, but that update takes a moment to reach every proxy and load balancer. If the Pod shuts down instantly, a few requests may still arrive at a Pod that is no longer listening, causing errors. A short delay before shutdown, often a few seconds, gives the network time to catch up so those stragglers are still served.

Whenever a cloud platform upgrades your nodes, it is performing cordon and drain behind the scenes.

Managed Kubernetes services and node-management tools automate the routine you just learned. To upgrade a pool of nodes, they typically bring up new nodes with the new version, then cordon and drain old nodes one at a time or in small batches, wait for Pods to settle, and finally delete the empty old nodes. Because they use the eviction mechanism, your disruption budgets are respected, which is why well-defined budgets matter even if you never drain a node by hand.

flowchart TD
    A[Start upgrade of node pool] --> B[Add new node with new version]
    B --> C[Cordon one old node]
    C --> D[Drain the old node]
    D --> E{Drain succeeded within time limit}
    E -->|Yes| F[Delete old node]
    E -->|No| G[Pause or fail upgrade and alert]
    F --> H{More old nodes}
    H -->|Yes| B
    H -->|No| I[Upgrade complete]
        

Many tools drain nodes on your behalf. Knowing them helps you predict what will happen to your Pods.

Controlling the Pace

Automation can drain many nodes quickly. To avoid harm, limit how many nodes can be disrupted at once, add delays between nodes, and run changes in maintenance windows for critical systems. Combining a sensible pace with disruption budgets keeps availability high even during large, fully automatic upgrades.

Drain is for planned work. Here is what happens when a node goes away without warning, and how Kubernetes tries to help.

If a node crashes or loses its network, the control plane eventually notices that it has stopped reporting and marks it as not ready. After a waiting period, which is five minutes by default, Pods without a suitable toleration are evicted from the failed node by the node lifecycle controller and recreated elsewhere. This process is slower and rougher than a drain, because the node cannot cooperate: the Pods on it are not given a graceful shutdown.

The power to drain a node is the power to disrupt many applications at once. It deserves careful control.

Cordoning requires permission to update node objects. Draining additionally requires permission to list and evict Pods across namespaces. Anyone who holds these rights can take capacity away from the whole cluster or cause an outage for applications they do not own. In shared clusters, these permissions should be restricted to platform administrators and trusted automation.

You should always be able to see which nodes are cordoned, how a drain is progressing, and how applications are coping.

Alerts Worth Having

Useful alerts include nodes that stay cordoned beyond an expected maintenance window, budgets that allow zero disruptions for an extended time, Pods stuck terminating, pending Pods after a node goes away, and sudden drops in ready replicas of important applications. These alerts catch both forgotten cleanup and hidden design problems before they turn a routine upgrade into an incident.

Cordon and drain are powerful and safe when used well, yet they come with real constraints.

Cordon, Drain, and Their Neighbors

Habits that make maintenance boring, and traps that make it painful.

Cordon only stops new Pods from being placed on the node and leaves running Pods alone. Drain cordons the node and then evicts the Pods so they run elsewhere. Cordon is the first half of drain.

Not for applications with multiple replicas, good readiness checks, graceful shutdown, and sensible budgets. Single-replica applications will have a short interruption while their Pod restarts elsewhere.

No. Uncordoning only allows new Pods to be placed there. Existing Pods stay on their new nodes until they are restarted or rebalanced by some other mechanism.

Usually because a disruption budget would be violated. The budget may demand too many available replicas, the application may have too few replicas, or replacement Pods may not be ready yet. Fix the budget, add replicas, or resolve what blocks the replacements.

By default the drain refuses to proceed because of them, since a DaemonSet would just recreate them on the same node. You tell the drain to ignore them, and they remain running, which is normally what you want.

Temporary local storage is deleted when the Pod leaves, so the drain asks for your explicit permission. Data on persistent volumes survives, because the volume is detached and reattached to the new location, subject to zone and storage-system rules.

No. Budgets apply only to voluntary disruptions that go through the eviction mechanism. For crashes, you rely on replicas, spreading across nodes and zones, and fast recovery.

It can be, if the cluster has plenty of spare capacity and applications are well spread, but it is safer to go one at a time, especially for stateful systems. Parallel drains multiply the chance of exhausting capacity or breaking quorum.

Only if you are permanently removing it. For maintenance, keep the node, perform the work, and uncordon it. If your platform replaces nodes rather than patching them, deleting the drained node is the normal final step.

It depends on the number of Pods and their grace periods. Many drains finish in a minute or two. Long drains point to large grace periods, stateful workloads, slow volume movement, or budgets that force waiting. Always set a timeout in automation.

Cordoning and draining are how you take a node out of service without letting anyone down. Cordoning closes the door to new arrivals. Draining then walks the existing residents out, politely and in order, respecting the rules that keep your applications available. Uncordoning reopens the door when the work is done. Behind the scenes, the same machinery powers managed upgrades, autoscaling, node consolidation, and automatic repair, so understanding it makes you better at designing for the realities of a living cluster.

The safest clusters are not those that avoid maintenance but those whose applications expect it. Run several replicas, spread them out, give them sensible disruption budgets, make them shut down gracefully, keep spare capacity, and monitor the process. With those habits in place, draining a node becomes a calm, routine, almost invisible event rather than a nervous midnight gamble.