A short definition first, so every later chapter has something solid to stand on.

Kubernetes runs your applications inside Pods, and Pods run on nodes, which are the computers of the cluster. A component called the scheduler decides which node each new Pod should go to. The Descheduler is a separate, optional program that looks at Pods that are already running and finds the ones whose current placement is no longer a good fit. It then removes those Pods using the safe eviction process, and the normal controller of each Pod, such as a Deployment, creates a fresh replacement. The scheduler places that replacement, and with luck it lands somewhere better.

The key idea is in the name. To schedule is to choose a place. To deschedule is to take a Pod out of its place so that it can be scheduled again. The Descheduler never chooses a node itself. It only decides who should leave.

To understand what goes stale, we first need to see how a placement decision is made.

When you create a Pod, it first has no node. The scheduler notices it and runs two big steps. The first step is filtering. The scheduler throws out every node that cannot run the Pod, for example because it lacks enough free CPU or memory, because it has a taint that the Pod does not tolerate, or because a rule in the Pod says it must run on a certain kind of machine. The second step is scoring. Among the nodes that remain, the scheduler gives each one a score based on things like how much room is left and whether the Pod would be nicely spread out, and it chooses the highest score.

Once the choice is made, the scheduler writes the node name into the Pod, and the job is finished. The Pod runs there until it ends, crashes or is removed. The scheduler does not wake up later to ask whether the choice still looks right.

This design is deliberate and good. It keeps the scheduler fast and predictable, and it avoids a world where running applications are constantly shuffled around. But it also means that placement can grow old, and that is the gap the Descheduler fills.

A choice that was perfect at 9 in the morning can be a poor choice by lunchtime.

The state of a cluster keeps changing. A placement that satisfied every rule when the Pod was created may no longer match the cluster today. The difference between the ideal placement and the real placement is often called drift. There are several common reasons for it.

Three actors share this job: the Descheduler decides who leaves, a controller makes a replacement, and the scheduler picks the new home.

The easiest way to hold the picture in your head is as a loop. The Descheduler wakes up, reads the state of the cluster from the API server, decides that a particular Pod is poorly placed, and asks the API server to evict it. The Pod’s owner, such as a Deployment, sees that a Pod has disappeared and creates a new one. The scheduler then sees an unplaced Pod and picks a node for it using the rules of the day.

flowchart LR
  D["Descheduler"] -->|"1 Read nodes and Pods"| A["API Server"]
  D -->|"2 Evict a badly placed Pod"| A
  A -->|"3 Pod removed"| C["Owner controller, such as a Deployment"]
  C -->|"4 Create replacement Pod"| A
  A -->|"5 Unplaced Pod appears"| S["Scheduler"]
  S -->|"6 Bind Pod to a better node"| A
        

This division of labor is important for two reasons. First, the Descheduler needs no special power to place Pods, so it cannot place them anywhere that the scheduler would not. Second, the replacement Pod passes through all the normal checks again, including quotas and admission rules, so nothing is bypassed.

The Descheduler is itself an ordinary program running in your cluster, and you choose how often it runs.

The Descheduler is packaged as a container image and is commonly installed with a Helm chart or with the manifests that its project provides. It normally lives in the kube-system namespace and is given a high priority class so that it is not itself removed when the cluster is under pressure. There are three ways to run it.

The service account and its permissions

Because the Descheduler talks to the API server, it needs an identity and permissions. It runs under a dedicated service account that is allowed to read nodes, Pods, namespaces and priority classes, and to create evictions. It does not need permission to create Pods or to change nodes. Keeping its rights this narrow means that even a bug in the program cannot do more harm than evicting Pods.

All the Descheduler’s behavior is controlled by one policy that you write, and nothing happens that the policy does not ask for.

The Descheduler does nothing on its own initiative beyond reading its policy. The policy is a small configuration object that is stored in a ConfigMap and mounted into the Descheduler Pod. In the current style, the policy is organized into profiles. A profile is a named group of settings that says which plugins to turn on and how each one should behave.

Types of plugin

The behavior of the Descheduler is split into plugins. Plugins fall into a few types, which the project calls extension points.

What a profile contains

A profile lists each plugin it wants, along with settings for each. It also configures the default evictor, which is the safety filter described in the next chapters. Several profiles can exist side by side, for example one for your general workloads and one for a special group of nodes. Within a pass, all Deschedule plugins run before Balance plugins, and the filters are consulted before any Pod is actually evicted.

Each run follows the same sequence of steps, from reading the cluster to counting what was evicted.

Understanding a single run helps explain why the Descheduler sometimes evicts nothing even when a cluster looks untidy. At each step, rules narrow down the candidates.

sequenceDiagram
  participant D as Descheduler
  participant A as API Server
  participant C as Deployment Controller
  participant S as Scheduler
  D->>A: List ready nodes and Pods
  A-->>D: Current cluster state
  D->>D: Plugins nominate Pods, filters remove unsafe ones
  D->>A: Create eviction for chosen Pod
  A->>A: Check disruption budget
  A-->>D: Accepted
  A-->>C: Pod deleted event
  C->>A: Create replacement Pod
  A-->>S: New unplaced Pod
  S->>A: Bind to a node
        

At the end of each run the Descheduler reports what it did through its logs and metrics. In Deployment mode, it then sleeps until the next interval. In Job and CronJob modes, it simply exits.

A tool that removes running Pods must be very careful, and the Descheduler has several layers of protection.

The Descheduler uses the Eviction API rather than simply deleting Pods. An eviction is a polite request. The API server checks any PodDisruptionBudget that covers the Pod. A PodDisruptionBudget is a rule that says “at least this many copies of my application must stay available”. If removing the Pod would break that promise, the eviction is refused, and the Pod stays. Accepted evictions also respect the Pod’s grace period, so the application has time to finish its requests and shut down cleanly.

What is protected by default

Before the Eviction API is ever called, the Descheduler’s default evictor already filters out Pods that are risky to move. By default these are not evicted.

Tuning the evictor

The default evictor has settings you can change. You can allow eviction of Pods with local storage or of failed bare Pods, you can require that another node could actually host the Pod before it is evicted, which is called the node-fit check, you can limit eviction to Pods with a certain label, and you can protect young Pods by setting a minimum age. You can also mark a single Pod with an annotation to say that it should or should not be evicted, which gives application owners a simple way to speak up.

This plugin spreads load by moving Pods from crowded nodes toward empty ones.

This is the plugin that most people think of first. It looks at how much of each node’s capacity is already promised to Pods, and it tries to even things out. It uses two sets of limits that you provide, one called thresholds and one called target thresholds, each given as percentages for resources such as CPU, memory and the number of Pods.

How a node is classified

A node whose use is below all of the thresholds is called underutilized. It has plenty of room. A node whose use is above any of the target thresholds is called overutilized. It is crowded. Nodes in between are considered fine and are left alone.

flowchart TD
  N["Each node"] --> Q1{"Below all thresholds?"}
  Q1 -->|Yes| U["Underutilized: can receive Pods"]
  Q1 -->|No| Q2{"Above any target threshold?"}
  Q2 -->|Yes| O["Overutilized: Pods may be evicted"]
  Q2 -->|No| OK["Balanced: left alone"]
        

How it chooses what to evict

The plugin only acts if there is at least one underutilized node and at least one overutilized node. It then evicts Pods from the overutilized nodes until they fall down toward the target level, or until the underutilized nodes have no more room to receive Pods. Among the Pods on an overutilized node, it prefers to evict less important ones first, ordering by priority and then by quality of service class, so that best-effort Pods go before guaranteed ones.

Useful settings

You can require that a minimum number of underutilized nodes exist before the plugin does anything, which avoids pointless moves in small clusters. You can also choose to measure thresholds as a distance from the average usage of the cluster instead of as fixed numbers, which adapts better when the cluster as a whole is very busy or very quiet.

Sometimes you want the opposite of spreading: you want to pack Pods tightly so that spare nodes can be removed.

Spreading Pods out is good for resilience, but it costs money in a cloud, because you pay for every node even if each one is half empty. If your goal is to save money, you want Pods packed onto as few nodes as possible, so that the empty ones can be shut down by a cluster autoscaler. This is called compaction or bin packing.

The HighNodeUtilization plugin does this. It finds nodes that use less than a given threshold and evicts Pods from them, hoping that those Pods will be placed onto other nodes that are already busier. Over time, lightly used nodes become empty. An autoscaler, seeing an empty node, can then remove it.

Several copies of the same application on one node is a hidden risk that this plugin removes.

Applications usually run as several identical copies, called replicas, so that if one fails the others keep serving. This only helps if the copies are on different nodes. If three replicas sit on the same node and that node fails, all three disappear together, and your application is down despite having replicas.

This can happen after node restarts or upgrades, when many Pods land on whichever node is available first. The RemoveDuplicates plugin looks at Pods that share the same owner, such as the same ReplicaSet or StatefulSet, and checks how they are spread. If one node holds more than its fair share, it evicts the extra ones so that the scheduler can place them on other nodes.

Rules about spreading are checked when a Pod is placed, and these plugins check them again later.

Kubernetes lets you describe how Pods should be spread across zones, nodes or other groups. A topology spread constraint says something like “keep the number of Pods in each zone within one of each other”. The amount of difference allowed is called the maximum skew. The scheduler respects this rule at the moment of placement. Afterwards, if Pods in one zone are deleted, the spread can become uneven again, and nothing corrects it.

RemovePodsViolatingTopologySpreadConstraint

This plugin checks whether the current spread breaks the rule, and if so, evicts enough Pods from the most crowded group to bring the difference back within the allowed skew. You can choose whether it should also handle soft constraints, which are preferences rather than strict rules.

RemovePodsViolatingInterPodAntiAffinity

Inter-pod anti-affinity says that certain Pods must not run next to each other, for example two copies of a database on one machine. The scheduler enforces it at placement time. If a conflict appears later, because a label changed or because of an unlucky sequence of events, this plugin evicts the Pods that break the rule so that they can be placed apart.

When the rules about a node change after a Pod arrived, these plugins make sure the Pod still belongs there.

RemovePodsViolatingNodeAffinity

A Pod can say that it must run only on nodes with a certain label, for example nodes with fast disks. This is called node affinity. The common form is written with a long name that ends in “ignored during execution”, and that ending is the catch. It means the rule is checked only at placement time. If an administrator later removes the label from the node, or the Pod’s rule is updated, the Pod keeps running on a node that no longer matches.

This plugin finds such Pods and evicts them, but only when another node exists where the rule would be satisfied. That second check prevents the plugin from evicting a Pod just to leave it homeless. You can choose whether it handles only the required form of affinity, only the preferred form, or both.

RemovePodsViolatingNodeTaints

A taint is a mark on a node that tells Pods to stay away unless they carry a matching toleration. Taints are often used to reserve nodes for special work or to prepare a node for maintenance. A taint with the “no schedule” effect blocks new Pods but does not remove Pods that are already on the node. This plugin removes the Pods that do not tolerate the taint, so the node is truly cleared for its new purpose.

Three simple plugins deal with Pods that have lived too long, crash too often, or have already failed.

When these help

PodLifeTime is a gentle way to make sure nothing runs forever without being refreshed. It is best used with a long limit, such as days, and combined with a disruption budget so that restarts happen slowly. The restart plugin is useful for problems tied to one node, such as a faulty disk or a network issue, because moving the Pod away may cure the symptom. It does not fix a bug in the application itself, and an application that crashes everywhere will simply crash on its next node too.

Because eviction causes real interruption, the Descheduler lets you cap how much it can do at once.

Even a correct policy can be too eager. Moving a hundred Pods in a few seconds would put a heavy load on the cluster and would be risky for the applications. The Descheduler therefore provides global limits.

Dry run

The Descheduler has a dry-run mode. In this mode, it goes through every step and writes in its logs which Pods it would evict, but it does not actually evict anything. This is the best way to learn what a new policy will do, because you can read the planned evictions and decide whether they make sense before any application is touched.

Several Kubernetes features move or remove Pods, and it helps to know which one does what.

Here is a story that shows every part working together.

A cluster has three nodes, named A, B and C. During a sale on an online shop, traffic rises and the autoscaler adds two more nodes, D and E. The Pods for the shop’s checkout service are created during the rush, and the scheduler places them wherever there is space. When the sale ends, many Pods are removed, but the old nodes A, B and C still carry most of the remaining Pods. Nodes D and E are almost empty. An engineer has installed the Descheduler as a CronJob with the LowNodeUtilization plugin turned on.

flowchart LR
  subgraph Before
    A1["Node A: crowded"]
    B1["Node B: medium"]
    D1["Node D: empty"]
  end
  subgraph After
    A2["Node A: comfortable"]
    B2["Node B: medium"]
    D2["Node D: comfortable"]
  end
  A1 -->|"Pods evicted and replaced"| D2
        

After the next run, the cluster is more even. Because the engineer used a small limit, only a few Pods moved at a time, and the checkout service stayed available throughout. If the engineer also uses topology spread constraints in the Deployments, the new placements will be spread across zones as well.

The Descheduler can only be gentle if the applications it touches are built to be moved.

Any Pod can be evicted at almost any time in Kubernetes, because of node maintenance, autoscaling, or failures. Applications that survive those events will also survive the Descheduler. So the real work of making a cluster safe for descheduling is making each workload robust.

Stateful applications

Databases and other stateful applications need extra care. A StatefulSet Pod usually keeps its storage when it is recreated, but it may take time to reconnect, rejoin a cluster and warm up its caches. Evicting several such Pods quickly can cause trouble. A common approach is to exclude them from descheduling altogether, or to protect them with strict disruption budgets and a long minimum age.

A program that is allowed to evict Pods is powerful, so its access should be small and well watched.

The Descheduler needs to read information about nodes, Pods, namespaces and priority classes, and to create eviction requests. It does not need to read Secrets, create Pods or change nodes. Granting only those rights through a dedicated service account is the first protection.

If you cannot see what the Descheduler is doing, you cannot trust it.

Logs

The Descheduler writes logs that explain each run: which plugin ran, which nodes were considered, which Pods were nominated, which were filtered out and why, and which were evicted. The detail depends on the log level, and a higher level is helpful when you are learning or debugging. In dry-run mode, the logs are the main way to see what would have happened.

Events and metrics

Evictions also appear as events attached to the Pods, so you can see them with ordinary Kubernetes tools. The Descheduler exposes metrics in a format that common monitoring systems read. These include counts of evicted Pods, grouped by plugin, namespace and result, and measurements of how long each run and each plugin took.

These mistakes are common and each one has a simple cure.

A cautious rollout turns the Descheduler from a risk into a quiet helper.

Most problems fall into a few patterns, and each has an obvious first check.

A calm way to investigate

First, turn on dry run so nothing else changes. Next, raise the log level and read one full run from start to finish. The logs usually say exactly why a Pod was or was not nominated. Then compare what the plugin expected with what the scheduler did when the replacement appeared. Most puzzling cases turn out to be a disagreement between these two.

Different teams use the Descheduler for different reasons.

Short answers to questions that beginners often ask.

No. It only evicts Pods. The scheduler is still the one that picks the new node for the replacement.

It can, as a Deployment with an interval, or it can run on a schedule as a CronJob, or once as a Job. The best choice depends on how quickly you need drift to be corrected.

By default it does not evict system-critical Pods, DaemonSet Pods, static Pods, Pods with local storage, or Pods without an owner. You can adjust these rules, so be careful when you do.

Yes, because it uses the Eviction API. If an eviction would break a budget, the API server refuses it, and the Pod stays where it is.

The scheduler decided that the same node was still the best choice. Add spread or affinity rules to the Pod, or make sure the scheduler’s preferences match the Descheduler’s goal.

Yes. Dry-run mode makes the Descheduler write what it would do to its logs without evicting anything.

By default the utilization plugins use the resource requests of Pods. Newer versions can use live metrics, but this must be configured on purpose.

A drain empties one chosen node, usually for maintenance. The Descheduler looks across the whole cluster and evicts only Pods that break a policy.

Not always. The autoscaler adds and removes nodes but does not move running Pods. The Descheduler helps when you need Pods rebalanced or packed so that nodes can actually be removed.

Yes. You can use priority levels, labels used by the evictor, or an annotation on the Pod to tell the Descheduler to leave it alone.

Kubernetes places each Pod once and then trusts that placement for the Pod’s whole life. As nodes join and leave, rules change and usage shifts, those old decisions become less ideal. The Descheduler closes that gap. It reads a policy, inspects the cluster, nominates Pods that no longer belong where they are, filters out the ones that are unsafe to move, and asks the API server to evict the rest, one careful step at a time. The scheduler then chooses new homes for the replacements. Used with care, small limits and well-built applications, it keeps a cluster balanced, tidy and cost-aware without anyone doing it by hand.