Before we meet taints, we need to understand how Kubernetes decides where each application runs, and why its default answer, “anywhere”, is sometimes wrong.

Pods, nodes, and the scheduler

Kubernetes is a system that runs your applications across many computers and keeps them healthy. Each of those computers is called a node. A node can be a physical server in a data centre or a virtual machine in the cloud. All the nodes together form a cluster.

Your application runs inside a pod, which is the smallest unit Kubernetes manages. A pod wraps one or more containers, and a container is a sealed package holding your program and everything it needs to run. When you ask Kubernetes to run a pod, something must decide which node will host it. That decision-maker is a control plane component called the scheduler.

The scheduler watches for new pods that have no node yet. For each one it picks a node and records that choice. Then a small agent called the kubelet, which runs on every node, notices the assignment and starts the containers. So the scheduler chooses, and the kubelet carries out the choice.

How the scheduler chooses by default

The scheduler works in two steps. First it filters, which means it removes every node that cannot run the pod, for example because it does not have enough free memory. Second it scores, which means it ranks the remaining nodes and picks the best one. By default, the scoring tends to spread pods around so no single node is overloaded.

This default is good for ordinary applications. It treats all nodes as equal servants of the cluster and places pods wherever there is room. The trouble starts when your nodes are not equal, or when some nodes should not be used by some pods at all.

When “anywhere” is the wrong answer

In real clusters, some placements are clearly bad. A small web server should not occupy a very expensive graphics card machine. A pod from one team should not land on nodes bought for another team. No new pod should be started on a node whose disk is failing. The list below shows the main problems that taints and tolerations were created to solve.

A taint is a mark on a node that says “keep away”. A toleration is a note on a pod that says “I can put up with that mark”.

The idea in plain words

A taint is a piece of information attached to a node. It tells the scheduler that ordinary pods should not be placed on this node. A pod that has a matching toleration is allowed to ignore the taint. Pods without a matching toleration are turned away.

The most important detail is that a toleration only gives permission. It never pulls a pod toward the node. A pod that tolerates a taint may still be placed on any other node that has room. If you want a pod to go to a particular node, you must combine the toleration with a separate attraction rule, which we explore in Chapter 8.

Where the names come from

The word “taint” means to mark something as spoiled or unwelcome, which is exactly the effect on a node: it is marked as unwelcome for ordinary pods. The word “toleration” means putting up with something. A pod tolerates a taint the way a person tolerates a strong smell: it does not mind, so it can stay.

The vocabulary you will use

Taints and tolerations touch several parts of Kubernetes, so let us meet each part before we watch them work together.

Labels versus taints

Labels and taints both live on nodes, and both influence scheduling, but they behave in opposite ways. A label is descriptive and neutral, saying something like “this node has a GPU”. It does nothing by itself. Pods can choose to look for it. A taint is a restriction, saying “do not come here unless you are allowed”. It acts on every pod automatically, whether the pod asked or not.

Who does what during scheduling

When you create a pod, the API server saves it without a node. The scheduler notices it, filters nodes using several checks, and one of those checks is the taint check. The scheduler then writes the chosen node into the pod record, and the kubelet on that node starts the containers. Later, if the node develops a problem, the node lifecycle controller may add a taint to it, and that new taint can even lead to the pod being removed again.

This means taints are used at two different moments. They influence where a pod is first placed, and they can force a pod out of a place it already occupies. We will study both moments carefully in the coming chapters.

A taint has three parts: a key, an optional value, and an effect. The effect is the part that decides how strongly the node pushes pods away.

Key and value

The key is a short name that describes why the node is marked, for example “dedicated” or “hardware”. The value is an optional extra detail, for example “machine-learning” or “gpu”. Together they form a label-like pair. You choose these words yourself for your own taints, and Kubernetes chooses them for the automatic taints it adds when a node is unhealthy.

A useful habit is to pick keys that explain the reason for the taint, so anyone looking at a node later can understand it without asking around. A key such as “workload-type” with the value “batch” tells a story. A key such as “x1” tells nothing.

The three effects

The effect decides what happens to pods that do not tolerate the taint. There are exactly three effects, and understanding the difference between them is the heart of this topic.

Several taints on one node

A node can carry many taints at once. The scheduler treats them like a series of locked doors. A pod must have a toleration for every firm taint, meaning every NoSchedule and every NoExecute taint, or it is rejected. If a pod fails to tolerate only PreferNoSchedule taints, it is still allowed, but the node becomes less attractive and scores lower.

If a node has taints for both “dedicated” and “hardware”, a pod that only tolerates “dedicated” will be turned away. This lets you build layered protection, where a pod must show several permissions before it can enter.

A toleration describes which taints a pod can put up with. It has five parts, and small differences in them change what matches.

How matching works

A taint and a toleration match when three things agree. The keys must be the same. The effects must be the same, unless the toleration leaves the effect empty, which means it matches every effect. And the values must agree, according to the operator. With Equal, the toleration’s value must be identical to the taint’s value. With Exists, the value is ignored.

There is one special case worth remembering. A toleration with an empty key and the Exists operator matches every taint on every node. Such a pod ignores all restrictions. This is used by a few system components that must run everywhere, and it should be used with great care in your own applications.

Toleration seconds: staying for a while

When a NoExecute taint appears on a node, a pod that tolerates it can remain. Without a time limit, it stays for ever. With a toleration seconds value, it stays only for that many seconds and is then evicted. This gives applications a grace period, which is useful when a node problem is likely to be short-lived.

For example, if a node briefly loses contact with the cluster, it is better not to move every pod immediately, because the node may come back a few seconds later. A limited toleration says, “Give this node a little while to recover before you kick me out”.

Taints and tolerations are not a single program. They are a small agreement between the node, the pod, the scheduler, and a few controllers.

Everything starts with two pieces of data. The node carries its taints, and the pod carries its tolerations. Both are stored in the cluster database through the API server. Two different components then read this data at different moments. The scheduler reads it when placing a new pod, and a controller reads it to decide whether running pods must leave. The figure below shows the placement path.

flowchart TD
    P["New pod without a node"] --> S["Scheduler"]
    S --> F["Filter step"]
    F --> T{"Does the pod tolerate every firm taint on the node?"}
    T -->|"No"| R["Node rejected"]
    T -->|"Yes"| O["Other checks: resources and affinity"]
    O --> C["Score step: prefer nodes with fewer ignored soft taints"]
    C --> B["Best node chosen and recorded"]
    B --> K["Kubelet starts the pod"]
        

Read the figure from top to bottom. The scheduler takes each candidate node and asks a simple question: does this pod tolerate every firm taint on the node? If not, the node is dropped from the list. If yes, the node continues through the other checks, such as free memory and affinity rules. During scoring, nodes that carry soft taints the pod does not tolerate are ranked lower, so the pod prefers cleaner nodes.

The components you will meet

Let us look inside the two moments when taints matter: when a pod is placed, and when a pod is pushed out.

Moment one: placement

The scheduler is built from many small plugins that each handle one concern. One of them is the taint and toleration plugin. In the filter step it looks at each node’s firm taints, which are the NoSchedule and NoExecute ones, and checks whether the pod has a matching toleration for each. A single unmatched firm taint is enough to reject the node for that pod.

In the score step, the same plugin counts how many soft taints, meaning PreferNoSchedule taints, the pod does not tolerate on each node. The fewer such taints a node has, the higher it scores. This is why a soft taint discourages placement without forbidding it: it only pulls a node’s score down.

Moment two: eviction

Eviction only happens for the NoExecute effect. When a NoExecute taint appears on a node, the eviction logic, which lives inside the node lifecycle controller, looks at every pod on that node. For each pod it works out one of three outcomes.

When a pod is deleted this way, the pod itself is only removed. If it belongs to a Deployment, a Job, or a similar owner, the owner notices that the pod count has dropped and creates a replacement. The scheduler then places that replacement, and it will avoid the tainted node because the new pod, like the old one, does not tolerate the taint. A pod created directly with no owner is simply gone, which is one reason to always run pods through a controller.

sequenceDiagram
    participant O as Operator or Controller
    participant A as API Server
    participant T as Taint Eviction Logic
    participant K as Kubelet
    participant R as ReplicaSet
    participant S as Scheduler
    O->>A: Add NoExecute taint to node
    A->>T: Node change event
    T->>T: Compare taint with each pod's tolerations
    T->>A: Delete pod after its allowed time
    A->>K: Pod deletion, stop containers gracefully
    A->>R: Pod count is below desired
    R->>A: Create replacement pod
    S->>A: Place replacement on a suitable node
        

The sequence shows that eviction is a chain, not a single action. The taint is written, a controller reacts, the pod is deleted, the kubelet stops the containers, the owner replaces the pod, and the scheduler finds a new home. Each link takes a moment, so eviction is quick but not instantaneous.

Why timing is not perfect

The scheduler works from a slightly delayed picture of the cluster. If a taint is added at almost the same moment that a pod is being placed, the pod may land on the node just before the taint takes effect. For NoSchedule, that pod is then left alone. For NoExecute, the eviction logic will catch it shortly afterward. This small window is normal in a distributed system and is one more reason to prefer NoExecute when you truly need a node cleared.

Taints repel and affinity attracts. Real designs almost always need both, and knowing when to use which prevents most scheduling surprises.

Two opposite ideas

A node selector is the simplest attraction rule. The pod says “I only want nodes carrying this label”. Node affinity is a more expressive version. It can say a rule is required or merely preferred, and it can use richer matching. Both attract pods toward nodes and both are written in the pod, using labels on the nodes.

A taint works from the other side. It is written on the node and repels every pod that lacks a matching toleration. The pod does not need to say anything for the taint to act on it, which is why taints are good at protecting nodes from the pods you forgot about.

Why you usually need both

Suppose you have a group of GPU nodes. If you only add a label and a node selector, GPU jobs will go to those nodes, but ordinary pods can still land there too, wasting the expensive machines. If you only add a taint and a toleration, ordinary pods stay away, but the GPU jobs are not forced to go there and may land on a regular node with no GPU and fail.

The solution is to use both. The taint keeps out the wrong pods, the toleration lets the right pods in, and the node affinity pulls the right pods to exactly those nodes. Together they give a group of nodes that is reserved for one workload and used only by that workload.

flowchart LR
    GP["GPU job pod: tolerates the taint and asks for the GPU label"] --> GN["GPU node: has the taint and has the label"]
    WP["Web pod: no toleration"] -.->|"repelled by the taint"| GN
    WP --> GEN["General node: no taint"]
        

In the figure, the GPU job pod carries a toleration and an attraction rule, so it reaches the GPU node. The web pod has no toleration, so the solid line to the GPU node is blocked and it flows to a general node instead. Each mechanism does one job, and none of them alone would give the full result.

Kubernetes uses taints on its own, all the time. Knowing these built-in taints explains many events you will see in a real cluster.

Taints added automatically

The node lifecycle controller adds taints to nodes whenever their condition changes, and removes them again when the problem clears. You do not need to manage these, but you should recognise them, because they explain why pods suddenly move or stop being scheduled.

Why unhealthy nodes wait five minutes

By default, an admission plugin adds two tolerations to every pod for the not-ready and unreachable taints, each with a limit of three hundred seconds. This means a pod on a failing node is not evicted the instant the taint appears. It has five minutes for the node to recover. If the node returns within that time, nothing moves. If not, the pod is evicted and replaced on a healthy node.

This default is a sensible compromise for many applications, but not for all. A shopping checkout service may want a much shorter wait, so users are not stuck on a dead node. A database with heavy local data may want a longer wait, because moving it is expensive. You can override the default by writing your own tolerations with your own seconds.

flowchart TD
    A["Node stops reporting"] --> B["After a short grace period the node is marked unhealthy"]
    B --> C["Controller adds the unreachable taint with NoExecute"]
    C --> D{"Does the pod have its own toleration for it?"}
    D -->|"Yes, with a longer time"| E["Pod stays until its time runs out"]
    D -->|"No, the default applies"| F["Pod removed after about 300 seconds"]
    E --> G["Replacement created on a healthy node"]
    F --> G
        

The figure shows that node failure is handled with the same tools you use yourself. The controller simply adds a taint, and each pod’s tolerations decide how long it waits. Nothing special is needed for failures, which is an elegant design: one mechanism handles both planned restrictions and unplanned problems.

Cordon, drain, and taints

When an operator cordons a node, Kubernetes marks it as unschedulable, and the effect on scheduling is the same as a NoSchedule taint. Draining goes one step further. It cordons the node and then asks the running pods to leave using the eviction process, which respects disruption budgets that protect applications from losing too many pods at once. Note that draining is not the same as a NoExecute taint. A drain is a gentle, controlled request, while a NoExecute taint evicts on its own schedule and does not consult those budgets in the same way.

DaemonSets and system pods

A DaemonSet needs a pod on every node, even nodes that are unhealthy or restricted. To make this work, the DaemonSet controller automatically adds tolerations for several of the built-in node condition taints to the pods it creates. That is why log collectors and network agents keep running on nodes where ordinary pods are kept away.

Pods that request special hardware such as GPUs benefit from a similar helper. An admission plugin can add a toleration automatically for pods that ask for such extended resources, when the nodes are tainted with the resource name. This saves developers from having to remember the toleration themselves.

A taint has a life: it is created, it acts, it is checked, and eventually it is removed. Each stage has traps worth knowing.

Where a taint lives matters. When you add a taint to a node by hand, it is saved on that node’s record in the cluster database. But if the node is replaced, for example by an autoscaler or after an upgrade, the new node starts life with none of your manual changes. The taint is gone, and unwanted pods can flood in. This is why long-lived taints should be set in the node pool configuration, so every node the pool creates receives them automatically.

flowchart TD
    A["Taint added to node"] --> B{"Which effect?"}
    B -->|"NoSchedule"| C["New pods without toleration refused, running pods stay"]
    B -->|"PreferNoSchedule"| D["New pods avoid the node if possible, running pods stay"]
    B -->|"NoExecute"| E["New pods refused, running pods without toleration evicted"]
    E --> F["Tolerating pods stay, forever or until their time ends"]
    C --> G["Taint removed later, node accepts pods again"]
    D --> G
    F --> G
        

Taints and tolerations are simple and powerful, but they are also easy to misuse. Here is an honest view of both sides.

The cost of complexity

Every taint you add creates a small piece of shared knowledge. Someone must remember it, every application team must know it, and every new cluster must repeat it. A small cluster with two node groups can hold this in one person’s head. A large platform with dozens of pools and hundreds of teams needs documentation, naming rules, and automated checks, or the taints turn into a maze.

The good news is that most of this cost can be avoided by keeping the design small. Use a handful of clearly named taints with clear purposes, and prefer defaults that work for most applications over special cases.

Choosing the right tool

Taints look like a security feature, but on their own they are a scheduling feature. Understanding the gap is essential before you rely on them to separate teams.

The central weakness: pods choose their own tolerations

A taint is set by whoever controls the node. A toleration is written by whoever creates the pod. In a cluster shared by many teams, that means any team that can create pods can also write any toleration it likes, including the one that matches a taint reserved for someone else. The taint stops accidental placement, but it does not stop a determined or careless team from adding the matching toleration.

To close this gap, cluster administrators add a second layer of control at the point where pods enter the cluster. This layer can restrict which tolerations each namespace may use, and can add required rules automatically. Only then does the taint start to behave like a real boundary.

Scheduling separation is not runtime isolation

Even a perfect taint design only decides where pods are placed. Pods from one team that share a node with pods from another still share the same operating system kernel and the same hardware. Taints can put different teams on different nodes, which greatly reduces the risk, but they do not add the strong walls that sandboxed runtimes or separate clusters provide.

For sensitive workloads, such as payment processing or regulated data, many organisations use dedicated node pools protected by taints as one layer, and add further layers such as network rules, stricter pod security settings, and sometimes separate clusters.

Taint problems rarely crash anything. They show up as pods that quietly wait, or pods that move when nobody expected. Good signals make both visible.

What to watch

Reading a pending pod

The single most useful skill is reading the reason a pod is pending. When the scheduler fails, it writes an event that summarises every node it considered. The message tells you, node by node, whether the problem was a taint, a lack of resources, or an affinity rule. A message saying that several nodes had a taint the pod did not tolerate is a direct pointer to the missing toleration.

Logs and alerts worth setting up

Send cluster events and audit logs to a central store, because events in Kubernetes expire after a short time. The audit log records who changed a taint and when, which is often the first question in an incident. Useful alerts include pods pending for longer than a chosen time with a taint-related reason, nodes that carry a taint nobody expects, sudden jumps in the number of evictions, and dedicated pools that are nearly empty or completely full.

A few well-proven designs cover most needs. A few tempting shortcuts cause most of the pain.

Patterns that work well

Anti-patterns to avoid

A short field guide collected from teams that run taints in production, and from the ways beginners get caught.

Best practices

• Name taints by purpose — a key such as “workload-type” tells the next person why it exists.
• Set taints at the node pool level — so replacement nodes inherit them automatically.
• Always pair with affinity — the taint keeps others out, the affinity brings your pods in.
• Keep tolerations narrow — match exact keys, values, and effects, and avoid master-key tolerations.
• Prefer NoSchedule for dedicated pools — it is predictable, and does not evict work by surprise.
• Use NoExecute deliberately — apply it only when you truly want running pods removed, and check disruption plans first.
• Roll toleration changes out first — add the new toleration to workloads before changing or adding the taint.
• Keep a general pool — make sure ordinary workloads always have somewhere to run.
• Tune toleration seconds by workload — do not accept the default five minutes for services where it is clearly wrong.
• Guard with admission policy — restrict which namespaces may use which tolerations.
• Document every taint — record its owner, purpose, and the workloads allowed to use it.
• Watch pending pods and evictions — treat them as the health signals of your scheduling design.

Common mistakes

Taints and tolerations appear in almost every serious Kubernetes platform, often behind settings that teams never see directly.

Quick answers to the questions that come up most often when teams start using taints and tolerations.

No. A toleration only permits the pod to be placed there. To make the pod prefer or require those nodes, you need node affinity or a node selector that matches a label on the nodes.

NoSchedule only affects new placements and leaves running pods alone. NoExecute also evicts running pods that do not tolerate the taint, immediately or after their allowed time.

Yes. A pod can list any number of tolerations. If a node carries several firm taints, the pod needs a matching toleration for every one of them.

The pod stays pending. The scheduling failure event lists the taints that blocked it, which tells you exactly what to add or which taint to remove.

Only the NoExecute effect does. NoSchedule and PreferNoSchedule influence future placement and leave running pods where they are.

Read the events on the pod. They summarise how many nodes were rejected and for what reasons, including untolerated taints, lack of resources, and affinity mismatches.

Yes. Managed services and autoscalers let you set taints and labels in the node pool or node template, so every node they create receives them automatically.

Not by themselves. Teams that can create pods can write their own tolerations. Combine taints with admission policy, network rules, and stronger isolation where the risk is high.

It is very close. Cordoning marks the node as unschedulable, which has the same effect on new pods as a NoSchedule taint. Draining adds a controlled removal of running pods on top of that.

By default, pods receive tolerations for the not-ready and unreachable taints with a three-hundred-second limit. You can set shorter or longer limits in your own pod definitions.

They do, but the DaemonSet controller automatically adds tolerations for several built-in node condition taints. For your own custom taints, you must add the tolerations yourself.

No. An unused toleration simply does nothing. That is useful when you prepare workloads in advance of adding a taint, which is the safe order for rolling out changes.

Taints and tolerations are a small pair of ideas with a very wide reach. A taint marks a node as unwelcome to ordinary pods, and a toleration gives a pod permission to ignore that mark. Three effects give you a soft preference, a firm refusal, and a refusal that also removes running pods. Kubernetes uses the same mechanism for its own housekeeping, from protecting control plane nodes to reacting to failing machines.

The power comes with responsibility. Tolerations permit but never attract, so real designs pair them with node affinity. Taints alone are not a security wall, so shared clusters need admission policy on top. And taints that live only on hand-edited nodes vanish when the nodes are replaced. Teams that respect these limits get cleaner clusters, lower costs, and calmer maintenance.